Network Protocol Methods

Computer systems need to communicate with one another.  One computer sends data to another, assuming the other is listening.  Down at the wire, the medium over which network information travels, communications are nothing more than electrons or photons.  At this level, the protocol is governed by the laws of physics. The mechanics of how the information is physically transported from point A to point B isn't relevant from the software perspective.  My system has data it wants to deliver to a remote region of the world.  As far as my software is concerned, the hardware could relay it via horseback.  It might take a year, but it would get there.

Of course, only the fastest delivery of information over networks will satisfy the demands of today's information-hungry applications.  There is so much information to absorb, it could not possibly fit in an isolated environment.  Information moves so fast that the only practical means of staying current is to continuously synchronize with other computers that have newer information.  In this regard, the barrier between stored information and information elicited from foreign sources has been permanently blurred.

How did we get to where we are today, and are we doing it effectively?  Big data is only big because software systems are decoupled, conjoined by cables and airwaves.  Big data translates into big connectivity.  Big connectivity can only mean that software systems that talk to one another also understand one another. There is an agreed-upon protocol that computer systems use to pack and unpack transmitted data.  But there is more to network protocols than how to extract information from data.  The web works the way it does because the protocol has application-level constructs that influence behavior.  What can we expect to see with future network protocols?  Will current protocols evolve into something that can support the ever growing vastness of our connectivity?

Sending and Receiving
Data doesn't traverse over a network spontaneously.  Applications instruct the data, influencing where it goes, how to get there, and what form to assume during it's journey.  These properties of network traffic are controlled through network methods.  A method pertaining to a network is something that an application uses to send and receive data.  Developers have a choice — we can get down to low-level socket details, specifying properties on individual network packets, or we can use higher level protocol methods that are more reflective of the architecture used at the application level.

Before we delve into the stuff applications use to make networks comprehensible from a development perspective, maybe it'll help us to think a little more about why additional semantics are necessary.  That is, we can't we simply use send and receive methods to exchange our data?

On the one hand, send and receive are simple — straightforward.  A handful of properties can tweak how our data is transferred and accepted on the remote end. Contrast this with a dozen methods to choose from, each with their own properties — a matrix of complexity emerges.  Conceptually, two methods means there are relatively few abstractions in the protocol to concern ourselves with.  We've got senders, receivers.  We've got packets, and routes.  We've got destinations and sources.  What this amounts to is point A and point B — with a few added details underneath.

The fact that we're able to use these seemingly low-level methods to control the flow of information over a network is due to even lower-level networking ideas. These ideas go beyond the IP layer of any network protocol stack.  Things that are handled by the operating system in the link layer, take care of a tangled mess for us.  If not for the operating system providing us with abstractions to overcome challenges associated with hardware compatibility, just imagine what the networking code for any application would look like.  So with the lowest-level network protocol details out of the way, the operating system has provided us with two basic methods, at the most fundamental conceptual level.

These two basic methods give applications everything they need to establish a connection and to communicate with remote entities — with meager networking code inside the application.  The challenge introduced by having simplistic network protocol methods is in the data itself.  It's easy to send and receive data.  It's difficult to make sense of that data.

Application Protocols
With the amount of data that needs to be transferred between modern applications over modern networks, the simplistic send and receive methods simply do not suffice.  Even smaller applications that aren't of any significant size in terms of complex features struggle to make sense of the data they're exchanging.  This is simply due to the complexity of the data, and the diversity of it.  Chances are, any given application you're using communicates with another service.  The two parties need to understand one another, the receiver needs to know how to unpack the data upon arrival.  This works well at the operating system level — we can send and receive raw data with ease.  But in the context of an application, there needs to be an agreed-upon format — a standard — before it becomes information.

This is why standardized data formats exist — to make communicating over a network easier for applications.  If two applications want to talk to one another, they might decide to use XML as the exchange format.  This way the sender knows how to pack the data and the receiver knows how to unpack and use that data.  The idea of standardized formats definitely makes life in a network much easier for developers in organizations extrinsic to one another.  One service makes interesting data available, in a particular format, perhaps even a selection of formats.  External entities can then consume that data, knowing how to read it, how to transform that data into information of value.

Standardized formats that applications send over the wire are of a different nature than that of the network method.  The data and it's assumed form is the what of the larger network picture.  The methods are the how.  Part of the problem we're seeing with the ever growing complexity of network-enabled applications is that the how isn't exactly clear-cut.  Why is this data being sent?  How should I respond to it? With standardized formats in place, it's easy to embed these types of semantics in the messages.  But it's difficult to sustain this approach on an Internet scale.

This is where HTTP comes in handy.  It's a protocol web applications and clients can use and readily understand without trying to stuff in new behavior.  That is, HTTP already has a set of methods as part of the specification that allows unambiguous intent to be transferred with each request.  You can GET a resource, POST a new resource, PUT new information on an existing resource, and DELETE a resource.  Like the concept of sending and receiving methods, the HTTP methods are simple, there are only four of them (aside from HEAD and OPTIONS which aren't widely used if at all).  Also like send and receive, HTTP methods are utilitarian — they don't pertain to any one specific application domain.

How well does it work?  How much longer will HTTP be able to support the activities of the web given it's current capabilities?

The Future of Network Methods
What does the future hold for methods that make computer networks functional, and not just a jumbled mess of connected cables?  Is what we have good enough to take us into the next five or ten years?  Well, if we consider how fast the web is growing today, that means a whole lot of new resources that'll come into being. That's many URIs to which HTTP methods can be applied.  Does that mean that we'll finally see a weakness of the protocol exposed, that four methods for interacting with networks in fact wasn't enough to do everything we needed?

It's a difficult question to answer because much of the global network activity isn't necessarily web based.  Not every agent wired to a network is a web browser, and HTTP isn't the silver bullet.  For example, real-time publish-subscribe systems use different standards such as AMQP.  Here we have an entirely different set of protocol methods with their own unique advantages, constraints, and limitations. So which network communications protocol are you going to adopt for your application?  Is it even a choice of one or the other, and can you interchange between two or more sets of defined network methods?

In the fast approaching future, the trouble is going to be dealing with large amounts of data, compounded by the large volume of network activity triggered by the growing online population.  We're shown examples of successful implementations that can handle anything thrown at them.  So you would think that we're safe with everything we have in place, right?  Well, that may not be the case, as we thought we had enough IPV4 addresses ten years ago.  So whats wrong with the current implementations?  If we can just hire the right people who've got the experience in constructing these hugely resilient infrastructures that can survive the apocalypse with minimal downtime, we don't have much to worry about.

Well, not everyone in IT is a rocket surgeon capable of cranking out enterprise-grade networked application suites while blindfolded.  We still need to strive for simplicity in the wake of an ever growing number of variables.  The small number of methods bundled with the HTTP protocol is why it's been such a success.  It's a small improvement over the send and receive methods, small enough to remain lucid yet powerful enough to transfer application operations over a network.

The real danger, I think, is in how some of the more powerful network protocols have a tendency to be misused.  HTTP is a good example, probably because it is so widely used, it is also widely misused.  For instance, sending POST requests to update existing resources, or deleting resources through means other than DELETE requests.  If enough applications continue to evolve in this manor, a lot of the value with HTTP is lost because different applications are inventing their own protocols on top of another.  We're simply inventing methods when they should be part of a standard, such was the case when we decided to improve on the send and receive methods by using a standardized format.  In reality, that's all there is to HTTP, and many other protocols — added data in the transferred network data. But we need to abide by the standards because deviations will only proliferate as the web evolves into something much too large to handle all these broken down contracts of communication.

The end result of such a breakdown?  The networks will always be there because we have such a firm grip on the low-level fundamentals that no developer need touch them.  Will the networks of the future, the web in particular, be all that it can be?  Will it provide the best possible service to all it's inhabitants under an array of circumstances?  It doesn't do that now.  I think it has potential to, but it'll take better cooperation between those who build the most flourishing systems on the web.  To standardize would be simply fantastic, because we could sit down and objectively measure how well our computer networks measure up with the protocols we have in place at the application level.  Maybe they don't.  But the solution isn't to invent on top of existing standards.  Let's make HTTP work the way it's supposed to.  Maybe then we'll see if the methods we have will do just fine in ten years from now.

The Social Protocol

What are social networks? They're websites people from around the globe use to communicate with one another. What is a website? They're a bunch of related resources linked together, and requested by web browsers. How are resources requested and returned? The browser sends an HTTP request to the website, and displays the result. This is how the overabundance of social networks work in their most basic sense – like websites do. Users around the globe, social network or otherwise, are connected via HTTP. I wonder, is HTTP an inherently social protocol, or does it appear that way because of the social phenomenon on the web? There are some interesting properties of the protocol that give it the appearance of being inherently social.

Just in case you don't think social networks are pervasive in the web space, try visiting any major site. You probably can't find many that don't have social network buttons embedded in the interface. The idea with these widgets isn't to build more incoming links. No, the idea is to be social about the subject matter. To talk about it. To rant and rave about it for that matter. Being social is about spawning new ideas through collaboration. Whether the majority of the discussions that take place over social networks add anything of value is subjective but nevertheless, powerful. We have networks of opinions we can traverse and mull over if we're so inclined. We even have tools that will search the these opinions, assessments, and wild theories, extracting the stuff we want. Think about the way a search engine works on the web - the same idea, only the data set we're querying is created by social networking tools. The data is linked together the same way the rest of web is, discussions about sirloin steak are probably linked to discussions about seasoning salt. Likewise on the rest of the web, only its pages that are linked, not discussions.

The world is a big place. The advent of the web made it an order of magnitude smaller. The web was designed for connectedness, hence the name. Collaboration, asking questions, getting answers, expressing opinions. What exactly is sent over the wire? We don't want to send raw text to our collaborators. We need content to be readable. We need to identify headings, and images. We need to emphasize certain points of interest in our content. A web browser parses a markup language and presents formatted information to the receiving user. This is the ML in HTML. Hyper-text is text with links to other URLs. The user supplies the link, and they're taken to the new location on the web. Addressability, a formatting language, a web browser that does all the underlying work. What else do we need to communicate over the web in a meaningful way?

Fast-forwarding a little, back to where social networks are the driving force of the web. How did the social phenomenon start? We used to have websites for sought-after information. Good luck finding an answer to your question should a web page dedicated to the topic not already exist. Of course, forums solved this problem originally, but there was something missing. Something that needed a personal opinion couldn't be done here. However, for general topics, forums are a great way to share knowledge. Opinions are expressed, knowledge conveyed, and not only is my question answered, but the thread lives on and helps thousands of people in the same situation I was in two months ago. Social networks took the concept of forums, groups, threaded-discussions and changed two things. First, you're no longer having a threaded-discussion with people you don't know. You're talking about stuff with friends, colleagues, people you've allowed access. This is your social circle, and your supposed to value their opinion over that of anonymous entities on the rest of the web. Second, the idea of a discussion being centered around a topic as been largely diminished in social networks. Essentially, in social networks, the user is the topic. Everything else is centered around the what they decide is relevant.

Back to the pre-social network web, HTML pages, web browsers, and the glue holding it all together – HTTP. The hyper-text transfer protocol is how we get from one page to another, load the page initially, and interact with the servers that store these pages. More importantly, HTTP gives the web a unified interface for these interactions. Browsers send HTTP requests to a location, a URL. An HTTP request has a method that tells the server what to do with the resource when it receives the request. For instance, if I want to retrieve an HTML page from a given URL, I send a GET HTTP request to that URL. GET means retrieve the HTML page and send it back to my browser. Say I'm on a site the lets me review cars. I sill out a review form and submit it. When I do this, I'm posting a new resource. For this I use the POST method. These two methods are the most common. When you're browsing the web, you might visit 50 to 100 pages. All of these requests are GET requests.

There are other, less-common HTTP methods as well - PUT, and DELETE. These methods also modify resources. If I want to delete a web page, I send a DELETE request to the page URL. If I want to update an existing resource, say change the text, I send a PUT request. Together, these four methods represent the unified behavior interface of the web. These are the actions of the web, create (POST), read (GET), update (PUT), and delete (DELETE). 

 

The social networks of today use HTTP because they're websites. HTTP has a unified interface, a small set of methods that tell the server what you want to do with the resource. Social networks generalized the concept of discussion topic and externalized their tools that allow readers of the web to point social network users to that web resource. Just like the TCP/IP protocol is general, you use it to send or receive data over the Internet, websites are general, you visit them seeking information or to perform whatever task you need. Just like HTTP provides a way for browsers to work with resource representations, social networks provide a way for users to connect with one another, regardless of the topic.

URLs are essential for sharing things on the web. Users point their browsers to URLs. In return, they get whatever the person on the other end is sharing. Generating new web resources, HTML pages and URLs is very easy today. Not just creating new resources, but managing their entire life cycle is trivial for most people. Social network or no social network, web applications make it easy to post new content, modify it, or get rid of it entirely. This is the social aspect of HTTP - the unified interface of GET, POST, PUT, and DELETE. Sometimes, you're required to supply specific parameters in your post to create a new recipe. You might need to specify a sorting column when retrieving a list of hockey teams. The question is, how do we get these parameters and how do we know which method to use?

Most resources on the web are simple. That is, they're meant to be read. This is why the default method used by a browser when pasting a URL is GET. The web is just like the real world in that you need to read before you write. This means you head to a website and see what it does, who owns it, does it interest you, is it safe, etc. Only afterward do we decide that we want to provide our own insights, thoughts, images, video, status, etc. These are objects we post onto the web using a URL endpoint. We update these objects later on by altering their attributes and putting them back. When they become irrelevant, we delete them. Or they're deleted for us. The work-flow, read about the service, post something, update it, is all derived from the end user content. The content dictates which which direction to take, which HTTP method to use.

To get a better idea of how HTTP is the ideal protocol for social work-flows, we need to think about RESTful web applications. That is, hypermedia as the engine of application state. Resources on the web don't have any state in the web browser. This means that what you're viewing right now is only a copy, a representation of the resource. The real one is on the server. Your web browser, being the application, needs direction on how to proceed, which URL to visit next, which method to use. Links do a good job of this. All the user needs to do is think "I want to go here" and click the link pointing to the new location. The browser says "GET me resource xyz". Now I can read the content and make a decision, what I want to do next. In this sense the user is the application. Imagine a different kind of web browser, one that doesn't care about presenting content to the user, one that only cares about making decisions a human would otherwise make. This kind of browser would maintain application state by continuously reading directions sent back from resources on the server and making the next move. There are, however, two things that this browser would struggle with.

The first, how does the browser know what it's options are? How does it know what it's next potential moves are, the URLs and expected methods. It would have to do what a human does, read the content in order to decide what to do next. So our new browser will need a representation it understands, XML for example. Inside the XML, we have content, probably related to the application domain our browser was built for. We also have link fields embedded in these server resource representations. These links aren't like regular links, they also have the HTTP method we're supposed to use. The browser then knows what method to use, it knows whether it is posting a new object or retrieving something that exists already.

The second problem, how does the browser know what the expected parameters are for any given resource? If a list of account transactions supports a page size, how does the browser know about it? The answer in HTML browsers is that the page designer implements a page size selector. The human operating the browser makes the decision by choosing an appropriate size. Our new browser has no luxery of human thought, only resource representations, URLs, and HTTP methods. They can only help guide our application if they provide the full extent of what can be done, how else can our application make an informed decision? It can't very well create new objects without knowing what fields are required. 

 

One option for these issues is an HTML form. Forms can store both the URL and the method required for any resource. Additionally, the input fields within make it easy for our browser to learn what the acceptable parameters for any given resource are. It doesn't need to make guesses by attempting to parse natural language, it simply reads the language it understands and acts accordingly.

The HTTP protocol itself isn't so much social as are the representations that HTTP clients interact with. If you visit someone's profile on a social networking site, that is a representation of what that person is doing, thinking, attending, reading, whatever. Based on what the visitor sees here, they make their decision on what to do next. Social networks make it so easy to hop around inside, jump from profile to profile, event to event. This is because they link to one another. Hypermedia. The engine of application state. If you update your profile, you're probably not sending a PUT request. If you delete a photo, you're probably not sending a DELETE request. This doesn't really matter because the ideas that make social networks social came from the web, refined and put in a confined container. HTTP is the interface that makes the web social.