Network Protocol Methods

Computer systems need to communicate with one another.  One computer sends data to another, assuming the other is listening.  Down at the wire, the medium over which network information travels, communications are nothing more than electrons or photons.  At this level, the protocol is governed by the laws of physics. The mechanics of how the information is physically transported from point A to point B isn't relevant from the software perspective.  My system has data it wants to deliver to a remote region of the world.  As far as my software is concerned, the hardware could relay it via horseback.  It might take a year, but it would get there.

Of course, only the fastest delivery of information over networks will satisfy the demands of today's information-hungry applications.  There is so much information to absorb, it could not possibly fit in an isolated environment.  Information moves so fast that the only practical means of staying current is to continuously synchronize with other computers that have newer information.  In this regard, the barrier between stored information and information elicited from foreign sources has been permanently blurred.

How did we get to where we are today, and are we doing it effectively?  Big data is only big because software systems are decoupled, conjoined by cables and airwaves.  Big data translates into big connectivity.  Big connectivity can only mean that software systems that talk to one another also understand one another. There is an agreed-upon protocol that computer systems use to pack and unpack transmitted data.  But there is more to network protocols than how to extract information from data.  The web works the way it does because the protocol has application-level constructs that influence behavior.  What can we expect to see with future network protocols?  Will current protocols evolve into something that can support the ever growing vastness of our connectivity?

Sending and Receiving
Data doesn't traverse over a network spontaneously.  Applications instruct the data, influencing where it goes, how to get there, and what form to assume during it's journey.  These properties of network traffic are controlled through network methods.  A method pertaining to a network is something that an application uses to send and receive data.  Developers have a choice — we can get down to low-level socket details, specifying properties on individual network packets, or we can use higher level protocol methods that are more reflective of the architecture used at the application level.

Before we delve into the stuff applications use to make networks comprehensible from a development perspective, maybe it'll help us to think a little more about why additional semantics are necessary.  That is, we can't we simply use send and receive methods to exchange our data?

On the one hand, send and receive are simple — straightforward.  A handful of properties can tweak how our data is transferred and accepted on the remote end. Contrast this with a dozen methods to choose from, each with their own properties — a matrix of complexity emerges.  Conceptually, two methods means there are relatively few abstractions in the protocol to concern ourselves with.  We've got senders, receivers.  We've got packets, and routes.  We've got destinations and sources.  What this amounts to is point A and point B — with a few added details underneath.

The fact that we're able to use these seemingly low-level methods to control the flow of information over a network is due to even lower-level networking ideas. These ideas go beyond the IP layer of any network protocol stack.  Things that are handled by the operating system in the link layer, take care of a tangled mess for us.  If not for the operating system providing us with abstractions to overcome challenges associated with hardware compatibility, just imagine what the networking code for any application would look like.  So with the lowest-level network protocol details out of the way, the operating system has provided us with two basic methods, at the most fundamental conceptual level.

These two basic methods give applications everything they need to establish a connection and to communicate with remote entities — with meager networking code inside the application.  The challenge introduced by having simplistic network protocol methods is in the data itself.  It's easy to send and receive data.  It's difficult to make sense of that data.

Application Protocols
With the amount of data that needs to be transferred between modern applications over modern networks, the simplistic send and receive methods simply do not suffice.  Even smaller applications that aren't of any significant size in terms of complex features struggle to make sense of the data they're exchanging.  This is simply due to the complexity of the data, and the diversity of it.  Chances are, any given application you're using communicates with another service.  The two parties need to understand one another, the receiver needs to know how to unpack the data upon arrival.  This works well at the operating system level — we can send and receive raw data with ease.  But in the context of an application, there needs to be an agreed-upon format — a standard — before it becomes information.

This is why standardized data formats exist — to make communicating over a network easier for applications.  If two applications want to talk to one another, they might decide to use XML as the exchange format.  This way the sender knows how to pack the data and the receiver knows how to unpack and use that data.  The idea of standardized formats definitely makes life in a network much easier for developers in organizations extrinsic to one another.  One service makes interesting data available, in a particular format, perhaps even a selection of formats.  External entities can then consume that data, knowing how to read it, how to transform that data into information of value.

Standardized formats that applications send over the wire are of a different nature than that of the network method.  The data and it's assumed form is the what of the larger network picture.  The methods are the how.  Part of the problem we're seeing with the ever growing complexity of network-enabled applications is that the how isn't exactly clear-cut.  Why is this data being sent?  How should I respond to it? With standardized formats in place, it's easy to embed these types of semantics in the messages.  But it's difficult to sustain this approach on an Internet scale.

This is where HTTP comes in handy.  It's a protocol web applications and clients can use and readily understand without trying to stuff in new behavior.  That is, HTTP already has a set of methods as part of the specification that allows unambiguous intent to be transferred with each request.  You can GET a resource, POST a new resource, PUT new information on an existing resource, and DELETE a resource.  Like the concept of sending and receiving methods, the HTTP methods are simple, there are only four of them (aside from HEAD and OPTIONS which aren't widely used if at all).  Also like send and receive, HTTP methods are utilitarian — they don't pertain to any one specific application domain.

How well does it work?  How much longer will HTTP be able to support the activities of the web given it's current capabilities?

The Future of Network Methods
What does the future hold for methods that make computer networks functional, and not just a jumbled mess of connected cables?  Is what we have good enough to take us into the next five or ten years?  Well, if we consider how fast the web is growing today, that means a whole lot of new resources that'll come into being. That's many URIs to which HTTP methods can be applied.  Does that mean that we'll finally see a weakness of the protocol exposed, that four methods for interacting with networks in fact wasn't enough to do everything we needed?

It's a difficult question to answer because much of the global network activity isn't necessarily web based.  Not every agent wired to a network is a web browser, and HTTP isn't the silver bullet.  For example, real-time publish-subscribe systems use different standards such as AMQP.  Here we have an entirely different set of protocol methods with their own unique advantages, constraints, and limitations. So which network communications protocol are you going to adopt for your application?  Is it even a choice of one or the other, and can you interchange between two or more sets of defined network methods?

In the fast approaching future, the trouble is going to be dealing with large amounts of data, compounded by the large volume of network activity triggered by the growing online population.  We're shown examples of successful implementations that can handle anything thrown at them.  So you would think that we're safe with everything we have in place, right?  Well, that may not be the case, as we thought we had enough IPV4 addresses ten years ago.  So whats wrong with the current implementations?  If we can just hire the right people who've got the experience in constructing these hugely resilient infrastructures that can survive the apocalypse with minimal downtime, we don't have much to worry about.

Well, not everyone in IT is a rocket surgeon capable of cranking out enterprise-grade networked application suites while blindfolded.  We still need to strive for simplicity in the wake of an ever growing number of variables.  The small number of methods bundled with the HTTP protocol is why it's been such a success.  It's a small improvement over the send and receive methods, small enough to remain lucid yet powerful enough to transfer application operations over a network.

The real danger, I think, is in how some of the more powerful network protocols have a tendency to be misused.  HTTP is a good example, probably because it is so widely used, it is also widely misused.  For instance, sending POST requests to update existing resources, or deleting resources through means other than DELETE requests.  If enough applications continue to evolve in this manor, a lot of the value with HTTP is lost because different applications are inventing their own protocols on top of another.  We're simply inventing methods when they should be part of a standard, such was the case when we decided to improve on the send and receive methods by using a standardized format.  In reality, that's all there is to HTTP, and many other protocols — added data in the transferred network data. But we need to abide by the standards because deviations will only proliferate as the web evolves into something much too large to handle all these broken down contracts of communication.

The end result of such a breakdown?  The networks will always be there because we have such a firm grip on the low-level fundamentals that no developer need touch them.  Will the networks of the future, the web in particular, be all that it can be?  Will it provide the best possible service to all it's inhabitants under an array of circumstances?  It doesn't do that now.  I think it has potential to, but it'll take better cooperation between those who build the most flourishing systems on the web.  To standardize would be simply fantastic, because we could sit down and objectively measure how well our computer networks measure up with the protocols we have in place at the application level.  Maybe they don't.  But the solution isn't to invent on top of existing standards.  Let's make HTTP work the way it's supposed to.  Maybe then we'll see if the methods we have will do just fine in ten years from now.

The Social Protocol

What are social networks? They're websites people from around the globe use to communicate with one another. What is a website? They're a bunch of related resources linked together, and requested by web browsers. How are resources requested and returned? The browser sends an HTTP request to the website, and displays the result. This is how the overabundance of social networks work in their most basic sense – like websites do. Users around the globe, social network or otherwise, are connected via HTTP. I wonder, is HTTP an inherently social protocol, or does it appear that way because of the social phenomenon on the web? There are some interesting properties of the protocol that give it the appearance of being inherently social.

Just in case you don't think social networks are pervasive in the web space, try visiting any major site. You probably can't find many that don't have social network buttons embedded in the interface. The idea with these widgets isn't to build more incoming links. No, the idea is to be social about the subject matter. To talk about it. To rant and rave about it for that matter. Being social is about spawning new ideas through collaboration. Whether the majority of the discussions that take place over social networks add anything of value is subjective but nevertheless, powerful. We have networks of opinions we can traverse and mull over if we're so inclined. We even have tools that will search the these opinions, assessments, and wild theories, extracting the stuff we want. Think about the way a search engine works on the web - the same idea, only the data set we're querying is created by social networking tools. The data is linked together the same way the rest of web is, discussions about sirloin steak are probably linked to discussions about seasoning salt. Likewise on the rest of the web, only its pages that are linked, not discussions.

The world is a big place. The advent of the web made it an order of magnitude smaller. The web was designed for connectedness, hence the name. Collaboration, asking questions, getting answers, expressing opinions. What exactly is sent over the wire? We don't want to send raw text to our collaborators. We need content to be readable. We need to identify headings, and images. We need to emphasize certain points of interest in our content. A web browser parses a markup language and presents formatted information to the receiving user. This is the ML in HTML. Hyper-text is text with links to other URLs. The user supplies the link, and they're taken to the new location on the web. Addressability, a formatting language, a web browser that does all the underlying work. What else do we need to communicate over the web in a meaningful way?

Fast-forwarding a little, back to where social networks are the driving force of the web. How did the social phenomenon start? We used to have websites for sought-after information. Good luck finding an answer to your question should a web page dedicated to the topic not already exist. Of course, forums solved this problem originally, but there was something missing. Something that needed a personal opinion couldn't be done here. However, for general topics, forums are a great way to share knowledge. Opinions are expressed, knowledge conveyed, and not only is my question answered, but the thread lives on and helps thousands of people in the same situation I was in two months ago. Social networks took the concept of forums, groups, threaded-discussions and changed two things. First, you're no longer having a threaded-discussion with people you don't know. You're talking about stuff with friends, colleagues, people you've allowed access. This is your social circle, and your supposed to value their opinion over that of anonymous entities on the rest of the web. Second, the idea of a discussion being centered around a topic as been largely diminished in social networks. Essentially, in social networks, the user is the topic. Everything else is centered around the what they decide is relevant.

Back to the pre-social network web, HTML pages, web browsers, and the glue holding it all together – HTTP. The hyper-text transfer protocol is how we get from one page to another, load the page initially, and interact with the servers that store these pages. More importantly, HTTP gives the web a unified interface for these interactions. Browsers send HTTP requests to a location, a URL. An HTTP request has a method that tells the server what to do with the resource when it receives the request. For instance, if I want to retrieve an HTML page from a given URL, I send a GET HTTP request to that URL. GET means retrieve the HTML page and send it back to my browser. Say I'm on a site the lets me review cars. I sill out a review form and submit it. When I do this, I'm posting a new resource. For this I use the POST method. These two methods are the most common. When you're browsing the web, you might visit 50 to 100 pages. All of these requests are GET requests.

There are other, less-common HTTP methods as well - PUT, and DELETE. These methods also modify resources. If I want to delete a web page, I send a DELETE request to the page URL. If I want to update an existing resource, say change the text, I send a PUT request. Together, these four methods represent the unified behavior interface of the web. These are the actions of the web, create (POST), read (GET), update (PUT), and delete (DELETE). 

 

The social networks of today use HTTP because they're websites. HTTP has a unified interface, a small set of methods that tell the server what you want to do with the resource. Social networks generalized the concept of discussion topic and externalized their tools that allow readers of the web to point social network users to that web resource. Just like the TCP/IP protocol is general, you use it to send or receive data over the Internet, websites are general, you visit them seeking information or to perform whatever task you need. Just like HTTP provides a way for browsers to work with resource representations, social networks provide a way for users to connect with one another, regardless of the topic.

URLs are essential for sharing things on the web. Users point their browsers to URLs. In return, they get whatever the person on the other end is sharing. Generating new web resources, HTML pages and URLs is very easy today. Not just creating new resources, but managing their entire life cycle is trivial for most people. Social network or no social network, web applications make it easy to post new content, modify it, or get rid of it entirely. This is the social aspect of HTTP - the unified interface of GET, POST, PUT, and DELETE. Sometimes, you're required to supply specific parameters in your post to create a new recipe. You might need to specify a sorting column when retrieving a list of hockey teams. The question is, how do we get these parameters and how do we know which method to use?

Most resources on the web are simple. That is, they're meant to be read. This is why the default method used by a browser when pasting a URL is GET. The web is just like the real world in that you need to read before you write. This means you head to a website and see what it does, who owns it, does it interest you, is it safe, etc. Only afterward do we decide that we want to provide our own insights, thoughts, images, video, status, etc. These are objects we post onto the web using a URL endpoint. We update these objects later on by altering their attributes and putting them back. When they become irrelevant, we delete them. Or they're deleted for us. The work-flow, read about the service, post something, update it, is all derived from the end user content. The content dictates which which direction to take, which HTTP method to use.

To get a better idea of how HTTP is the ideal protocol for social work-flows, we need to think about RESTful web applications. That is, hypermedia as the engine of application state. Resources on the web don't have any state in the web browser. This means that what you're viewing right now is only a copy, a representation of the resource. The real one is on the server. Your web browser, being the application, needs direction on how to proceed, which URL to visit next, which method to use. Links do a good job of this. All the user needs to do is think "I want to go here" and click the link pointing to the new location. The browser says "GET me resource xyz". Now I can read the content and make a decision, what I want to do next. In this sense the user is the application. Imagine a different kind of web browser, one that doesn't care about presenting content to the user, one that only cares about making decisions a human would otherwise make. This kind of browser would maintain application state by continuously reading directions sent back from resources on the server and making the next move. There are, however, two things that this browser would struggle with.

The first, how does the browser know what it's options are? How does it know what it's next potential moves are, the URLs and expected methods. It would have to do what a human does, read the content in order to decide what to do next. So our new browser will need a representation it understands, XML for example. Inside the XML, we have content, probably related to the application domain our browser was built for. We also have link fields embedded in these server resource representations. These links aren't like regular links, they also have the HTTP method we're supposed to use. The browser then knows what method to use, it knows whether it is posting a new object or retrieving something that exists already.

The second problem, how does the browser know what the expected parameters are for any given resource? If a list of account transactions supports a page size, how does the browser know about it? The answer in HTML browsers is that the page designer implements a page size selector. The human operating the browser makes the decision by choosing an appropriate size. Our new browser has no luxery of human thought, only resource representations, URLs, and HTTP methods. They can only help guide our application if they provide the full extent of what can be done, how else can our application make an informed decision? It can't very well create new objects without knowing what fields are required. 

 

One option for these issues is an HTML form. Forms can store both the URL and the method required for any resource. Additionally, the input fields within make it easy for our browser to learn what the acceptable parameters for any given resource are. It doesn't need to make guesses by attempting to parse natural language, it simply reads the language it understands and acts accordingly.

The HTTP protocol itself isn't so much social as are the representations that HTTP clients interact with. If you visit someone's profile on a social networking site, that is a representation of what that person is doing, thinking, attending, reading, whatever. Based on what the visitor sees here, they make their decision on what to do next. Social networks make it so easy to hop around inside, jump from profile to profile, event to event. This is because they link to one another. Hypermedia. The engine of application state. If you update your profile, you're probably not sending a PUT request. If you delete a photo, you're probably not sending a DELETE request. This doesn't really matter because the ideas that make social networks social came from the web, refined and put in a confined container. HTTP is the interface that makes the web social.

HTML Forms

Forms are useful tools for collecting information, whether they're physical, or or your computer screen. When we need specific bits of information, brevity is of virtue. Collecting these data fragments are why forms exist. A form value is as simple as a yes or no. They're as complex as a line or two of text. The web-equivalent of a paper form is the HTML form element. Given that a form is used to capture information, and, that is what it is good at, why aren't forms used more often in web pages? Not using a form element to gather user input is a classic example of reinventing the wheel. So why do we do it?

Maybe we don't like to use forms because we associate them with legacy-style HTML pages from the nineties. But forms aren't visual elements. Its the elements within the form that are displayed, the text inputs, the radio buttons, and selectors you can see. If a form is a non-visual element, that is, a structural element used to group others, why go out of our way to submit data by other means?

The ultimate purpose of a form element is pragmatic - it is functional, not presentational. There are two other data items forms store in addition to user input - the action and the method. The form action is a URI. A URI usually represents some data resource on the server. The form method corresponds to the HTTP method used to submit the form. When a form is submitted, the HTTP request is made with the stored action and method.

Another example of an element that stores an action and a method is an anchor tag. Links store the action in the href attribute. Just like forms, actions are URIs that represent resources on the server. So where exactly is the method stored? Links store the HTTP GET method implicitly. There is no way to click a link that sends a POST request without intervening with Javascript. Unlike forms, however, links take on the dual role of being both a functional and a presentational element - you can see links on the web page.

Since links can't send anything other than GET requests, we are more likely to use Javascript - assuming we want to make a POST request by clicking a link. This gives us full control over the HTTP request. When a user clicks a link, or any other element listening for click events, some Javascript function is triggered that will assemble the HTTP request. The request is then sent asynchronously behind the scenes. This work flow is especially easy with libraries such as jQuery. All it takes is a few lines of code to build the request and send it.

There is nothing inherently wrong with this approach - only writing code to do exactly what a form does is discouraged. This is especially true with GET requests that are sent as a result of user input. For example, we have a page that lists objects in a table. We also have a select element that filters the list by some field. When that selector changes, the page is refreshed with the new GET parameters. The request is triggered by binding the selector change event to a Javascript function. The function then builds the URI and sends the request. Alternatively, the select element is part of a form that stores the URI, and the method. Our function would simply submit the form.

Using forms as they were intended is good web-programming practice. They are perfect for collecting small pieces of information and map well to abstractions that exist on a server. Forms store HTTP methods, action URIs, group input elements, and assemble HTTP requests so that you don't have to. They exist for a practical purpose. Use them to your advantage and you'll be surprised at how much code you don't have to write.

Passing URIs

Uniform Resource Identifiers (URIs) are what enable us to find things on the web. These things refer to a resource, uniformly identified by a string. A resource is any digital media that has been made available on the Internet. At a higher level, search engines are what allow us to find resources that live somewhere in the web. Without a URI, there would be nothing useful for the search engines to display in their results. Additionally, it would be impossible for search engines to crawl websites without theURIs that make up the structure of the site.

APIs can be built with a set of URIs as well. These URI-centric APIs are sometimes referred to as RESTful APIs. RESTful APIs have a close association with the HTTP protocol. Because of this we can pass parameters to resources through GET or POST requests made by a client. But these are often primitive types that can be represented as a string. For instance, if I'm using someAPI to update my user profile, a numeric user ID might be a POST parameter I need to send. This is necessary so the application knows which user to update. But what if I were able to pass an entire URI as the identifier for the resource I want to update? Does that even make sense? Well, lets first think about how applications identify resources internally.

The most common way for a web application to identify a resource internally is by a primary key in a database table. This key is typically an integer value that is automatically incremented every time a new record is inserted. This key is unique for each record in that table. This makes the primary key of a database table an ideal candidate for using as part of a URI. You'll often see integers as part of a URI, for instance "/user/4352/". There is a good chance that the number is a unique primary key in the database. This uniqueness maps well toURIs because every URI should be unique in one way or another.

One potential problem with using primary database keys in URIs is that different records in different database tables may share the same key. This doesn't necessarily weaken the uniqueness of the URI because it is still referring to a different type of resource. Consider two database records in two different database tables. These records both have the same integer primary key value, 5. TheURIs for these two resources are still unique because they are referring to two entirely different concepts. So the first URI might be "/user/5/" and the second URI might be "/group/5/". But what if you don't care about the resource type?

A canonical URI might be composed of a UUID instead of the primary key of a database table. UUIDs themselves are unique and may refer to any resource. That is, a UUID doesn't need a context in order to be unique. If our above two URIs were to use UUIDs, they might look something like "/user/cadb1d94-5305-11df-98a5-001a929face2/" and "/group/d8eee85c-5305-11df-8d08-001a929face2". As you can see, we really don't need "user" or "group" as part of the URI. We could refer to a resource canonically with something like "/data/cadb1d94-5305-11df-98a5-001a929face2/". This could refer to either a user or a group. This can be both flexible and dangerous.

Having a canonical URI based on a UUID can be flexible because the client requesting the resource doesn't need to know the context. The client might have acquired this URI and has no idea what exactly it is a representation of. Even with just theUUID , the client now has the ability to discover the type of resource this URI is pointing to based on the data it returns. This can also be dangerous for exactly the same reason. If a client doesn't know how to handle the data returned by a canonical URI, chances of the the client malfunctioning are higher. The data representations returned by URI resources are a lot like interfaces; different data types can still provide the same interfaces by having a subset of common keys.

The location part of a URI might also be useful for passing as parameters to web applications. Until now, I've only been talking about the path in which the server must look for the resource. But this is making the assumption that the resource in question still lives on the same server. By only passing primary database keys or UUIDs as parameters, we leave the location aspect out of the equation. It might be more flexible to pass a full URI as a parameter. Even if the URI location is pointing to the same location in which the request arrived. It really isn't a big deal for a server to check when processing requests. If the resource lives here, we simplydissect the URI and process as usual. Otherwise, we forward the request to the appropriate location. I realize I'm oversimplifying this a little too much but the idea is to think about passing wholeURIs as parameters, not so much the details of how we should go about implementing a full-fledged distributed computing platform.

So remember that canonical URIs composed of UUIDs can be useful when treated with care. If context is important, don't use them. Stick to using primary database keys if it helps keep things simple. Try experimenting with a simple web application that will accept a full URI instead of an ID string of some sort. A flexible solution might even accept either or.

Hashing URIs

Clients of URIs often need to specify some sort of client parameter that is returned as part of the HTTP response. This parameter is intended to preserve the application state by encoding it into the URI.

A common practice is instead of passing the client state as a GET HTTP parameter, to provide the client state as a hashed URI segment. This segment would represent the client state. It is a preferable URI practice because the application state is still encoded as part of the resource URI but doesn't need to be sent to the server.

One aspect of this practice to be weary of is mistaking application state for resources. If the intent is to use the application state as the hashed URI segment, only use it for application states and not data resources.

Consistent HTTP Codes

Does it pay in terms of security to have consistent HTTP error codes returned to the requesting clients? Does it make more sense to return an HTTP 404 when something isn't found and an HTTP 403 when something is forbidden? Or does it make more sense to always use either 404 or 403 exclusively?

Simple Zope HTTP Server

I've seen mention of every conceivable Python web framework as of late. Mostly performance comparisons. One I don't see that often is Zope. It is a mature framework that does everything under the sun. I think it isn't mentioned that often by developers because of the lack of documentation.

The good news is that Zope code is easy to read and understand. I decided to experiment and create a really simple HTTP server with Zope HTTPServer. I find the task-oriented view of each HTTP request both interesting and useful. Here is what I came up with:

import asyncore
from zope.server.http.httpserver import HTTPServer
from zope.server.http.httptask import HTTPTask
from zope.server.taskthreads import ThreadedTaskDispatcher

class MyServer(HTTPServer):
  
   def executeRequest(self, task):
      
       task.start()
      
       self.log_info("Starting request task.")

       task.response_headers['Content-Type'] = 'text/plain'
       task.response_headers['Content-Length'] = 5

       task.write(task.request_data.command)
       task.write(" ")
       task.write(task.request_data.uri)   

       task.finish()

       self.log_info("Rquest task finished.")
      
if __name__ == "__main__":
  
   dispatcher = ThreadedTaskDispatcher()
   dispatcher.setThreadCount(10)
  
   MyServer('', 8084, task_dispatcher = dispatcher)

   try:       
       while True:
           asyncore.poll(5)
   except KeyboardInterrupt:
       dispatcher.shutdown()

Django HTTP Response

The three main elements of an HTTP response are the body, the headers, and possibly cookie values. The Django HttpResponse object has support for all three, which is a requirement considering Django is a web application framework.

Whats cool about this Django class is how easy it is to test if a header exists. This is done by implementing the __contains__ method. It seems trivial when building classes, to add something like this in later, but it is easy to just forget about it.

It makes interacting with HttpResponse instances that much easier because there is one less step. Instead of doing if "key" in response_instance.headers().keys(), or something like that, Django HttpResponse instance allow developers to do if "key" in response_instance. The reason this functionality is implemented for headers and not for cookies is because headers are used more frequently than cookies in mos circumstances.

Publishing CSV

I recently came across this Python recipe for serving CSV data from a CherryPy web controller. The recipe itself is fairly straightforward. It is a simple decorator that can be applied to a method which is also a CherryPy controller. The decorator will take the returned list data and transform it into a CSV format before returning the response.

You sometimes have to wonder about the CSV format, it is quite non-descriptive for humans to read. Fortunately, that isn't the reason it was created. The CSV format is easy for software to understand without requiring a lot of heavy lifting. This is why it is still widely used. Virtually any data can be represented with it. But what happens when something goes wrong? Typically, if an application complains about some kind of data it is trying to read, a developer of some sort needs to take a look at it. Good luck with trying to diagnose malformed CSV data, especially a large chunk of it.

The fact of the matter is, the CSV format is still a requirement in many contexts. Especially in terms of providing the data as opposed to consuming it. It is most likely going to be easier to provide a CSV format to a client than it is to say the client needs to be altered to support SOAP messages. If that were the case, there would certainly be many upset people using your service, or, nobody using your service at all.

As the recipe shows, transforming primitive data structures into CSV format isn't that difficult. Especially with high level languages that have a nice CSV library like Python does. The HTTP protocol is more than likely going to be the protocol of choice. In this case, the important HTTP headers to set are Content-Type and Content-Length.

Odds are, the CSV format isn't the format of choice for most web application designs. That is, the developers building APIs aren't going to use CSV. They are probably going to use something more verbose like JSON or some XML variant. This just makes the clients that need to interact with this data much easier. Also, they much more common. Chances are that CSV support will be an afterthought. This isn't an uncommon change request for any web application project though. No one building a web application can expect the initial chosen format to suffice throughout the lifetime of the application. What this means is that exposing the data to the web is the easy part. It is coming up with a sustainable design that is the challenge.

Many web application frameworks provide support for multiple response formats. And, if CSV isn't one of them, chances are that there is a plugin that will do it.

So, even if the framework doesn't support CSV data transformation functionality, as is the case with CherryPy, the Python CSV functionality will do just fine on its own. That is, the controller can be extended with CSV capabilities as is the case in the recipe. Below is an illustration of a web controller with CSV capabilities.



Here the Controller class inherits CSV capabilities. The DomainObject class, which for our purposes here could be anything that is part of the application domain and needs to be exposed to the web. With this design, as is the case with CSV functionality offered by the web framework of choice, the responsibility of the CSV transformation falls outside of the domain entirely.

Below is an alternate design that give CSV data transformation capabilities directly to the DomainObject class.



So which design is more realistic? Probably the former simply because the use case for CSV data transformation outside of the web controller context isn't all that common. But does the latter design even make sense? Are we giving too much responsibility to a business logic class? Well, I would argue that it depends on how portable the domain design classes need to be. Sure, with the CSV capabilities being given to a domain class, we are violating a separation of concerns principle, albeit, only slightly. It isn't as though the class itself is being altered to support a specific data format. If this is a valid trade-off in a given context, I would strongly consider keeping data format transformation out of the web controllers.

HTTP Response Schema

The HTTP protocol is by far the most prominent protocol on the web today. Most people browsing the web aren't even aware of its existence. They just see a strange "http://" string in front of where they want to go. The great part is that most users of the web don't need to know what HTTP is or how it works. The browser and the web server have a firm understanding of how to use the protocol.

Web clients, including the browser need to understand how to handle HTTP responses. Contained within each HTTP response is a numeric code indicating the status of the request. Although HTTP is a stateless protocol, the client needs to know what happened with each request made to the server. One may consider the response code to be a meta state. The response code can still support statelessness on the server.

Each HTTP response code has a distinct explanation for why the requested responded the way it did. The client can also interpret a more general category, or nature of the response because of the ranges in which each specific code falls under. Here is a brief idea of what the ranges mean:

• 200 - Success.
• 300 - Client action required.
• 400 - Client error.
• 500 - Server error.

In an object-oriented design, these ranges of HTTP response codes can be broadly generalized. This helps to conceptualize how the client can handle several specific response codes in the same way as is illustrated below.

Python BaseHTTPRequestHandler Request Parsing

When a client sends an HTTP request to a server, that request is sent as a raw string of HTTP request data. Each component of the HTTP request, while in the transit state, is still considered a single monolithic component. This raw HTTP request must be parsed. It is the parsing action that transforms this raw HTTP request string into a coherent set of HTTP request components. Python provides a module in the standard distribution that provides basic HTTP request parsing capabilities. The class used for this is called BaseHTTPRequestHandler. As the name suggests, it is the most basic of all HTTP request handling capabilities provided by Python and must be sub-classed to carry-out any meaningful functionality. What we are interested in here isn't such much the extending of the BaseHTTPRequestHandler class as much as how the HTTP request parsing of the class is broken down. This fundamental action of web programming in Python is rarely seen because many web application frameworks do this for us. Even when using the fundamental BaseHTTPRequestHandler class, developers need not concern themselves with parsing individual HTTP requests. This is also taken care of by the class as a step in the request handling activity. It is useful, however, to see how the most fundamental HTTP request parsing functionality works and if there are any weaknesses with the approach.

The following is an illustration of how an HTTP request is parsed by the BaseHTTPRequestHandler class. Keep in mind, this is only a high-level view of what actually happens. There are many smaller steps kept out of the illustration for simplicity.



One question to ask here is if any of these actions can be parallelized. One candidate might be the initialization of the HTTP headers. The reason being, there is no prerequisite input to the header initialization that isn't available before the actions previous to the header initialization are executed. Suppose this action were carried out in a new thread, would there really be any significant performance gain. Unlikely. Is it an alternative design worth exploring? I would think so. See below for an illustration of the header initialization executing in parallel with the other HTTP parsing actions.

Initializing the CherryPy server object

The CherryPy Python web application framework contains several abstractions related to an HTTP server. One of which is the Server class defined in _cpserver.py. In fact, the top-level server instance of the CherryPy package is an instance of this class. The Server class inherits from the ServerAdapter class which is defined in servers.py. Interestingly, the class serves as a wrapper, for other HTTP server classes. The constructor of ServerAdapter will accept a both a bus and a httpserver parameter. The bus parameter is a CherryPy website process bus as described here. The httpserver parameter can be any instance that implements the CherryPy server interface. Although this interface isn't formally defined, it can be easily inferred by looking at the CherryPy code.

So we have a Server class that inherits from ServerAdapter. The idea here is that many server instances may be started on a single CherryPy website process bus. One question I thought to ask was "if Server inherits from ServerAdapter and ServerAdapter is expecting a httpserver attribute, how do we specify this if the ServerAdapter constructor isn't invoked with this parameter?" In other words, the ServerAdapter isn't given a parameter that it needs by the Sever class.

It turns out that the start() method is overridden by Server. This method will then ensure that a httpserver object exists by invoking Server.httpserver_from_self(). Developers can even specify an httpserver object after the Server instance has been created by setting the Server.instance attribute with the server object we would like to use. This is the first attribute checked by the Server.httpserver_from_self() method. The Server.instance attribute defaults to None, and if this is still the case once Server.httpserver_from_self() is invoked, it will simply create and return a new CPWSGIServer instance to use.

Now we can feel safe, knowing that there will always be an object available for ServerAdapter to manipulate. We left off at the Server.start() method creating the httpserver attribute. Once this is accomplished, the ServerAdapter.start() method is invoked because it is now safe to do so. One observation about the implementation; I'm not convinced that calling the ServerAdapter.start() method with an instance of self is the best design. This is the only way that Server instances can invoke behaviour on the ServerAdapter instance, even though in theory it is the same instance by inheritance. At the same time, we wouldn't be able to override the method and then call the inherited method if we were to call ServerAdaptor.__init__() from Server.__init__(). The alternative would be to have unique method names between the two classes. Then again, this might end up taking away from the design quality of ServerAdapter. So the question is, which class is more important in terms of design. Just something to think about, not that the current implementation is defective by any means. CherryPy is probably one of the more stable Python packages in existence.

Django WSGI handlers

WSGI handlers in the Django Python web application framework are used to provide the WSGI Python interface. This means that the Django framework can easily support middleware projects that manipulate the response sent back to the client. The code for the WSGIHandler class and the supporting WSGIRequest class can be found in the wsgi.py Python module. Here is a simple illustration of what the module provides.



Just as a brief overview of what this module contains, I'll simply describe each of the four classes shown above. First, the HttpRequest class provide the basic HTTP request attributes and methods. The BaseHandler class provides attributes and methods common to all handlers. The WSGIRequest class is a specialized HTTP request. The WSGIHandler class is a specialized handler.

The WSGIHandler class depends on the WSGIRequest class because it instantiates the request instances. This is done by setting the WSGIRequest class as the request_class attribute. There are two aspects of the WSGIHandler class that I find interesting.

First, this class isn't instantiated. There is no __init__() method defined and all attributes are class attributes. However, the __call__() method is defined so this class can actually be invoked is if we were creating an instance only we get a response object returned instead. At the beginning of the __call__() method, the WSGIHandler class acquires a threading lock. This lock is than released after the middleware has been loaded.

Second, each middleware method is than cumulatively invoked on the response so as to filter it. Very cool!