Widgets And Frameworks

Web frameworks are important because today, the target user environment is the web.  Why is this?  Because the web means anyone can use your application.  There are no barriers to installing the software and no restrictions on the hardware.  All the user needs is a decent web browser.

The web application framework itself is relevant because it enables developers to write solid software by reusing common components.  Common patterns used in all web applications.  Without these tools supplied by the framework, nothing would ever get off the ground — a diffuse time to market for smaller development teams.

What patterns are used in web frameworks?  Database connectivity, URL mapping, authentication.  The list goes on, each framework providing their own flavor of these major components.  Each is considered a pattern because any web application needs to implement it.  This quickly burdens the developer.  The role of a web framework is to parameterize these patterns and make them accessible.

What role do widgets play in web application frameworks?  By widgets, I mean standalone Javascript toolkits that have a selection of cool user interface stuff.  This includes jQuery UI, Dojo, and ExtJS.  Making these Javascript widget tool kits — which are a framework on their own — integrate seamlessly into our web application frameworks is disjointed to say the least.

Not an easy fit
Web application frameworks don't give us all the front-end browser code necessary for a finished product.  If it were a matter of delivering only HTML and CSS to the browser, we'd be all set.  This isn't the case unfortunately.  Not if there is any kind of user interaction involved — which is almost certainly the case.  For the kind of experiences users have come to expect, developers need to employ Javascript or Flash technologies.

In theory, building these rich user experiences is fairly straightforward — alter the HTML templates and serve the additional resources — Javascript, Flash, etc.  This can be achieved without any headaches — granted the project is a small one.  There many resources to worry about, not as big a user base compared to something enterprise-grade.  Every aspect of the software is scaled back slightly — and so the cause for error is scaled back accordingly.  For example, having a very large Javascript file or dozens of smaller Javascript files can be hard to maintain.  Not on their own — but because Javascript is different — it depends heavily on the DOM.

So as far as the development process goes, we need sound approach to ensuring there is some degree of coupling between individual HTML templates and their corresponding Javascript modules.  The days of implementing a monolithic Javascript file are quickly leaving us behind.  Especially if given the challenges of managing external widget library dependencies.  Imagine an application with a hundred or more HTML templates — I can't think of an optimal method where a single Javascript module suffices.

Technically speaking, uniting widget frameworks and web application frameworks wouldn't be overly strenuous.  All this entails, and this is generally how they're put together now, is serving new resources and having your HTML content ask for those resources.  The place where widgets don't fit seamlessly into the web application framework is in the back-end the web application.  How does a given view or controller (whatever the framework terminology alludes) know about which widgets are used?  Are these widget configurations persistent?  For that matter, are they configurable at all from the web framework?  For me this is a problem — widgets are a big part of the overall picture — web application frameworks should accommodate widget frameworks.

Frameworks and forms
A crucial feature of any software with a web user interface is forms.  Forms collect user feedback.  They can be simple — a single input field and a submit button.  They can be complex — a registration form that nobody in their right mind would fill out.  Web application frameworks do a pretty good job with tools that both render and validate forms.

There is good reason for.  Web frameworks define the database schema for the application.  The database is populated by user input.  Forms.  There is an obvious connection between the two concepts — this is why frameworks take great care to supply the database with what it expects.

Take Django for example.  The Django framework has a forms module that'll assist with defining forms, the validation, and rendering the form widgets for the browser.  Not unlike how Flask uses WTForms to validate and present forms.  The Pylons project also provides developers with form processing capabilities — they require a little more effort than the other Python frameworks.

What really stands out is the connection forms and their individual input widgets have the framework itself.  They're an essential ingredient to the application — a formative piece of the database.  There is a mapping between the database schema and what is presented to the end user so they can populate it.  This fits perfectly with the ideal of the web application framework — to take a pattern and expose it as a cohesive API for the developer.  This concept is further exemplified by Django where we're able to point the form subsystem to a database model and let it figure out how the form should be rendered and validated.  This isn't so easy with more general widgets that aren't necessarily used for collecting input and storing it.

Widgets and data
If we're able to utilize the framework form processing tools to store data, how does the framework help us present data to the user?  Well, since a web application framework is only responsible for delivering the content to be rendered by the browser, all we've got is templates.  These assist in generating the markup necessary for displaying fancy Javascript widgets.

For instance, a common user interface pattern is to group logical categories using tabs.  Tabs are visually appealing and allow for intuitive navigation.  This is a top-level layout widget — meant to replace a typical sidebar navigation item.  Or consider a smaller widget — a progress bar.  This type of widget, unlike tabs, isn't a layout container.  It displays something specific about the application data.

What both layout container widgets and smaller application data widgets have in common is that they both depend on specific application details that reside on the server, within the framework.  The tabs widget needs to know what specific tabs to display and what content within each tab.  The progress bar widget probably depends on a computed number — derived from user data from the database.  So what is the preferred approach to displaying both widget types in any given application?

Navigational constructs are typically done in the template.  Maybe there is a nested div and a tag structure the Javascript widget is expecting — like the jQuery UI tabs widget.  However, the developer still needs to instantiate their widget when the page loads.  This means that the Javascript code needs to know about the template and what the rendered HTML output will look like.  This is somewhat error-prone as the developer will need to maintain dependencies between the Javascript widgets and the HTML structure they're expecting.

More challenging is the application user data presented by widgets — like a progress bar.  The progress bar needs the value to display — but where does it come from?  This might be a JSON response, returned from one of the application controllers.  Again, as with defining layout widgets, the problem lies in maintaining the connection between what the Javascript widget is expecting to see and what the framework delivers.  The problem isn't so much technical — it is more of a design problem.  One that leads to a loosely coupled system.  Too loose.

Solutions and challenges
Web frameworks already do a great job of taking common patterns on the server and making tools developers can use to efficiently develop code.  Might it be that web application frameworks have no business managing widgets?  Should this task be left to the templates and other static files?  I think there are patterns we're missing here that can be taken advantage of — it's still only an emergent thought.

One benefit to having the framework manage the connection between application data and widgets is the potential to be widget framework agnostic.  There are a lot of widget tool kits out there and a lot of similarities between them.  The commonalities across all toolkits — these are the interesting widgets.  The basic widgets that all web applications use in one form or another could potentially be integrated into web frameworks.

What might we do with the layout widgets to help standardize and improve the coupling between Javascript and framework data?  After all, layouts, if they can be standardized in some fashion, are a valuable asset.  You'll generally have a web application framework of choice and you'll almost certainly be creating more than one application.

I think layout widgets — stuff specific to the user interface and not necessarily what the application does — will be easier for frameworks to implement.  If the framework is able to introduce the concept of a widget, maybe storing it's configuration, then properly outputting the required markup and the required Javascript isn't a stretch.  For example, the configuration for a tabs widget would store what tabs are actually rendered, the contents of each tab, and should produce the necessary Javascript to instantiate it.

Widgets specific to application data — like a grid — aren't as easy.  This is because they're more toward what the application does.  Another challenge is that widgets generally depend on some data source like JSON that is generated from the user data.  This isn't so easy to stardardize on.  The framework would have to know about the data schema in addition to the visual configuration.  This means the application would have to store configuration that dictates what the widget will look like, where to get the data it needs, and how that data should be interpreted for presentation.  Not impossible, just more challenging.

Perhaps a good start to making web application frameworks a little more aware of patterns in how browser widgets interact with the framework itself is to discover patterns you're using.  How would you benefit if you're able to store some configuration that defines how and what the Javascript widgets do?  Every application is different — and so every combination of widgets used brings a new set of challenges.  This is how we make progress and is how the current features of frameworks came into being — by experimentation and pattern discovery.

Applying Django Middleware

The Django Python web application framework supports the notion of middleware. What exactly is middleware? In the Django context, middleware are Python packages that do not explicitly belong to any particular Django application. Nor do these middleware packages belong to the Django package although Django does ship with some common middleware. The middleware Python packages that are independently installed sit in the middle.

So whats the point of separately installing Django components if they aren't part of either Django or the applications that use them? Well, since middleware components can be enabled in a Django application's settings, these middleware components may be shared between different applications that use the same Django installation. This keeps the Django core small.

The actual middleware components themselves are composed of classes. These classes must implicitly provide a Django middleware method. These methods can be one of process_request(), process_view(), process_response(), and process_exception().

The middleware components that an application wants to use are invoked automatically by the Django framework. All the application needs to do is enable the desired middleware components. The Django framework uses the BaseHandler class and the WSGIHandler class to invoke the middleware behavior. These classes are illustrated below.



Below is an illustration showing how the to classes, BaseHandler and WSGIHandler, collaborate to process all enabled middleware methods.

Combining Multiprocessing And Threading

In Python, there are two ways to achieve concurrency within a given application; multiprocessing and threading. Concurrency, whether in a Python application, or an application written in another language, often coincides with events taking place. These events can be written directly in code much more effectively when using an event framework. The basic need that the developer using this framework has is the ability to publish events. In turn, things happen in response to those events. Now, what the developer most likely isn't concerned with is the concurrency semantics involved with these event handlers. The circuits Python event framework will take care of this for the developer. What is interesting is how the framework manages the concurrency method used; multiprocessing or threading.

With the multiprocessing approach, a new system process is created for each logical thread of control. This is beneficial on systems with more than one processor because the Python global interpreter lock isn't a concern. This gives the application potential to achieve true concurrency. With the threading approach, a new system thread, otherwise known as a lightweight process is created for each logical thread of control. Applications using this approach means that the Python global interpreter lock is a factor. On systems with more than one processor, true concurrency is not possible within the application itself. The good news is that both approaches can potentially be used inside a given application. There are two independent Python modules that exist for each method. The abstractions inside of each of these modules share nearly identical interfaces.

The circuits Python event framework uses an approach that will use either the multiprocessing module or the threading module. The circuits framework will attempt to use the multiprocessing module method to concurrency in preference to the threading module. The approach to importing the required modules and defining the concurrency abstraction is illustrated below.



As you can see, the core Process abstraction within circuits is declared based on what modules exist on the system. If multiprocessing is available, it is used. Otherwise, the threading module is used. The only downfall to this approach is that as long as the multiprocessing module is available, threads cannot be used. Threads may be preferable to processes in certain situations.

The Cherrypy Web Site Process Bus

The Cherrypy web application framework defines several process utilities for communication across several website components. One such utility is the Web Site Process Bus. This bus is implemented as a class called Bus. The bus implements a micro publish subscribe event system along with some default subscribers to carry-out the default server behaviour. An interesting note about this publish-subscribe framework; every subscriber is guaranteed to execute, even if previous subscribers raise an exception.

The Bus class defines several methods that will publish basic server state change events such as starting and stopping. Here is an illustration of what the Bus class looks like.



Here, we can see that the Bus class depends on the _StateEnum enumeration. This enumeration holds all possible states a Cherrypy server may be in at any given time.

The subscribe() method will subscribe the specified callback to the specified channel. There is also an optional priority parameter that may be set.

The unsubscribe() method will unsubscribe the specified callback from the specified channel.

The publish() method will publish the specified parameters to the specified channel.

The _clean_exit() method will check if the bus is in an EXITING state before exiting the bus.

The start() method will publish an event on the start channel and put the bus in a STARTING and START state.

The exit() method will publish an event on the exit channel and put the bus in a EXITING state.

The restart() method will set the execv attribute to True and invoke the exit() method.

The graceful() method will publish an event on the graceful channel.

The block() method will block execution in the current thread of control until the bus is in the EXITING state. Once reached, all threads are then joined so the current thread will wait until they have terminated. Finally, if the execv attribute is set to True, _do_execv() is invoked.

The wait() method will block execution until the bus is in the specified state.

The _do_execv() method will attempt to restart the process in which cherrypy is running.

The stop() method will publish an event on the stop channel and put the bus in a STOPPING and STOPPED state.

The start_with_callback() method will start a new thread for the specified callback and then invoke the start() method.

I think having a platform independent bus like this in place is a very smart design decision. It is very useful and helpful for web applications to be able to reason about the state of the process in which it is running. This is especially true in today's distributed environments.