An idea for loading Pylons middleware

Any given web application framework, Python based or otherwise, must support the concept of middleware. Some frameworks have better support for middleware than others but they all support it, even if the support is rigid and implicit. The reason to associate the concept of middleware with web application frameworks is because the middleware is code that runs in between the request and the response. In most cases, this would be an HTTP request-response pair but the same concept applies to other protocols. In the Python world of web applications, there isn't much support for this idea. Most Python web application frameworks support the concept of extension or plugins. These components can achieve the same effect as middleware components but are more geared toward domain-specific problems. Middleware is supposed to be more solution generic in that it will affect the framework as a whole. The best support of this concept is in the Pylons Python web application framework.

The middleware concept plays a big role in Pylons. In fact, several of the core Pylons components act as middleware. For instance, the routing of an HTTP request, the HTTP session, Caching, are all middleware components in Pylons. These components, as well as other middleware components, may affect the state of the core Pylons WSGI application object. Some of these middleware components will surely affect the state of the HTTP response that is sent back to the client. The middleware architecture in web application frameworks is a wise design decision because it helps reduce the complexity of the code found in the core application object. This happens by providing a better distribution of responsibilities. If all this code were located in the core application object, it would be quite messy and hard to maintain. The middleware construct also offers developers more extensibility opportunities. The entire framework functionality may be altered on account of some middleware component. Below is a simple illustration of how the Pylons web application framework interacts with components.



When a new Pylons skeleton project is generated, a middleware Python configuration script is built and placed in the project directory. It is this generated Python script that is responsible for loading the middleware components to be used in the application. This is an important task t perform when the application is started because the core application object needs its' core middleware components. Without them, much crucial functionality will be missing. However, a nice experiment to try out with the Pylons framework would be to alter this generated middleware loading code. Rather than have static Python code which directly instantiates these components, they could be loaded by iterating through a set of entry points. This way, Pylons middleware components could be discovered and loaded as they are installed on the system. This, however, could be a more challenging feat in reality. The default Pylons middleware components would need to be entry points themselves. These components could always have entry points added to them. At the very least, it would be an idea worth exploring.

ECP and the future extension module architecture

Developers that have been using the Enomaly Elastic Computing Platform over the past year, myself included, have encountered some bottlenecks in the ECP extension module architecture. These aren't show-stoppers in every case but sometimes, they are. It is mostly an issue of architectural design such as "what is the rationale behind this API method?" and "if I build it this way, what is the impact of change in other areas going to bring?" Once we took a step back to think about such questions, we came to the conclusion that these questions should be apparent to any developer using the platform or at least easily answered by Enomaly. Right now, much of the ECP extension module framework isn't apparent how to use and we even have a hard time explaining it.

So, this has led the ECP development team to address some of the issues highlighted here.

One of the first major problems is a problem of uniformity and consistency among the core extension modules that ship with ECP. These extension modules aren't exactly consistent with one another. Some modules will use sections of the API as intended while others will use different sections and others, still, don't use the API at all. In fact, some extension module logic in ECP is coded directly in the core system. This doesn't necessarily cause any harm to anyone who wants to install the base system because these modules are "part" of the base system. They are simply constructed as extension modules. Earlier on in ECP's lifespan, we needed to construct an extension module API and these core modules were the perfect way to test out our ideas. So, either way, these core extension modules could have been built in to the core code base. But, it would be nice if they weren't so irreplaceable.

The first step is to introduce a new level of consistency among the extension module that are distributed with ECP. If nothing else, they can serve as useful examples for extension module developers.

The next major defective area within the ECP extension module API is the API itself. Or, rather, lack thereof. What I mean here is that there are plenty of smaller areas of ECP that should be extensible but aren't. For example, if there is some thing small in the ECP front-end GUI that a developer wants to extend, they must replace the entire template, duplicating many already existing elements. This means that many of the core elements, including GUI widgets, would need to become part of a bigger extension module framework. That is, we would need to move them to extension modules. And that is fine with me. A smaller core is easier to maintain and thus more stable.

There have already been some changes introduced to the ECP extension module framework in the past year. We identified the need for extension modules to store their own static data such as javascript and CSS files. To address this, we introduced methods to register these static components.

There is still a decent amount of work to do in order to realize these changes. They have been identified and that is a very good thing. Also, this is by no means a closed list of issues with the extension module framework that need fixing. This is a good starting point. I've already begun fixing the consistency problem with the core extension modules.

Trac RecaptchaRegisterPlugin problems

The RecaptchaRegisterPlugin Trac extension has a few minor defects I noticed while testing it out for production use. The first problem, the captcha input would disappear if any other fields were invalid. The second problem, if the captcha field was invalid, any existing for data was lost. Here is my updated version of process_request() that addresses both issues.

#RecaptchaRegister trac plugin fix.

# IRequestHandler methods
def process_request(self, req):
  self.check_config()
  action = req.args.get('action')

  if req.method == 'POST' and action == 'create':
      response = captcha.submit(
          req.args['recaptcha_challenge_field'],
          req.args['recaptcha_response_field'],
          self.private_key,
          req.remote_addr,
          )
      if not response.is_valid:
          data = {'acctmgr' : { 'username' : req.args['user'],
                                'name' : req.args['name'],
                                'email' : req.args['email'],
                              },
          }
          data['registration_error'] = 'Captcha incorrect. Please try again.'
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, None
      else:
          ret = super(RecaptchaRegistrationModule, self).process_request(req)
          h, data, n = ret
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, n
  else:
      ret = super(RecaptchaRegistrationModule, self).process_request(req)
      h, data, n = ret
      data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
      data['recaptcha_theme'] = self.theme
      return "recaptcharegister.html", data, n

Trouble extending Trac navigation.

I'm in the process of building a new Trac site. I wanted to add new menu items to the main navigation. Hiding or rearranging the default menu items in Trac 0.11 is quite straightforward. It can all be done in the Trac configuration. However, new menu items need to be part of a component. Hence, the need for the NavAdd plugin.

It looks like the plugin has not yet been updated to fit the 0.11 plugin architecture, although it does work with 0.11. The Trac plugin API changes weren't too drastic. I did notice some strange behaviour though. It turns out that any menu items added with the NavAdd plugin can not be considered active.

What? No active menu items? Well, it isn't really the NavAdd plugins' fault. It turns out, that in order to have an active menu item, the current request needs to be handled by the same component that produces the menu item. I discovered this by looking at the timeline Trac component. This component actually implements the IRequestHandler interface. This is necessary because the timeline component handles requests to the /timeline URI. The menu item produced by this component becomes active anytime the /timeline URI is visited.

So, this design works well for components that have URIs. But what if I want to add menu items that point to wiki pages? And what if I want to have my menu item become active when visiting those pages? There is currently no way to do this. My suggestion would be that the INavigationContributor interface adds a new uri field to be returned from get_navigation_items(). When page corresponding to this uri becomes active, so does the custom menu item.

Enomaly ECP update

There is some major refactoring happening with the core extension modules that ship with Enomaly ECP. In the current stable version (2.1) of ECP, there is some javascript and other static resources that are required by some of these core extension modules. In the case of javascript, the code is simply defined in the core jquery.enomalism2.js javascript file.

One of the limitations imposed on ECP 2.1 is the lack of dynamic extension module resource integration. Any external extension module will not be able to alter the ECP interface to the extent that may be desired. The good news is that this limitation has been realized and is being addressed. In ECP 2.2, much more static resource flexibility will be offered to extension modules.

Since the core extension modules are no different than external extension modules that are not shipped with ECP, the initial refactoring will happen in these modules. This will at least provide a good starting point on how to take advantage of this new core functionality.

Everything should be iterative.

In software, everything should be iterative. Not only that, but the size of the set being iterated over should never be statically set in code. Functions and methods should never return a multiplicity of one. The exception in this case would be if the function simply performs some action and we are not interested in the result.

This is the backbone of extensibility in software development. As soon as you impose multiplicity limits on objects, you are essentially eliminating any chance for extension in that component. If you have some method of some object that returns another software object that is not iterated-over, the chance for that method to return a dynamically determined number of is gone.

What if the behavior caused by the result of some method invocation should differ depending on the length of the number of objects returned? This is seldom the case and you can always have alternative flows of control depending on the length of the result. This extra bit of code in extreme cases is a small price to pay for the extensibility this patter offers.