The Plugin Ecosystem

The idea of monolithic software is a rarity for most folks. Even an operating system, which is itself a big monolithic beast as far as most users and developers are concerned, has it's own plugin ecosystem. The operating system does a lot of work, but the vast majority of that work is in support of plugins, or applications, that run on top of it. Like it or not, its these plugins that make the software on which they run useful. They support the idea of an ecosystem, new plugins evolving from old ones, old and stable plugins becoming dependencies of other plugins — once the plugin development community forms, it seems there is no stopping it. The ecosystem that plugin development sets forth opens up the door to an interesting development culture, one that seems to have a strong experimental, and thus, strong educational feel to it.

How Gaphor Checks Association Ends.

With the Unified Modeling Language, comes a well defined meta model. It is this meta model that defines the semantics of the modeling elements available within the UML. An important feature of this meta model is consistency enforcement. The semantic rules defined in the UML meta model not only supply the UML modeling element but also govern how those elements may be used in relation to one another. If these rules, specified by the meta model could be verified for any given UML model, it could provide a means of ensuring a consistent model. As an example UML element that has the potential to be validated for consistency, consider the two ends of an association. One end could be a subset of the opposite end. There are a couple ways that this arrangement could go wrong. There could be missing names or there could multiplicity inconsistencies. The Gaphor UML modeling tool provides a checkmetamodel plugin that is capable of validating the association ends, as well as other potential meta model issues. This plugin can be viewed by selecting "Tools", "Check UML model". Illustrated below is the dialog that will be displayed. Only errors will be displayed here so if there is no content, the current Gaphor UML model is consistent with the Gaphor UML meta model.



When the checkmetamodel plugin validates the UML model, there are three functions involved. These functions are check_associations(), check_association_ends(), and check_association_end_subsets(). The check_association_end_subsets() function is important when validating association ends. It is the only validation that actually takes place on associations. The check_associations() and check_association_ends() functions exist for infrastructure purposes. If new functionality were to be added to Gaphor association or Gaphor association end checking, it would be placed in one of these functions. As the same suggests, the check_association_end_subsets() function will make sure that any association ends that are subsets of the opposite association end are consistent with one another. There are two basic tests to make this happen. First, the plugin will make sure that a given association end that is a subset of the opposite association end, contains names that actually exist in the opposite set. There can't be a subset property which contains elements that do not exist in the referenced set. Next, all multiplicity upper bounds in the subset need to be consistent with the upper bounds of the opposite set. This will ensure a multiplicity consistency amongst association end subsets.

With Gaphor, these meta model consistency checks are especially important because the meta model is loaded into Gaphor as any other UML model would be. If the core Gaphor UML meta model fails, there isn't much hope for any models created by users. The checkmetamodel plugin also offers potential for adding additional checking functionality.

The Trac component loader

All modern web application frameworks need replaceable application components. Reasons for this requirement are plenty. Some applications will share common functionality with other applications such as identity management. However, having the ability to extend this functionality or replace it entirely is absolutely crucial. Technology requirements change too fast to assume that a single implementation of some feature will ever be sufficient for any significant length of time. Moreover, tools that are better for the job that your component currently does will emerge and developers need a way to exploit the benefit of these tools without having to hack the core system. Trac is a Python web-framework in it's own right. That is, it implements several framework capabilities found in other frameworks such as Django and TurboGears. Trac is highly specialized as a project management system. So, you wouldn't want go use Trac as some generalized web framework for some other domain. Project management for software projects is such a huge domain by itself that it makes sense to have a web framework centered around it. Trac defines it's own component system that allows developers to create new components that build on existing Trac functionality or replace it entirely. The component framework is flexible enough to allow loading of multiple format types; eggs and .py files. The component loader used by the Trac component system is in fact so useful that it sets the standard for how other Python web frameworks should load components.

The Trac component framework will load all defined components when the environment starts it's HTTP server and uses the load_components() function to do so. This function is defined in the loader.py module. This load_components() function can be thought of as the aggregate component loader as it is responsible for loading all component types. The load_component() function will accept a list of loaders in a keyword parameter. It uses these loaders to differentiate between component types. The parameter has two default loaders that will load egg components and .py source file components. The load_components() function will also except an extra path parameter which allows the specified path to be searched for additional Trac components. This is useful because developers may want to maintain a repository of Trac components that do not reside in site-packages or the Trac environment. The load_components() function also needs an environment parameter. This parameter refers to the Trac environment in which the loader is currently executing. This environment is needed by various loaders in order to determine if the loaded components should be enabled. This would also be a requirement of a custom loader if a developer was so inclined to write one. There is other useful environment information available to new loaders that could potentially provide more enhanced functionality.

As mentioned, the load_components() function specifies two default loaders for loading Trac components by default. These loaders are actually factories that build and return a loader function. This is done so that data from the load_components() function can be built into the loader function without having to alter the loader signature which is invoked by the load_components() function. This offers maximum flexibility. The first default loader, load_eggs(), will load Trac components in the egg format. This does so by iterating through the specified component search paths. The plugins directory of the current Trac environment is part of the included search path by default. For each egg file found, the working set object, which is part of the pkg_resources package, is then extended with the found egg file. Next, the distribution object, which represents the egg, is checked for trac.plugins entry points. Each found entry point is then loaded. What is interesting about this approach is that it allows subordinate Trac components to be loaded. This means if there is a found egg distribution containing a large amount of code and a couple small Trac components to be loaded, only the Trac components are loaded. The same cannot be said about the load_py_files() loader which is the second default loader provided by load_components(). This function works in the same way as the load_eggs() function in that it will search the same paths except instead of looking for egg files, it looks for .py files. When found, the loader will import the entire file, even if there is now subordinate Trac components within the module. In both default loaders, if the path in which any components were found is the plugins directory of the current Trac environment, that component will automatically be enabled. This is done so that the act of placing the component in the plugins directory also acts as an enabling action and thus eliminating a step.

There are some limitations with the Trac component framework. The _log_error() function nested inside the _load_eggs() loader shouldn't be a nested function. There is no real rationale for doing so. Also, loading Python source files as Trac components is also quite limiting because we loose any notion of subordinate components. This is because we can't define entry points inside Python source files. If building Trac components, I would recommend only building eggs as the format.

How Trac plugins are loaded.

Trac supports more than one method of loading plugins. One method is to place the .py file into the plugins directory of a given Trac environment directory. The other method is to install the plugin to site-packages as one would any other Python package. I prefer the latter method as it offers lower-coupling and hence a more robust deployment. The same effect can be achieved by using the former method to install Trac plugins but you'll have a tighter coupling to your Trac environment and I don't think this is desired.

Trac plugins are an excellent example of how to use entry points in Python. In order to function and be visible to Trac, the plugin must define a trac.plugins entry point that returns a list of all plugin components.

This functionality is implemented by the loader.py Trac module. Here is a simple illustration of what it does.



Here, the Trac loader retrieves a list of all Python eggs currently installed on the system. It will determine if the egg is a Trac plugin by examining the entry points provided by each egg. If it finds the trac.plugins entry point, all components of that plugin will be loaded.

How to manage technical documentation for varying levels of competency?

An interesting question on slashdot asks exactly this. Two things spring immediately to mind for me:

1. Trac
2. Is this possible?

The Trac wiki system would be my first choice simply because I'm familiar with it. Anyone with moderate Trac experience can teach the concepts to other people fairly easily. Developers and anyone else using the system.

So, the basic problem re-stated; "how do I provide a simple and easy way for people of with different levels of knowledge toward a given subject access to that information?". Using Trac, you could start by getting all required content into a page. This includes every possible detail imaginable.

Next, suppose we have written a Trac plugin that defines processors you can use to wrap around sections of text based on the required expertise. For instance, you could have the following processors defined:

• Expert-topic
• Intermediate-topic
• Moderate-topic
• New-topic

The second part of this theoretical plugin would need to extend the user accounts to allow the ability to specify what the user knows and at what level. Of course, each page would also need to be categorized as well.

The question of is this possible comes not from the technical end but from the business side of things. My answer to this is another question. How accurately can users' knowledge for a given topic be rated? This problem is eliminated if we were allow users to rank themselves in regards to topic competence.

Trac RecaptchaRegisterPlugin problems

The RecaptchaRegisterPlugin Trac extension has a few minor defects I noticed while testing it out for production use. The first problem, the captcha input would disappear if any other fields were invalid. The second problem, if the captcha field was invalid, any existing for data was lost. Here is my updated version of process_request() that addresses both issues.

#RecaptchaRegister trac plugin fix.

# IRequestHandler methods
def process_request(self, req):
  self.check_config()
  action = req.args.get('action')

  if req.method == 'POST' and action == 'create':
      response = captcha.submit(
          req.args['recaptcha_challenge_field'],
          req.args['recaptcha_response_field'],
          self.private_key,
          req.remote_addr,
          )
      if not response.is_valid:
          data = {'acctmgr' : { 'username' : req.args['user'],
                                'name' : req.args['name'],
                                'email' : req.args['email'],
                              },
          }
          data['registration_error'] = 'Captcha incorrect. Please try again.'
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, None
      else:
          ret = super(RecaptchaRegistrationModule, self).process_request(req)
          h, data, n = ret
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, n
  else:
      ret = super(RecaptchaRegistrationModule, self).process_request(req)
      h, data, n = ret
      data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
      data['recaptcha_theme'] = self.theme
      return "recaptcharegister.html", data, n

Trouble extending Trac navigation.

I'm in the process of building a new Trac site. I wanted to add new menu items to the main navigation. Hiding or rearranging the default menu items in Trac 0.11 is quite straightforward. It can all be done in the Trac configuration. However, new menu items need to be part of a component. Hence, the need for the NavAdd plugin.

It looks like the plugin has not yet been updated to fit the 0.11 plugin architecture, although it does work with 0.11. The Trac plugin API changes weren't too drastic. I did notice some strange behaviour though. It turns out that any menu items added with the NavAdd plugin can not be considered active.

What? No active menu items? Well, it isn't really the NavAdd plugins' fault. It turns out, that in order to have an active menu item, the current request needs to be handled by the same component that produces the menu item. I discovered this by looking at the timeline Trac component. This component actually implements the IRequestHandler interface. This is necessary because the timeline component handles requests to the /timeline URI. The menu item produced by this component becomes active anytime the /timeline URI is visited.

So, this design works well for components that have URIs. But what if I want to add menu items that point to wiki pages? And what if I want to have my menu item become active when visiting those pages? There is currently no way to do this. My suggestion would be that the INavigationContributor interface adds a new uri field to be returned from get_navigation_items(). When page corresponding to this uri becomes active, so does the custom menu item.

Gaphor plugin example.

The Gaphor UML modeling application provides an example hello world plugin. Obviously this plugin doesn't serve any useful purpose in the real world but it does give a good example of how Gaphor can be extended. The general layout for Gaphor plugins is similar to most Python packages. It has a setup.py module which enables the plugin to be installed independently of Gaphor.

Gaphor discovers new plugins through entry-points. The plugin must declare an entry-point that is available in the application. In this case, the hello world plugin wants to insert itself into gaphor.services.

The overall goal of the hello world plugin is to alter the menu in Gaphor and display a simple dialog. The actual plugin logic is contained in a single class called HelloWorldPlugin as illustrated below.



Here, the HelloWorldPlugin class provides both the IService and IActionProvider interfaces. The import items of interest are the menu_xml attribute and the helloworld_action() method.

The menu_xml attribute is an XML string that specifies where the new menu item for the plugin is placed within Gaphor.

The helloworld_action() method is responsible for implementing the action. In the case of the hello world plugin, it will display a dialog. Although not illustrated in the diagram, this method is actually decorated as an action. The decorator provides the action id, label, and tooltip.