Trac Web Interface

The Trac project management system provides a nice set of web interfaces. What makes these interfaces nice to work with is that they are concise and to the point. The make defining new Trac web components an intuitive process. There are two interfaces of note in the set of Trac web interfaces. These are IRequestHandler and IRequestFilter. A simple illustration of what these interfaces look like conceptually is shown below.



Both interfaces obviously need to inherit from the Interface class since they themselves are interfaces.

The IRequestHandler interface is responsible for handling HTTP requests. Developers cannot define Trac web components that do not implement this interface. The two responsibilities of components that implement this interface are two determine if the component can handle the request and to return a response. This is different from some frameworks in that the component itself carries the responsibility of matching a URL.

The IRequestFilter interface is responsible for pre and post request handler processing. The pre processing occurs once a handler has been selected but before a response is generated by the handler. The post processing occurs once a response is generated.

Web components in Trac generally implement both of these interfaces. This means that the pre and post processing functionality uses the component itself as parameters. It is a very nice API to work with and other frameworks could certainly benefit from modeling after it.

Navigating Trac Tickets

The Trac project management system allows users of the system to perform queries for tickets. In Trac, tickets are entities that help keep track of outstanding tasks, bug fixes, or basically anything the needs to be accomplished for a given software project. Trac default set of useful ticket queries that users can easily execute. The default set of ticket queries allow interested parties to common groups of tickets such as those belonging to a particular developer.

When viewing a ticket, Trac displays a set of links in the top right of the screen that allow the user to view the previous ticket or the next ticket. There is also a link to return to the the query results. The next ticket and previous ticket links are useful because tickets can be navigated without the need to return to the query. However, the ticket order is based on the ticket number. This presents a problem if the user is only interested in navigating back and forth between tickets in the current query result set. The next ticket can always be completely irrelevant to the tickets the user is interested in.

The really useful aspect of the link back to the query results is how the presentation of the query results can change. Tickets that have been edited while you were viewing a specific ticket, change appearance in the query results once the user returns to the query results. You will also notice that the current user is part of the URL back to the query results.

This feature is incredibly useful in a high paced development environment with many developers. It also helps if a small group of developers are collaborating on a small group of related tickets because changes to tickets by other people become immediately apparent.

Configuring Trac Ticket Workflow

Trac is a project management system written in Python. Trac has a powerful issue tracking system in which new issues are created as tickets. The system is flexible enough to allow for custom ticket types. This means that Trac tickets not only allow for issue tracking, but any conceivable software development task as well. There are many other customizations that can be made to Trac tickets such as custom fields and custom work-flow. Tickets in Trac go through a series of states during their lifetime. The current state of a given ticket determines the actions available on that ticket. The state of a ticket in Trac terminology is the ticket status. The ticket work-flow that determines the actions available, the permissions required, and the state the ticket must be in in order to transition are all configurable. In other words, if a given ticket is closed, the only available action for the ticket is to reopen it. This is because of the default Trac ticket work-flow configuration.

In the Trac configuration, ticket actions can be configured to allow for certain ticket states to be active in order to execute the action. So, if you wanted to allow more actions other than reopen on closed tickets, this can be specified in the Trac configuration. Multiple current states may be specified as criteria. Only a single target state may be specified. Below is a simplistic example of a ticket work-flow configuration element.

[ticket-workflow]

action = current_state1, current_state2, current_state3 -> target_state

action.permission = TICKET_MODIFY

action.operation = ticket_operation_name

Trac Database Connection Wrapping

Web applications written in Python tend to use some kind of object-relational mapping technology such as SQLObject or SQLAlchemy. Trac, however, has not yet introduced such dependencies nor do they define an object-relational mapper of their own. Trac uses the more traditional database connection and database cursor abstractions. This isn't to say that there aren't any higher-level abstractions defined. There are and they are really designed quite well and are therefore easy to comprehend.

Of course, object-relational mapping technology isn't the be-all and end-all of all web applications. In fact, if you can do without the overhead involved you may be better off as long as you can clearly define a separation of concerns.

For instance, Trac defines a database drivers for MySQL as well as others. This is typically the selling point for object-relational technology. Trac's approach is to define two base classes for the core database abstractions; IterableCursor and ConnectionWrapper. These classes provide the generic functionality shared by all database drivers. A brief idea of the design is illustrated below.

The Trac Ticket Database Model

In Trac trunk, we can get a glimpse into the schematics behind the Trac ticket. The internals of the Trac ticket database model are interesting to see because the ticket is such a central concept in the Trac system. Below is an illustration of the Ticket database model class.



Here are some brief highlights of what each method does.

• __init__() - Performs initialization actions. If the ticket id was passed, the values of the ticket in the database will be populated in this instance. Otherwise, the default values will be used.
• id_is_valid() - Return true if the specified ticket id is valid.
• _get_db() - Return the database connection.
• _get_db_for_write() - Return the database connection for writing.
• _init_defaults() - Initialize the default field values. In addition to initializing the default field values, the default options that are available for a given ticket field are also initialized.
• _fetch_ticket() - Initialize this ticket instance using specific ticket field values. This is done by first executing a query to load the stored ticket data from the database.
• __getitem__() - Support the getitem operator to retrieve ticket field values.
• __setitem__() - Support the setitem operator store ticket field values.
• get_value_or_default() - Return the value of the field or the default value. The default value is only returned if there is a problem retrieving the actual value because it does not yet exist.
• populate() - Populate the ticket fields using the specified dictionary. Only valid keys in the supplied dictionary, that exist as ticket fields, will be populated.
• insert() - Insert this ticket into the database. This method will not insert the ticket if the ticket id already exists in the database.
• save_changes() - Update the database with any changes made to this ticket. This includes updating the ticket changelog.
• get_changelog() - Retrieve the changelog data for this ticket.
• delete() - Delete this ticket from the database.

How Trac Determines The Wiki Processor

Within Trac is a powerful wiki syntax engine that offers a wide array of tools to the author who creates wiki pages. Richly formatted Trac wiki pages can be created in several ways. Just using the default wiki syntax to format plain text is often enough to get started. However, Trac offers much more. There are two other key components to the Trac wiki syntax; processors and macros. Macros invoke specific Python functionality, possibly with supplied parameters, to inject dynamic page content. Processors on the other hand, wrap around a specific chunk of wiki text and manipulate it in controlled ways. For instance, a plain processor would wrap some wiki text in three curly brackets. Doing this would simply transform the wrapped text to pre-formatted output. Wiki page authors can also invoke more exotic processors, such as html or syntax highlighting processors. To use a specific processor, the first line in the curly brackets must specify the processor name. The name of the processor must also be preceded by #!. So, for example, an html processor definition in Trac might look like {{{#!html hello world}}}. So, how does Trac know about these specific types of processors? Under the hood, The WikiProcessor class takes on this responsibility. The key task of this class is to load the required processor rendering functionality. Illustrated below is an elided view of the WikiProcessor class, showing only key attributes.




The logic executed by the WikiProcessor class to determine which processor is to be used in the rendering process takes place in the constructor, which accepts a processor name parameter. First, the constructor will check if the specified processor name is part of the builtin processor set. The builtin processor set in actually hard-coded in the constructor. If the specified processor name is found in this set, the processor attribute of the WikiProcessor instance becomes the method associated with the builtin processor name. If the specified processor isn't a builtin processor, the processor could potentially be a macro. All the macro providers, which contain actual macros, are retrieved from the WikiSystem class and iterated over. Each macro provider is then checked, to see if the specified processor is a macro. If it turns out that the specified processor is a macro, the macro_provider attribute of the WikiProcessor instance becomes the macro provider containing the matching macro. Also, the processor attribute of the WikiProcessor instance will become a method for rendering the macro. Finally, if the specified processor isn't a builtin and isn't a macro, the default rendering functionality is used.

This is an extremely flexible way to determine where the extensible wiki processing functionality resides. The one issue in the WikiProcessor constructor is that there are three if statements, all in the same context with no else. This means that all three if statements are executed no matter what. Not only is this inefficient, but the else adds a grouping aspect to the code. However, this is trivially easy to remedy.

Trac component registration and management

Trac is a highly flexible project management system written in Python and based around a component architecture. In fact, a large portion of the base Trac system is indeed a set of components. Example components from this set would include the Trac ticketing system or the Trac wiki formatting engine. Using a component based architecture is a smart design decision in the majority software solutions for more reasons than one. Perhaps the most compelling reason to implement a component based architecture is the replaceability that components provide. Components both require and provide interfaces which means that these components can easily be swapped for a different component that provides the same interfaces as the original component. At the very core of Trac are a small set of classes that define how components in Trac work. Like any well designed software core, it is small and unlikely to change drastically in the future. The other benefit of the core being small is the fact that this core is depended upon by all Trac components in any given Trac installation. This core is not only required for interface purposes, but also for component registration and management. This way Trac always knows during its' lifetime what components are available to it. The core set of classes for dealing with components in Trac are ComponentMeta, ComponentManager, and Component.

The most important class here for Trac component developers is the Component class. This class is the external interface Trac provides to the outside world. The Component class is intended to be generalized or extended by each component within the Trac plugin. The ComponentMeta class is used to register defined components within a Trac environment by performing meta operations. That is, by transforming the original Component class as necessary. The ComponentManager class acts as a storage pool for all components in a Trac environment. Any time the Trac system needs access to any given component, it is retrieved through this class. This provides a centralized place for all components to live. Although the Component class is all the developers need concern themselves with, since the behavior of the other two classes is encapsulated, it is nonetheless useful to have a general idea of why they exist.

The Component class states that ComponentMeta class is its' meta class. Given this declaration in Python, when Component gets instantiated, the result returned from ComponentMeta.__new__() is what the instance will ultimately be. This is a useful feature of the language because it allows the behavior of the original class to be modified based on the context. The ComponentMeta.__new__() method has all the contextual data provided to it as parameters, including the original class, the name, base classes, and constructor parameters. The ComponentMeta class not only registers the various interfaces provided by the component in question, but will also redefine the Component constructor while still preserving the functionality of the original constructor. It does this by defining a nested maybe_init() function inside the ComponentMeta.__new__() method. The nested maybe_init() function will become the new Component constructor. The reason redefining the original constructor is so that a ComponentManager instance may now be passed to the Component constructor. This ComponentManager instance will then store the component. What really makes this useful is that if the original constructor existed within the Component in question, it is still invoked by the new maybe_init() constructor.

The ComponentManager is where Trac components are stored once loaded. As mentioned above, the ComponentMeta class dynamically injects functionality into the Component instantiation process that will store itself in a ComponentManager instance. Components stored in the ComponentManager instance be be retrieved by simple attribute access, using the component name as the attribute. This is implemented by the ComponentManager using the __get__() method. If a component is requested by this method that is not currently enabled, the ComponentManager will enable it before returning it. Otherwise it will simply return it. Developers also have an opportunity with Trac to subclass the ComponentManager and override the empty methods it invokes when enabling components. This could potentially be useful if enabling a component is a meaningful event.

The Trac component loader

All modern web application frameworks need replaceable application components. Reasons for this requirement are plenty. Some applications will share common functionality with other applications such as identity management. However, having the ability to extend this functionality or replace it entirely is absolutely crucial. Technology requirements change too fast to assume that a single implementation of some feature will ever be sufficient for any significant length of time. Moreover, tools that are better for the job that your component currently does will emerge and developers need a way to exploit the benefit of these tools without having to hack the core system. Trac is a Python web-framework in it's own right. That is, it implements several framework capabilities found in other frameworks such as Django and TurboGears. Trac is highly specialized as a project management system. So, you wouldn't want go use Trac as some generalized web framework for some other domain. Project management for software projects is such a huge domain by itself that it makes sense to have a web framework centered around it. Trac defines it's own component system that allows developers to create new components that build on existing Trac functionality or replace it entirely. The component framework is flexible enough to allow loading of multiple format types; eggs and .py files. The component loader used by the Trac component system is in fact so useful that it sets the standard for how other Python web frameworks should load components.

The Trac component framework will load all defined components when the environment starts it's HTTP server and uses the load_components() function to do so. This function is defined in the loader.py module. This load_components() function can be thought of as the aggregate component loader as it is responsible for loading all component types. The load_component() function will accept a list of loaders in a keyword parameter. It uses these loaders to differentiate between component types. The parameter has two default loaders that will load egg components and .py source file components. The load_components() function will also except an extra path parameter which allows the specified path to be searched for additional Trac components. This is useful because developers may want to maintain a repository of Trac components that do not reside in site-packages or the Trac environment. The load_components() function also needs an environment parameter. This parameter refers to the Trac environment in which the loader is currently executing. This environment is needed by various loaders in order to determine if the loaded components should be enabled. This would also be a requirement of a custom loader if a developer was so inclined to write one. There is other useful environment information available to new loaders that could potentially provide more enhanced functionality.

As mentioned, the load_components() function specifies two default loaders for loading Trac components by default. These loaders are actually factories that build and return a loader function. This is done so that data from the load_components() function can be built into the loader function without having to alter the loader signature which is invoked by the load_components() function. This offers maximum flexibility. The first default loader, load_eggs(), will load Trac components in the egg format. This does so by iterating through the specified component search paths. The plugins directory of the current Trac environment is part of the included search path by default. For each egg file found, the working set object, which is part of the pkg_resources package, is then extended with the found egg file. Next, the distribution object, which represents the egg, is checked for trac.plugins entry points. Each found entry point is then loaded. What is interesting about this approach is that it allows subordinate Trac components to be loaded. This means if there is a found egg distribution containing a large amount of code and a couple small Trac components to be loaded, only the Trac components are loaded. The same cannot be said about the load_py_files() loader which is the second default loader provided by load_components(). This function works in the same way as the load_eggs() function in that it will search the same paths except instead of looking for egg files, it looks for .py files. When found, the loader will import the entire file, even if there is now subordinate Trac components within the module. In both default loaders, if the path in which any components were found is the plugins directory of the current Trac environment, that component will automatically be enabled. This is done so that the act of placing the component in the plugins directory also acts as an enabling action and thus eliminating a step.

There are some limitations with the Trac component framework. The _log_error() function nested inside the _load_eggs() loader shouldn't be a nested function. There is no real rationale for doing so. Also, loading Python source files as Trac components is also quite limiting because we loose any notion of subordinate components. This is because we can't define entry points inside Python source files. If building Trac components, I would recommend only building eggs as the format.

Babel in the Trac trunk.

With the Trac 0.11 branch, internationalization and localization are not possible. I do, however, like the approach that are taking to incrementally implementing this feature. In the 0.11 branch, the gettext() function is actually defined in translation.py. Although it doesn't do anything useful in this branch of the software, it is implemented. More importantly, it is used throughout the application. The pipes of the Trac translation architecture have been assembled, they just don't have anything flowing through them yet.

In the trunk version of translation.py, we have something radically different. There is a new TranslationsProxy class. This class is considered a proxy because the original translation functions still exist only now they invoke methods in this class. Here is an illustration of the TranslationsProxy class.



The attributes are as follows.

• The _current attribute is the current thread of control.
• The _null_translations attribute is an empty message catalogue.
• The _plugin_domains attribute is a dictionary that contains locale domains that may be added to the TranslationsProxy in a programmatic way.
• The _plugin_domains_lock is a threading lock that is acquired when loading

The need for the _current and _plugin_domains_lock attributes arises when activating a locale. The threading lock is acquired while loading plugin translation domains. Once loaded, the lock is released and the translation message catalogue is set as an attribute on the current thread. This allows different threads of control to have different message catalogues.

I really enjoy the way the plugin message catalogues are merged with the existing catalogue in a thread-safe way. Just based on this feature alone I'm very exited about the next major Trac release.

I wonder if some of the other Python web-frameworks out there handle internationalization message catalogue extensibility this way? I'm sure TurboGears doesn't. I've found that having only a single catalogue for my applications to be very limiting.

How Trac plugins are loaded.

Trac supports more than one method of loading plugins. One method is to place the .py file into the plugins directory of a given Trac environment directory. The other method is to install the plugin to site-packages as one would any other Python package. I prefer the latter method as it offers lower-coupling and hence a more robust deployment. The same effect can be achieved by using the former method to install Trac plugins but you'll have a tighter coupling to your Trac environment and I don't think this is desired.

Trac plugins are an excellent example of how to use entry points in Python. In order to function and be visible to Trac, the plugin must define a trac.plugins entry point that returns a list of all plugin components.

This functionality is implemented by the loader.py Trac module. Here is a simple illustration of what it does.



Here, the Trac loader retrieves a list of all Python eggs currently installed on the system. It will determine if the egg is a Trac plugin by examining the entry points provided by each egg. If it finds the trac.plugins entry point, all components of that plugin will be loaded.

How to manage technical documentation for varying levels of competency?

An interesting question on slashdot asks exactly this. Two things spring immediately to mind for me:

1. Trac
2. Is this possible?

The Trac wiki system would be my first choice simply because I'm familiar with it. Anyone with moderate Trac experience can teach the concepts to other people fairly easily. Developers and anyone else using the system.

So, the basic problem re-stated; "how do I provide a simple and easy way for people of with different levels of knowledge toward a given subject access to that information?". Using Trac, you could start by getting all required content into a page. This includes every possible detail imaginable.

Next, suppose we have written a Trac plugin that defines processors you can use to wrap around sections of text based on the required expertise. For instance, you could have the following processors defined:

• Expert-topic
• Intermediate-topic
• Moderate-topic
• New-topic

The second part of this theoretical plugin would need to extend the user accounts to allow the ability to specify what the user knows and at what level. Of course, each page would also need to be categorized as well.

The question of is this possible comes not from the technical end but from the business side of things. My answer to this is another question. How accurately can users' knowledge for a given topic be rated? This problem is eliminated if we were allow users to rank themselves in regards to topic competence.

New ECP community site.

I'm pleased to announce that the new Enomaly ECP community site is up and running. Feel free to report bugs, request features, or check out the documentation.

Trac RecaptchaRegisterPlugin problems

The RecaptchaRegisterPlugin Trac extension has a few minor defects I noticed while testing it out for production use. The first problem, the captcha input would disappear if any other fields were invalid. The second problem, if the captcha field was invalid, any existing for data was lost. Here is my updated version of process_request() that addresses both issues.

#RecaptchaRegister trac plugin fix.

# IRequestHandler methods
def process_request(self, req):
  self.check_config()
  action = req.args.get('action')

  if req.method == 'POST' and action == 'create':
      response = captcha.submit(
          req.args['recaptcha_challenge_field'],
          req.args['recaptcha_response_field'],
          self.private_key,
          req.remote_addr,
          )
      if not response.is_valid:
          data = {'acctmgr' : { 'username' : req.args['user'],
                                'name' : req.args['name'],
                                'email' : req.args['email'],
                              },
          }
          data['registration_error'] = 'Captcha incorrect. Please try again.'
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, None
      else:
          ret = super(RecaptchaRegistrationModule, self).process_request(req)
          h, data, n = ret
          data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
          data['recaptcha_theme'] = self.theme
          return "recaptcharegister.html", data, n
  else:
      ret = super(RecaptchaRegistrationModule, self).process_request(req)
      h, data, n = ret
      data['recaptcha_javascript'] = captcha.displayhtml(self.public_key)
      data['recaptcha_theme'] = self.theme
      return "recaptcharegister.html", data, n

Trouble extending Trac navigation.

I'm in the process of building a new Trac site. I wanted to add new menu items to the main navigation. Hiding or rearranging the default menu items in Trac 0.11 is quite straightforward. It can all be done in the Trac configuration. However, new menu items need to be part of a component. Hence, the need for the NavAdd plugin.

It looks like the plugin has not yet been updated to fit the 0.11 plugin architecture, although it does work with 0.11. The Trac plugin API changes weren't too drastic. I did notice some strange behaviour though. It turns out that any menu items added with the NavAdd plugin can not be considered active.

What? No active menu items? Well, it isn't really the NavAdd plugins' fault. It turns out, that in order to have an active menu item, the current request needs to be handled by the same component that produces the menu item. I discovered this by looking at the timeline Trac component. This component actually implements the IRequestHandler interface. This is necessary because the timeline component handles requests to the /timeline URI. The menu item produced by this component becomes active anytime the /timeline URI is visited.

So, this design works well for components that have URIs. But what if I want to add menu items that point to wiki pages? And what if I want to have my menu item become active when visiting those pages? There is currently no way to do this. My suggestion would be that the INavigationContributor interface adds a new uri field to be returned from get_navigation_items(). When page corresponding to this uri becomes active, so does the custom menu item.

Perl now uses GIT

Not that I am interested in the perl language but apparently they are now using the GIT version control system.

What is interesting is how fast this software is gaining popularity. I'm still using Trac + SVN.

Trac provides a good example of RESTful resources

The Trac issue tracking and wiki system provides a good example of a RESTful web resource. Good RESTful resources are connected. This means that the associations between resources are within the resource. If a given resource is associated with another resource, the second resource should be navigable from the first.

This is the basic concept behind links in hypermedia.

Trac, however takes this a step further with a trivial feature that is much more valuable than it may seem at first glance. Trac integrates very nicely with subversion. In a large percentage of cases, one ore more changesets are associated with a ticket. It is also fairly trivial to link to a changeset from within a ticket. Here is the interesting feature. The title attribute of the anchor that links to the changeset is the message associated with the changeset. Here is what I mean.



This subtle feature has saved me the time required to actually follow a link to view the changeset resource on numerous occasions. Most of the time, we're only interested in the message. Trac has realized this usability issue and addressed it.