The SQLObject attribute collection functionality

One of the many available Python database object-relational mappers, SQLObject has some interesting meta class functionality. There are many ways to declare a database table using an object-relational mapper such as SQLObject. However, since dealing with declarative meta data in the database, it would be a logical step to treat the Python declarations as meta data as well. A developer using SQLObject to declare a database table will define a class that extends the SQLObject class. Then, the fields of the database table will correspond to the class attributes of the newly derived SQLObject class. These can't just be any primitive Python type attributes. They have to be of a specific SQLObject type. When dealing with meta attributes such as these, much examination of these classes is required. SQLObject has functionality which will collect collect meta attributes from classes.

In the context of SQLObject, the rationale behind such a requirement as class attribute collection is simple. The SQLObject base classes are not always going to know which parameters are used with the class. When instantiating SQLObject instances, the keyword parameters correlate to the fields declared for that instances' database table. In several scenarios, SQLObject needs to inspect a given class and return a list of attributes that were passed to the constructor. This list of parameters at this point are unknown to the class, although, the SQLObject attribute collection functionality assumes that this each attribute in this list is a class attribute of the class being inspected. Since much of the behavior invoked on SQLObject instances is transformed, and called by meta classes, the attributes may not necessarily be needed by the instance. The attribute collection functionality in SQLObject has an opportunity to remove these attributes from the instance and perform any other transformations it sees fit. It also accepts parameters to help it determine what exactly it is modifying.

The _collectAttributes() function is where this functionality is implemented. The parameters that can be passed to this function are as follows.

• cls - This is the actual class that is being inspected. It is assumed that all attributes being collected are a part of this class.
• new_attrs - This parameter is the dictionary of keyword parameters that are passed to the constructor of SQLObject instances. The values in this dictionary are SQLObject column types.
• look_for_class - The column type to look for. Only attributes of this type will be returned.
• delete - If true, all attributes will be deleted from the class.
• set_name - The actual attribute name will change to the key of the dictionary item corresponding to the attribute.
• sort - If true, a sorted list will be returned. The sorting is based on field creation order.

Illustrated below is an over-simplified conceptualization of how the _collectAttributes() function is executed.

Trac component registration and management

Trac is a highly flexible project management system written in Python and based around a component architecture. In fact, a large portion of the base Trac system is indeed a set of components. Example components from this set would include the Trac ticketing system or the Trac wiki formatting engine. Using a component based architecture is a smart design decision in the majority software solutions for more reasons than one. Perhaps the most compelling reason to implement a component based architecture is the replaceability that components provide. Components both require and provide interfaces which means that these components can easily be swapped for a different component that provides the same interfaces as the original component. At the very core of Trac are a small set of classes that define how components in Trac work. Like any well designed software core, it is small and unlikely to change drastically in the future. The other benefit of the core being small is the fact that this core is depended upon by all Trac components in any given Trac installation. This core is not only required for interface purposes, but also for component registration and management. This way Trac always knows during its' lifetime what components are available to it. The core set of classes for dealing with components in Trac are ComponentMeta, ComponentManager, and Component.

The most important class here for Trac component developers is the Component class. This class is the external interface Trac provides to the outside world. The Component class is intended to be generalized or extended by each component within the Trac plugin. The ComponentMeta class is used to register defined components within a Trac environment by performing meta operations. That is, by transforming the original Component class as necessary. The ComponentManager class acts as a storage pool for all components in a Trac environment. Any time the Trac system needs access to any given component, it is retrieved through this class. This provides a centralized place for all components to live. Although the Component class is all the developers need concern themselves with, since the behavior of the other two classes is encapsulated, it is nonetheless useful to have a general idea of why they exist.

The Component class states that ComponentMeta class is its' meta class. Given this declaration in Python, when Component gets instantiated, the result returned from ComponentMeta.__new__() is what the instance will ultimately be. This is a useful feature of the language because it allows the behavior of the original class to be modified based on the context. The ComponentMeta.__new__() method has all the contextual data provided to it as parameters, including the original class, the name, base classes, and constructor parameters. The ComponentMeta class not only registers the various interfaces provided by the component in question, but will also redefine the Component constructor while still preserving the functionality of the original constructor. It does this by defining a nested maybe_init() function inside the ComponentMeta.__new__() method. The nested maybe_init() function will become the new Component constructor. The reason redefining the original constructor is so that a ComponentManager instance may now be passed to the Component constructor. This ComponentManager instance will then store the component. What really makes this useful is that if the original constructor existed within the Component in question, it is still invoked by the new maybe_init() constructor.

The ComponentManager is where Trac components are stored once loaded. As mentioned above, the ComponentMeta class dynamically injects functionality into the Component instantiation process that will store itself in a ComponentManager instance. Components stored in the ComponentManager instance be be retrieved by simple attribute access, using the component name as the attribute. This is implemented by the ComponentManager using the __get__() method. If a component is requested by this method that is not currently enabled, the ComponentManager will enable it before returning it. Otherwise it will simply return it. Developers also have an opportunity with Trac to subclass the ComponentManager and override the empty methods it invokes when enabling components. This could potentially be useful if enabling a component is a meaningful event.

Registering configuration values in ECP

The Enomaly Elastic Computing Platform has an extension module API that allows developers to register new ECP components. These components include new web controllers, new RESTful API controllers, and so on. One component that cannot be registered are configuration values. Extension modules can be viewed as smaller applications that are executed within ECP. Therefore, these smaller extension module applications will need to be configured. There are always going to be values that should be configurable within any application such as storage locations. Currently, extension modules must implement their own settings abstractions. This functionality already exists in the ECP core and the way configuration values are accessed and stored should be consistent and hence the need for the custom settings class. It would make sense for extension modules in ECP to have the ability to register their own configuration values. This way, configuration values would be accessed and stored in the exact same way across the platform. An additional complication arises when trying to use the configuration editor. The configuration editor is tightly-coupled with TurboGears widgets and thus requires that all extension modules be tightly-coupled with TurboGears widgets. Ideally, when configuration values are registered, which currently is not possible, additional meta data suitable for generating a display widget for the configuration value could also be registered.

The current implementation of the ECP Settings class uses managed Python attributes to seamlessly save and load configuration values. Every time a managed Settings attribute is accessed, the Variable class will attempt to load the variable. Likewise, when a managed Settings attribute is altered, the Variable class will attempt to store the configuration value. It is easier to use managed attributes for simple storage and retrieval operations. The alternative is to use the Variable class directly. In fact, earlier implementations of ECP did exactly that. Every time a configuration value was needed, we had to invoke Variable.load() while specifying a default value in case the configuration value didn't exist. The new Settings class was introduced to help alleviate some of this troubled configuration access. A single instance of the Settings class is created in the configuration.py module. This instance can then be used throughout the ECP application, including extension modules. Configuration categories are also incorporated into the Settings class. This is done by using the same concept as the Settings class for each category. This category class is then set as an attribute of Settings. This allows us to access configuration values in the form of settings.kvm.bridge. This syntax offers much more readable code when used in context. However, the problem with this method of managing configuration values was soon after realized. There will always be a need to add new configuration values. Most noteably, extension modules are going to need this capability since developers are going to want to access configuration values in the same way as the rest of ECP. There is a need to be able to register new configuration values. This eliminates the extensibility problem of adding new configuration values. If every time a new configuration value needed by an extension module, or the core application for that matter, needs to be added to the Settings class, it will grow exponentially and become very challenging to maintain. Additionally, the configuration editor is very tightly coupled to TurboGears widgets because extension modules need to display these configuration values in the configuration editor. This is done by the extension module defining a hook that passes in TurboGears widgets used to display the configuration values for the extension module in the configuration editor. This isn't the ideal method since this also couples the extension modules to ECP dependencies (TurboGears). Ideally, the widgets for displaying configuration values should be generated by the configuration editor based on minimal meta-data provided by the extension module at registration time.

The new approach to ECP configuration value management is to have configuration values registered in the Settings class. The same approach of using managed attributes to access and store configuration values is still used. What is different is the ability to register a value and have these managed attributes automatically built for the developer. This is accomplished by introducing a new MetaSettings class. The purpose behind this new class is to dynamically construct new categorization classes and methods that will become attributes of the settings instance. There is also a new settings.register() method that can be used to register new configuration values. The end result of using settings.register() to register a new configuration value is the same syntax as before when using the configuration values. The name of the module passed to settings.register() will become an attribute of the settings instance. There are also meta-data parameters in the settings.register() method that allow developers to specify a title and description of the configuration value. In the current ECP configuration management implementation, this information must be specified in the TurboGears widget. With the new implementation, the managed attribute functionality found in the ECP core no longer needs to be duplicated. There is now a much more uniform interface.

With this new configuration registration functionality in place, there is now an opportunity for great improvements in the configuration editor. We could now potentially eliminate the coupling to TurboGears widgets and have each configuration widget generated automatically. Grouping by extension module is now also possible in the configuration editor.