Hashing URIs

Clients of URIs often need to specify some sort of client parameter that is returned as part of the HTTP response. This parameter is intended to preserve the application state by encoding it into the URI.

A common practice is instead of passing the client state as a GET HTTP parameter, to provide the client state as a hashed URI segment. This segment would represent the client state. It is a preferable URI practice because the application state is still encoded as part of the resource URI but doesn't need to be sent to the server.

One aspect of this practice to be weary of is mistaking application state for resources. If the intent is to use the application state as the hashed URI segment, only use it for application states and not data resources.

Handling Callback Variables

With Javascript applications that make asynchronous Ajax requests, does it make sense to store a global variable that indicates the callback that should be executed? That is, depending on the current state of the application, a different callback function may need to execute for the same API call. Should a global variable be set before the request is made and checked each time by the response?

That doesn't feel right to me. Neither does sending application data along with the API request just to be used by the callback. That would mean sending application state to the server which isn't a good thing as far as I'm concerned.

I like the idea of listening for certain events that the callback might emit during execution. The callback might emit a certain type of event each time it is run. The events are always emitted but the application may not always be listening. When the application is in a certain state, it will be listening. The handlers of these events could be considered secondary callbacks.

Twisted System Events

One of the key features of the Twisted Python web framework is the ability to define reactors that react to asynchronous events. One concept of the the Twisted reactor is the system event. The ReactorBase class is inherited from all reactor types in Twisted, as the name suggests. It is this class that provides all other reactors with a system event processing implementation.

An event, in the context of the Twisted reactor system, has three phases, or states. These states are "before", "during", and "after". What this provides for developers is a means to conceptually organize triggers that are executed when a specific event is fired. The "before" state should execute triggers that are supposed to verify certain data, or perform setup tasks. Anything that would be considered a pre-condition is executed here. The "during" state is overall goal of the event. Triggers that are executed in this state should do the heavy processing. Conceptually, this is the main reason a trigger was registered to execute with the specified event type in the first place. Finally, the "after" state executes triggers that should perform post-condition testing, or clean-up type tasks.

Illustrated below are the various states that a Twisted system event will go through during its' lifetime. The transitions between states are quite straightforward. When there are no more triggers to execute for the current state, the next state is entered.



Event triggers are registered with specific event types by invoking the ReactorBase.addSystemEventTrigger() method. This method accepts an event state, callable, and event type parameters. The callable can be any callable Python object.

The type of event in which triggers can be registered to can be anything. The event type is only the key for a stored event instance. The _ThreePhaseEvent class is instantiated if not already part on the reactor. That is, if a trigger has already been registered for the same event type, that means an event instance has been created. The _ThreePhaseEvent instance for each event type is responsible for executing all event triggers in the correct order. Using the Twisted system event functionality means that dependencies between event states may be used to achieve desired functionality.

Python State Machine

State machines are handy during software development to help visualize the changing states of either a single instance, or a group of instances over time. Modeling state machines is also useful for visualizing the events that cause instances to change state. It is for this reason that the UML includes states and state machines.

State machines aren't limited to visualizations. They can also be directly implemented. The StatePy Python package provides a basic set of abstractions that developers can use to implement such a state machine. The elements include states and machines. Also included are events that trigger state transitions. There is currently no support for defining composite states but this package seems to be early on in development. Below is a simple example of how to setup a state machine.

#Example; Creating Python States.

#Do imports.
from statepy.state import State, Machine, Event

#Printing state.
class StatePrinting(State):
  
   #Constructor.  Initialize the State.
   def __init__(self, *args, **kw):
       State.__init__(self, *args, **kw)
  
   #On state entry.
   def enter(self):
       print "Printing..."

   #On state exit.
   def exit(self):
       print "Finished printing..."

#Sorting state.
class StateSorting(State):
  
   #Constructor.  Initialize the State.
   def __init__(self, *args, **kw):
       State.__init__(self, *args, **kw)
  
   #On state entry.
   def enter(self):
       print "Sorting..."

   #On state exit.
   def exit(self):
       print "Finished sorting..."
  
   #Provided state transitions.
   @staticmethod
   def transitions():
       return {"FINISHED_SORTING":StatePrinting}       

#Building state.
class StateBuilding(State):
  
   #Constructor.  Initialize the State.
   def __init__(self, *args, **kw):
       State.__init__(self, *args, **kw)
  
   #On state entry.
   def enter(self):
       print "Building..."

   #On state exit.
   def exit(self):
       print "Finished building..."
      
   #Provided state transitions.
   @staticmethod
   def transitions():
       return {"FINISHED_BUILDING":StateSorting}

#Main.
if __name__=="__main__":
  
   #Initialize the machine.
   machine_obj=Machine()
  
   #Start the machine in a building state.
   machine_obj.start(StateBuilding)
  
   #Pass the transition events to the machine.
   machine_obj.injectEvent(Event("FINISHED_BUILDING"))
   machine_obj.injectEvent(Event("FINISHED_SORTING"))

Stateful Python Lists

The Python programming language defines list objects as one of the primitive language types. Lists, are similar to arrays in more traditional programming languages. Of course, Python lists are mutable types, as are arrays. The main difference between Python lists and traditional arrays are similar in most other comparisons of Python to other languages; Python lists are more flexible and elegant. However, Python lists are not magic. One situation where Python lists have the potential to easily fall apart is when used in a multi-threaded environment. This fact isn't exclusive to lists, the same risk involved with passing data across threads applies to all types. The good news is that this can be easily remedied with lists. Python list instances support the notion of "pushing" and "pulling" data. This is a useful thought construct because this transfers directly to the design. Any given list instance can be "pushing" state when adding data to the list or a "pulling" state when retrieving data from the list.

The Set class in the boduch Python library is essentially a Python list that publishes events. These events that are published by Set instances have the potential to be handled in a multi-threaded environment. If Set instances are being both "pushed" to and "pulled" from in a multi-threaded environment, this could be very dangerous. Data that is expected to exist in the set instance when "pulled" may not have arrived yet. In this scenario, it would be useful to know what state the Set instance is in. By combining both the Set and StateMachine classes as demonstrated below, we end up with a "stateful" Python list. The implementation of the StatefulSet shown here isn't fully thread safe. This is because it is incomplete. However, the code used in the main program is thread safe.

#example; Creating a multi-threaded Python list with a boduch Set.

#Necessary imports.
from boduch.event import threaded, subscribe, EventSetPush
from boduch.handle import Handle, HandleSetPush
from boduch.data import Set
from boduch.state import StateMachine
from boduch.type import is_type
from boduch import PRIORITY_MINOR, PRIORITY_CRITICAL

#A handle for Set.push() method invocations.
class HandlePreStatefulSetPush(Handle):
   #This gets a higher than critical priority since it is
   #intended to run before the core behavior.
   priority=PRIORITY_CRITICAL+1
   def __init__(self, *args, **kw):
       Handle.__init__(self, *args, **kw)
      
   def run(self):
       #Check if we are handling a StatefulSet event.  If
       #so, set goes into a pushing state.
       set_obj=self.get_event_data("set")
       if is_type(set_obj, "StatefulSet"):
           set_obj.change_state("PUSHING")

#A handle for Set.push() method invocations.
class HandlePostStatefulSetPush(Handle):
   #This handle gets a minor priority since it is 
   #intended to run after all the pushing is complete.
   priority=PRIORITY_MINOR
   def __init__(self, *args, **kw):
       Handle.__init__(self, *args, **kw)
      
   def run(self):
       #Check if we are handling a StatefulSet event.  If
       #so, set goes into a ready state.
       set_obj=self.get_event_data("set")
       if is_type(set_obj, "StatefulSet"):
           set_obj.change_state("READY")

#A stateful version of the Set class.
class StatefulSet(StateMachine, Set):
   def __init__(self, *args, **kw):
       #Initialize the base classes and populate the 
       #potential set states.
       StateMachine.__init__(self, *args, **kw)
       Set.__init__(self, *args, **kw)
       self.add_state("READY")
       self.add_state("PUSHING")
      
   def _push(self, data):
       #Change to a pushing state before actually pushing data.
       self.change_state("PUSHING")
       self.push(data)
      
   def _get(self, index):
       #Don't attempt to return anything while in a pushing state.
       while self=="PUSHING":
           print "Hold on, still pushing data."
           pass
       return self[index]
  
#Subscribe the custom stateful handles.
subscribe(EventSetPush, HandlePreStatefulSetPush)
subscribe(EventSetPush, HandlePostStatefulSetPush)

#Main program.
if __name__=="__main__":
   #Enable threaded event mode.
   threaded(True)
   #Instantiate a stateful set.
   set_obj=StatefulSet()
   #Populate the set while retrieving data in a thread-safe manor.
   for i in range(0,10):
       set_obj._push("SET DATA"+str(i))
       print set_obj._get(i)
  

Creating State Machines in Python

In the UML, state machines help modelers understand the various states the system in question goes through. When the system is in one state, the available behavior differs from when the system is in another state. Visualizing these states by drawing them as UML elements is an extremely powerful tool. State machines are not only powerful for examining the potential states of a given system, but also in helping to understand the transitions that could potentially take place between these states. In object-orientated systems, the concept of objects being in one state or another is often implemented in code, even if implicitly. Given that this is object-oriented code we are working with, and the whole idea behind object orientation is to assist with creating realistic abstractions, why not use the concept of state machines directly in code? The 0.1.6 version of the boduch Python library tries to realize this possibility. Although this is the first iteration of this new feature, with lots of work left to do, it seems promising. Below is an example of the new StateMachine class in use.

#Example; Using boduch state machines.

from boduch.state import StateMachine
from boduch.predicate import Equal

#Define two predicate functions used to return
#the predicate variables.
def get_o1():
   return o1

def get_o2():
   return o2

#The two predicate variables.  Notice they 
#aren't equal.
o1=1
o2=2

#Construct a state machine instance.
state_obj=StateMachine()

#Construct a predicate instance using the 
#two previously defined predicate functions.
predicate_obj=Equal(get_o1, get_o2)

#Populate the state machine with two states.
state_obj.add_state("RUNNING")
state_obj.add_state("STOPPED")

#Add a state transition to the machine using
#the previously defined predicate.  Once true,
#the machine will be in the target state.
state_obj.add_transition("STOPPED", predicate_obj)

if __name__=="__main__":
   #Change the machine to a RUNNING state.
   state_obj.change_state("RUNNING")
   print state_obj.current_state
  
   #Change the value of the predicate variable.
   o2=1
   print "o2 now equals o1"
  
   #Check for any state changes.
   state_obj.transition()
   print state_obj.current_state
  

Be brief overview of the above code follows below.

• Import the classes required. There are StateMachine and Equal.
• Define the predicate functions. These functions are used by the predicate associated with the state machine. They are invoked each time the predicate is evaluated.
• Define the predicate variables. These variables are global and are returned by the predicate functions. Any changes to these variables are reflected by invoking the predicate functions.
• Create a new StateMachine instance. This is the main state machine involved in the example.
• Create a new Equal predicate instance and use the predicate functions as operands.
• Populate the state machine instance with some states. A state can be any Python object.
• Add a new transition to the state machine using the previously defined Equal instance. When this predicate instance evaluates to true, the target state, in this case, "STOPPED", will become the active state.
• Set the initial state to "RUNNING".
• Change the value of one of the predicate variables. The predicate can now be considered equal.
• Test the state machine for any state changes. Here, there are changes because the transition that was previously added to the state machine will evaluate to true, causing the target of the transition to become the active state.
• The state machine is now in a "STOPPED" state.

New ECP exception and enhanced state restoring behaviour

Over the past few days, the ECP team has made some notable enhancements in the trunk. The first being the addition of a new exception called E2LibvirtError. As the title suggests, this exception is raised for Libvirt-related issues. The Python Libvirt library already defines an exception class. However, there are many types of errors that can happen from within Libvirt. The idea behind this new E2LibvirtError class is to provide better information when something bad happens in libvirt. For instance, in the Python Libvirt library, there is only one exception type. If this exception gets raised, a short message is displayed. This is the default message that gets initialized with the Python base exception class.

The problem here being that Libvirt can manage several different hypervisors on any given system. Thus, there are several layers within Libvirt in which something can go wrong. In ECP, the Libvirt exception is caught, and the generic message is recorded.

The new E2LibvirtError exception exploits additional exception information encapsulated within the basic Libvirt exception instance. I don't mean encapsulated in the traditional object-oriented sense. I mean the information is there, and ECP should use it for the benefit of the end user. The new ECP exception, when instantiated, will take several error codes from the original Libvirt exception and produce a much more meaningful error message.

This leads me to the changes made in the restore_machines_state() functionality. The rationale hasn't changed, only the implementation. We simply handle table existence and local machine existence detection much better than the current version. If the function finds a machine that is not running and it should be (because that was the state the machine was in when ECP last shut down), it will attempt to start it. We've already added the new E2LibvirtError exception handling to this function when attempting to start the machine since this is a Libvirt operation. I've already been seeing much more useful error messages in the logs. These new error messages should also be viewable in the web front-end via the error dialog box when something Libvirt-related goes wrong.

This does increase the Libvirt coupling in ECP a considerable amount. However, given the level of functionality that ECP would have without Libvirt, I think it is a fair trade off.