Showing posts with label exceptions. Show all posts
Showing posts with label exceptions. Show all posts

Thursday, October 21, 2010

Exceptions From Within

How do you properly raise and handle exceptions? This seems like a straightforward design problem. Accompanying exception handling in your code are questions about depth. For example, function a() raises an exception. Function b() calls a(). Should b() handle the exception? Or, should the exception continue outward in the call stack? The distance in the call stack, between the exception being raised, and the exception being caught, or handled, is up to the developer. This includes raising exceptions from within the exception handler itself.

Are these concerns a matter of coding-style? Or, can we establish a pattern that helps us decide where to raise and handle exceptions? Defining a generic exception handling pattern is hard to do because anything has the potential to raise an exception - intended or otherwise. From a functional perspective, exception handling is all about preventing unexpected events from disrupting your program. From a coding-style perspective, exception handling is all about knowing exactly where any given exception originated. Both are hard to do.

Picture the low-level functions in your code. The atomic functions that don't call anything else. These functions will, at one point or another, raise exceptions. The exceptions raised here will propagate upward in the call stack. Think of an exception as signal that notifies the rest of your application something went wrong. The signal is received at each point in the call stack. The only way an exception will stop propagating upward is if a handler for that exception type lives at that point.

Now, picture a higher-level function. A function that calls other, low-level functions. If this higher-level function knows about the exceptions that might be raised by the functions it calls, it can define a handler for them. This is the how exceptions are handled from a context outside that of which they are raised. All the handler needs to know about are the types of exceptions that might be raised. This is a key exception-handling concept - they are classes and therefore represent a type of thing that can go wrong. Any number of problems may cause an IOError to be raised. The handler only cares that it is an IOError. Anything else will continue upward in the call stack.

An exception handler has two components. The first, a try block, attempts to execute something. The second, an except block, conditionally runs if the first block fails. Is it plausible for us to raise exceptions from within the try block? Imagine an exception handler that handles a TypeError. If the try block itself raises a TypeError, the except block will handle it and the exception signal will stop propagating.

Raising exceptions from within the try block can reduce code verbosity. If I call a function that may raise an exception, you call that function inside a try block. Inside that same try block, you may call another function that doesn't raise an exception. For instance, the function might not return what you expected. In this case, you want your except block to run. It wont, however, because no exception was raised by the function call. Rather than alter the low level function, you can check the result and explicitly raise the exception that enables the except block.

Although this use method of exception handling is useful, it doesn't always feel right. Exceptions are indeed better suited for propagating outward in the call stack instead of being raised and handled in the same statement. Another drawback is that you are potentially blocking exception handlers even higher up in the call stack that might take a different approach to handling the same type of exception.

Friday, May 14, 2010

Exception Meaning

In object-oriented programming languages, there is a special type of abstraction called an exception. Depending on which language you're using, exception instances can be raised or thrown and caught or excepted. The terminology used is somewhat irrelevant as the general concept remains the same; exceptional behavior has taken place and needs to be dealt with.

Most languages provide a set of standard exceptions that are always available. These exceptions typically build on a base exception class. For instance, in Python, all standard exceptions inherit the base Exception class. This means that developers can also write their own exception types. So if a standard set of exceptions already exists, why bother writing new exceptions?

Exceptional behavior in programs don't just randomly happen. As always, there is a cause and effect. If your program is behaving exactly as it was intended, there is a reason it is doing so. Imagine you have a Python program that only raised Exception exceptions. You could pass some data indicating why the exception occurred to the exception instance. This way the handler could figure out what went wrong.

What if the handler doesn't know what to do with the exception? That is, it doesn't know how to handle the error. The exception handling behavior has been invoked for no good reason. Consider looking up a dictionary key that doesn't exist. In this case, you'd get a KeyError exception. This type of exception has real value because it knows exactly what went wrong. The normal behavior provided a key and expected a value in return. The correct exceptional behavior can now execute because it knows exactly what went wrong.

If the normal behavior doesn't define an exception handler for actions that might raise an exception, any exceptions that are raised will propagate outward in the call stack. So, if we have one function that calls a second function, and that second function raises an exception, the first function will receive the exception if unhandled by the second function. Say the second, nested function raises a KeyError exception. The first, outer function better be prepared to handle KeyError exceptions.

The problem here is that KeyError exceptions may not have any meaning in the first function. If the second function has a responsibility to do a dictionary lookup, it should take on the responsibility of handling KeyError exceptions. The second function could raise a custom exception in the event of a KeyError exception that has some meaning in the context of the first function. It is useful to provide a smaller set of possible exception types as the context grows outward as this gives exceptional behavior more meaning.

Wednesday, March 31, 2010

Polymorphic Libvirt

The Libvirt virtualization library is great for interacting with heterogeneous hypervisors. Depending on the hypervisor connection, most domains support the same virtual machine interfaces. For instance, you can you the same code to start and stop virtual machines using Libvirt for both KVM and Xen.

But when the operation isn't supported for a particular hypervisor type, you better be prepared to handle this scenario. Libvirt only defines a single exception type but we can determine that it is a hypervisor interface support issue by looking at the error code.

This is shown below. The following operation on the domain will fail on KVM.
import libvirt

if __name__ == "__main__":

conn = libvirt.open("qemu:///system")

for id in conn.listDomainsID():

domain = conn.lookupByID(id)

try:
domain.reboot(0)
except libvirt.libvirtError, e:
if e.get_error_code() == libvirt.VIR_ERR_NO_SUPPORT:
print "Cannot Reboot"

Tuesday, March 2, 2010

Exceptional Development

Exceptions are raised when an exceptional condition is true within a program. Most object-oriented languages define a base exception class which developers can use to derive custom exception hierarchies.

These exception hierarchies can grow to be quite large, even in production systems. This makes sense if we want to use the exception handling mechanism in a polymorphic way. We handle all exceptions at one level of the hierarchy, including all descendant exceptions, while ignoring all exceptions at a higher level up.

These hierarchies allow us to reason about what as gone wrong. So using an aggressive approach to exception handling during initial development might make a lot of sense. Construct a list of every conceivable exceptional path that isn't part of the successful path. Generalize some of the exceptions so you avoid unnecessary duplication.

With this first initial exception hierarchy, you can pound out a first iteration quickly. The fact that none of these exceptions in the hierarchy are raised is an indicator that the iteration is complete.

Wednesday, November 18, 2009

External Exception Handlers

Erica Naone has put together an interesting entry on new software called ClearView. The software was created by Martin Richard's research group at MIT.

The goal behind this software is to key a watchful eye on running applications. Now, these application would have to be of some significance, otherwise, it probably would be worth babysitting them to this extent. But for applications containing sensitive data, something like ClearView can be very helpful.

The idea behind ClearView is to analyze the behavior of a given program while it is behaving as expected. This allows ClearView to determine what the rules of good behavior for a particular application are. Once these rules are broken, ClearView knows that the application is misbehaving and can address it. The real nice thing is, the fixes are applied to the running binary, not the source code.

The best way to picture ClearView from a high level perspective is to picture the running application as a piece of protected code running inside an exception handler. Obviously, this isn't in the code, and, isn't even real for that matter. But, it is similar to what happens. ClearView goes beyond what typical exception handlers do. It tries to fix the problem rather than point fingers. Maybe exception handlers inside code could learn from ClearView in the future, or maybe not.

Thursday, March 12, 2009

ECP and SQLObject

It seems that everyday we have a new reason to move away from using SQLObject as an object-relational mapper in ECP. The latest issue with SQLObject has been rather challenging to work-around. The problem comes from the ErrorMessage class defined in SQLObject. Here is what the class looks like.
#SQLObject ErrorMessage class.

class ErrorMessage(str):
def __new__(cls, e, append_msg=''):
obj = str.__new__(cls, e[1] + append_msg)
obj.code = int(e[0])
obj.module = e.__module__
obj.exception = e.__class__.__name__
return obj
The problem we are experiencing with ECP is the fact that this class always raises an IndexError. The reason being, the ErrorMessage.__new__() method makes the assumption that the 0 and 1 indices will always be available in the e parameter. The e parameter is supposed to be an instance of Exception.

The question that now arises is how do we handle this? In this case, we have exceptions being raised by other exceptions. The ErrorMessage class could simply be fixed by adding exception handling for IndexError exceptions. However, now that the error is fixed, how do we ship this fix along with our application? ECP will currently install SQLObject from pypi. One solution would be to build our own SQLObject package and point the ECP setup to a custom repository that contains this patched-version. One problem I find with this solution is that it could potentially introduce a myriad of other deployment problems.

Another solution is to perform the patch inside of ECP. In this scenario, we don't actually patch the SQLObject package. The SQLObject package would remain as is on the system so that other Python applications using SQLObject wouldn't experience any side-effects as a result of ECP providing a different SQLObject. And this is the approach we are taking. Once ECP has started up, we import the mysqlconnection module and replace ErrorMessage entirely. Here is how it is done.
#ECP approach to patching SQLObject.

from sqlobject.mysql import mysqlconnection

class NewErrorMessage(str):
def __new__(cls, e):
if not isinstance(e, Exception):
e = e.args[0]
else:
try:
dummy = e[1]
except IndexError:
e = e.args[0]

obj = str.__new__(cls, e[1])
obj.code = int(e[0])
obj.module = e.__module__
obj.exception = e.__class__.__name__
return obj

mysqlconnection.ErrorMessage = NewErrorMessage
What is shown here is a new implementation of the ErrorMessage class; NewErrorMessage. The interface of the original class is kept in tact. What has changed is the exception handling inside the exception. We first test if the e parameter is in fact an Exception instance. Next, we test for IndexError exceptions and rebuild the e parameter if necessary. The method then continues on as in the original implementation. Finally, we then replace the ErrorMessage class with NewErrorMessage. This all happens in enomalism2d so that the new error message class is available right away, before it is actually needed.

As an afterthought, I'm wondering what led SQLObject to this issue to begin with. That is, how can a class so tightly associated with exception handling be the culprit for bigger problems such as this one? Is it that this class is taking on too many responsibilities and thus adding to the risk of raising unforeseen exceptions itself? I wouldn't think so. The ErrorMessage.__new__() method isn't exactly overwhelmed with code. Besides, the exceptions defined in ECP do a fair amount of work when instantiated (including interacting with SQLObject). When the ECP exceptions are raised, they never raise inadvertent exceptions.

Perhaps special care needs to be taken when defining exceptions that do any work. If nothing else, SQLObject provides us with a lesson learned. As developers, we need to be absolutely certain that any given exception we have defined ourselves can be raised under any circumstances. They cannot fail. It would obviously be nice of no code failed at all throughout an entire application. That is obviously not a reality though. The code will fail at some point and having stable exception to deal with will make your code one step closer to being fail safe.

Monday, March 9, 2009

Trying to break Python memory consumption

I've been trying to find consistent method to raise a MemoryError exception in Python and have so far been unsuccessful. I'm mostly interested in real-world usage scenarios such has executing huge number arithmetic. Here is what my latest test looks like.
#Example; Massive memory consumption.

if __name__=="__main__":
list_obj=[]
cnt=0
while cnt<10**24:
list_obj.append(cnt**2)
cnt+=1
Here, we are simply appending progressively larger integers to a list. Executing this on my Ubuntu laptop results a huge increase in memory consumption. Because we are constructing a massive list object, this would make sense. However, a memory error isn't raised for me. Figuring that my laptop has enough memory to not let any kind of memory mishap take place for awhile, I fired-up a virtual machine with much less memory capabilities and executed the same program. The Python process ends up being killed. No MemoryError exception. What gives?

If I want to handle memory errors in Python, how can I deal with it if the process is terminated before I get a chance to?

Wednesday, February 25, 2009

Interfaces and errors

The basic idea behind defining interfaces and classes that provide those interfaces is to create a contract. This contract specifies that any instances of this class will carry out the behaviour specified within the contract faithfully. It is said that if some instance that provides a given interface, any context in which that instance is used that requires behaviour specified in the contract, the instance is behaving as expected. The instance is error free.

On the other hand, if some behavior is invoked, on some instance, in some required interface context, and the instance does not conform to the contract, the instance does is not behaving correctly.

For example, here is an example of a functional provided interface.

Here we have a simple interface called IDoable. This interface specifies that any classes that provide this interface must implement a do_something() method. The Doer class that provides this interface is valid and functional because it implements the do_something() method.

What about an invalid provided interface? Consider the following non-functional example.


This time, the Doer class does not faithfully carry out the contract specified by the IDoable interface. So if an instance of Doer is used in the context of a required IDoable interface, this will not work as expected.

Depending on the implementation language, the instance may not even have an opportunity to behave in certain contexts because of the lack of one or more required interfaces. Other languages, don't care what type of instances are used in a given context let alone what interfaces they provide. So in either case, the failure of an instance to provide an interface can be taken care of. There will be a compilation error or a runtime error.

This still leaves us with some unanswered questions. In the event of a runtime error, due to the lack of a required interface, is this the result of a design error or should the instance simply not be there. In the case of the design error, chances are that the instance is valid in the context, there is simply a mistake in the implementation of the class. This could in fact be as simple as a missing operation or an invalid operation signature. What about when the instance has no business being in the context which is attempting to invoke behaviour on it? This is obviously not a design flaw in terms of the required interface and the instance that does not provide it. How do you deal with such situations? In interpreted languages, this can be a little trickier. Then, the question is why was this instance placed in the specified context in the first place if it doesn't belong there? We know that all classes that provide the interface in question are implemented perfectly.

There really is no answer. You really have to look at the architectural layers of your application at this point. If instances that cause these types of mishaps belong to the problem domain, you could be in good shape. If hierarchies of problem classes are constructed in such a way as to provide a means to handle these interface errors in the problem domain as well.

Other problem developers can stumble over is the over-reliance on interface conformance. When instances provide the necessary interface in a given context, this doesn't mean that the invoked behavior will execute error-free. Should these errors be handled outside of the problem domain. Again, this is really dependent on your architectural layers and how they are implemented. If rigorous testing results in flawless class implementations in all the problem domain and the application domain, we could then tie the interface errors to the problem domain. Anything else can be considered at the application level and be dealt with accordingly.

Building all these interfaces around the problem domain is a lot of work. Sometimes it is simpler to just define some common exceptions and deal with them, be it application or domain layer.

Tuesday, February 24, 2009

New ECP exception and enhanced state restoring behaviour

Over the past few days, the ECP team has made some notable enhancements in the trunk. The first being the addition of a new exception called E2LibvirtError. As the title suggests, this exception is raised for Libvirt-related issues. The Python Libvirt library already defines an exception class. However, there are many types of errors that can happen from within Libvirt. The idea behind this new E2LibvirtError class is to provide better information when something bad happens in libvirt. For instance, in the Python Libvirt library, there is only one exception type. If this exception gets raised, a short message is displayed. This is the default message that gets initialized with the Python base exception class.

The problem here being that Libvirt can manage several different hypervisors on any given system. Thus, there are several layers within Libvirt in which something can go wrong. In ECP, the Libvirt exception is caught, and the generic message is recorded.

The new E2LibvirtError exception exploits additional exception information encapsulated within the basic Libvirt exception instance. I don't mean encapsulated in the traditional object-oriented sense. I mean the information is there, and ECP should use it for the benefit of the end user. The new ECP exception, when instantiated, will take several error codes from the original Libvirt exception and produce a much more meaningful error message.

This leads me to the changes made in the restore_machines_state() functionality. The rationale hasn't changed, only the implementation. We simply handle table existence and local machine existence detection much better than the current version. If the function finds a machine that is not running and it should be (because that was the state the machine was in when ECP last shut down), it will attempt to start it. We've already added the new E2LibvirtError exception handling to this function when attempting to start the machine since this is a Libvirt operation. I've already been seeing much more useful error messages in the logs. These new error messages should also be viewable in the web front-end via the error dialog box when something Libvirt-related goes wrong.

This does increase the Libvirt coupling in ECP a considerable amount. However, given the level of functionality that ECP would have without Libvirt, I think it is a fair trade off.

Saturday, January 17, 2009

ECP e2_exception extension module.

The e2_exception extension module for ECP allows administrators to view exceptions that have been raised by ECP. By default, exceptions are stored in the database. This means that they can be viewed again at a later time. The e2_exception extension module, when installed, provides a table view of exceptions that have been stored in ECP.



You can also view the tracebacks for each individual exception.



The e2_exception extension module also adds exception resources to the ECP RESTful API. Clients can query for stored exceptions, and delete them. Future versions of the extension module will add more features to the API as well as visual enhancements to the GUI.

Monday, September 8, 2008

Building an exception based API

Building an exception based API is useful for several reasons. One benefit I'm going to discuss here is the ability to model your system as a mesh of signals, or events. When dealing with exceptions, most people tend to lean toward using signal in the terminology. I, however, don't which term is used. The goal of creating an exception based API is to move away from the traditional synchronous invocation. Using events or signals achieves this.

I'm going to move out of the problem space here for a moment and into the solution space. In a programming language, when an exception is raised by some function or method, the normal flow of control is interrupted. I most cases, this is caused by a more primitive error such as in index error or name error. The flow of control then propagates outward, toward the highest level in the code. Along the way, there may be exception handlers. There are specialized code blocks that will execute if the appropriate exception is raised will the code in the context of the exception handler is executed.

The idea I'm trying to get across here is that exceptions can easily be modeled as signals. The code that raises the exception often has no idea if this exception will take place before the code is executed.

One approach to modeling an exception based API is to build a custom exception for the various flows of control. For example, we might model something like the following for a RESTful framework:


Here we have a Application object that represents application, or controller. The application object interacts directly with the Request object. The request object represents the request that is sent to our application from some client. Most web frameworks already support this abstraction out of the box. The remaining objects, GET, POST, PUT, and DELETE, are all modeled here as exceptions. Although there is no way to differentiate the regular objects from the exceptions, this is not the focus of this diagram. The focus here is to show how exceptions can be used as more than just exception handlers.

Another point to note, the sequence of messages dispatched from the Application object are not meant to represent the ordering of the message. It is meant to represent to potential messages that may be dispatched in response to a client request.

Monday, July 14, 2008

Exception driven development

In software, an exception is an event that occurs unexpectedly.  An exception is exceptional behavior.  In any modern object-oriented programming language, developers can define their on exceptions that inherit from the languages' base exception class.  Defining your own exceptions and using them as flow control mechanisms in your code, if used cautiously, can yield very robust code.

The main use case for using this pattern is to simplify large branching flows of control.  For example, say you have defined some method m1 that is invoked in another method you currently implementing, m2.  The control flow in m2 is determined by the outcome of m1.  There could potentially be several new branching flows in m2 each with subsequent levels of branching.  This would eventually make m2 unmaintainable.  An alternative is to implement m2 to only handle the basic success case.  Any other cases are handled by custom exceptions.  In this case, the invocation of m1 would have to handle everything that m1 can raise.  Other than that, it will carry on as normal.  m2 shouldn't care so much about m1s' failures.

The exceptions handle all the exceptional flows in you application.  That is the basic idea.  You have to use your imagination to build an exceptional vocabulary in the domain in which you are working.  The exercise of building a vocabulary of what can go wrong in your system is a useful tool in its own right.