Things You Don't Need

It's hard to get rid of things, even harder to not acquire them in the first place. Perhaps that's because the new thing can only be seen as a positive change - its acquirer blind to any down sides. Dependencies form in strange ways. Even if it's probably bad for you and everything around you, it still manages to root itself into the new environment. You almost have to kill it to get rid of it. It's like you need the thing you don't need in order to realize you don't need it. Maybe that's why the wealthy are generally unhappy? They have the thing, realize they don't need it, and yet, cannot get rid of it. They probably can get rid of it, actually, they just choose not to for one reason or another, and it's also a hard decision to make in the first place. Why get rid of something when it can be kept around? We must have acquired it for some reason or another, right? To enhance our existence. To fix an already existing problem we've been having with something else we've acquired? Could be anything, really, we tend not to keep track of this stuff do we? Maybe we should. Maybe paying conscious attention to what we need, and what we do not need is an essential daily exercise.

Designing For Big Data

Information is all around us. We're producing more data on a daily basis than ever before, and there is no sign of this phenomenon slowing down. Most likely, this is due to the increasing number of network-connected devices. Everyone has a laptop, a tablet, and two phones it seems — each generating data in their own unique ways. Going beyond personal computing devices that we control for the most part, there is a growing number of unmanned devices that are pushing data across networks. These autonomous agents are usually sensing data about their surrounding environment, and publishing the raw data for either some centralized software that knows how to make sense of that data, or in a more distributed fashion where other agents might want to know what's going on. These autonomous agents have a profound impact on the global measurement of available information because they never sleep. There is an endless stream of data about the current state of the world. We're measuring just a portion of our existence through these autonomous tools, a tiny fraction of what we could potentially measure. Imagine what the automated information stream will look like 5, 10, 20 years from now...

Tabling Ideas

I once had an idea for a new user interface feature. It would inform the user about the health of their resource quota by showing a small indicator in the corner of the screen. The key to this component was that it was small, unobtrusive, and visually intuitive. Colors and icons would do most of the work in indicating any change in health. It was never implemented. Was it a valuable feature to have? I thought so. But the reason it was never implemented was because the idea was tabled — pushed into the background of our then development efforts. That's where it stayed. I came up with the idea fairly early on in the development life cycle. There wasn't a huge code-base to contend with yet, and so less chance of causing problems by introducing a new feature. But all that changed — as the code base grew and the feature set ballooned, my notification idea collected dust. It grew old, frail, and obsolete. Eventually it died.

Building Generic Links

I've been wondering lately — what good having the server-side web application generate URLs when the client side code is just as capable?  Indeed, there is no way we'd ever want to hard-code the application URL paths inside the Javascript code.  So why even entertain the notion?

Speaking in terms of overhead — with each URL on a given page, some CPU expense is incurred when constructing each URL.  I'm making a presumption here in assuming URLs are in fact being constructed and not just hard-coded in templates.  In reality, the overhead is inconsequential, but in some generic circumstances, I see absolutely no reason why these URL's can't be assembled locally on the user's browser.

Why URLs Are Constructed
It's for the best that URL's passed to the browser by the web application are constructed rather than stored statically.  Ad-hoc construction of URLs doesn't solve any problems either.  For example, if on one page you're formatting a string that'll be returned as a URL, and on another page, another set of URL construction functionality, it's difficult to establish patterns of reuse.  Sure, we're able to share some of this code among our application's components — but why should we have to define this capability ourselves?

Look no further than Django's URL resolution architecture for an example of how URLs are specified in one place and one place only.  These are base URLs — they can take on different forms by inserting values into path segments.  For instance, search strings or object identities.

The rest of the Django application can construct — resolve in Django-speak — the URL to use.  What I really like about this methodology is that URLs are treated just like classes in object oriented programming languages.  The URL blueprint is defined once, individual instances of the URL are constructed as many times necessary.  Some URL instances are simple — they take no additional attributes. They have no path segments to fill in or no query strings to append.  In the case of more complex URL instances, we have the URL resolution framework at our disposal — a consistent approach to instantiate our URL blueprints, passing in any additional details the URL class requires in order to make them unique and identifiable.

The theme here being that URLs can, and should be, a generalized concept — just like other systems we build.  We're able to abstract away much complexity into classes, instantiating objects, letting the class machinery take care of eliminating what would be otherwise redundant repetitions of URL construction code.  Sticking with the Django URL resolution architecture as the example, because it illustrates so well how URLs can be constructed, consider user interface code.  The templates.  The Django template system comes with a URL tag to help instantiate URLs directly in the user interface.  Again, when templates are rendered, we're injecting URL instances into the template.  It's from here that we can follow the logical extension into generic URLs in Javascript.

Javascript And URLs
Javascript can do a lot with URLs that the web application, running on the web server doesn't necessarily need to deal with.  But why would we want to use Javascript?  Shouldn't we stick with the URL construction kit offered by the web application framework?  Well, for the most part, I'd say yes, it is a good idea to use things like the url tag, if you're building a Django template.  Stick with the infrastructure provided for building URLs and you'll never worry about whether the correct URL is passed to the user interface.  Just reference the URL blueprint by name and let the URL instance take care of the rest.

User interfaces in web applications follow patterns.  That is a given.  From one page to the next, controls repeat themselves.  Things like pagination come to mind. Any given page with a list of objects on it uses pagination controls as a means to avoid dumping every object on a single page — making it impossible for the user to navigate in a cogent manor.  The common items, or, generic items rather, are the list and the paginator.  The list of objects and how they're rendered can be generalized inside the Django template.  As can the pagination controls.  These controls move back and forth through a set of objects — usually by appending constraints on the URL.

So how does this paginator change from one set of objects to the next?  How does it distinguish between one type of object list and the next?  Well, the reality is that it doesn't change much for different object types, or, at all.  Forward in motion and backward in motion.  Generic behavior that doesn't necessarily warrant using the URL resolution system.  Any Javascript code that runs on the page already knows about the current URL — just look it up in the window object.  We know how to move forward and backward through a list — just update the current URL.  For instance, the browser URL is simply updated from /objects/page1 to /objects/page2.  There isn't a need to invoke the URL construction code for this — let the client take care of generic things like this where it makes sense while saving server CPU cycles for more important things.

Software Luxury

Many things in this world can be considered luxurious — cars, homes, hotels, and yes, software too.  Attributes that establish an object as luxury include things like the material, the cost, and the brand.  It's difficult, sometimes, to state why exactly something is luxurious — at least in terms of visible properties.  We say things like, "it's a Rolex" or "it's a Mercedes" — the brand in it's own is luxurious.

Products that don't have the recognizable brand name to back them might be of outstanding craftsmanship and quality.  Yet, these things aren't considered luxurious items.  Perhaps they're used in industry to perform a specific task — instead of tooling around on a winding round or displayed as a decorative ornament.  Aside from the brand name, how can we identify luxurious things?  This is particularly interesting when it comes to software because we think of software as a tool more than a sumptuous want.

Tools and toys
To secure more insight into what makes something a luxury and what makes something a necessity, it's helpful to classify things more broadly.  A toy, be it a boat, a ski-doo, or a train set, isn't a requirement for day-to-day living.  They're a form of entertainment — something to keep us occupied when we're not busy other duties.  Yet, not all toys are luxurious.  When you're busy playing with rudimentary item, the desire to progress upward is unavoidable.

Tools are necessary for any trade.  They're how we're able to succeed at any task.  Tools get the job done, so even if we're not doing something enjoyable, tools make the job a little more bearable.  Tools are the screwdrivers, the text editors, the trucks — the things we need when we're not busy entertaining ourselves.  Once again, there is always room for improvement, a way to execute the task more efficiently or more comfortably.  If only there were a more luxurious shovel.

These broad categories — tools and toys — occupy the vast majority of our wakeful existence.  And they don't exist irrespective of one another either — they both share the same drive to improve, to become even more joyful, or perhaps less painful.  And that's just how the tools and toys of our lives evolve — through the constant pressure to become more integrated with what we're accustomed to.  Or, perhaps we're not accustomed to something luxurious, but that doesn't stop us from observing what luxurious items do, the joy they bring, and try to mimic that in the next generation of lower-end products.

Software necessity
Software necessity is hard to define — at least in a broad sense.  If you're performing an accounting task, you're probably going to need a spreadsheet.  If your doing development, you're going to need an editor.  These things are absolutely necessary, but they don't address the more granular idiosyncrasies associated with them.  For example, most spreadsheet applications have the required functionality built-in — the formulas, the importing capabilities, and so forth.  But what about defining your own formulas — what if your organization has a standardized set of formulas to plug into the data?  This is a specific requirement the application must support.  A necessity.

Software tends to carry a more stringent definition for everyday work necessities.  It clearly isn't enough to say I need a text editor to do development work.  You're probably used to features of a specific editor or IDE.  So this doesn't mean you can't do your work, it just means you can't do it effectively without contrived workarounds.

The necessity of software goes beyond simple features — checklists that determine the usefulness, the productivity one can expect from a given software package.  There is the matter of where software runs, what protocols can it support — things of that nature.  If an organization is determined to stick with legacy software that isn't web-friendly, they risk being left behind.  Staying current with the times is a necessity not just because it's allows new innovations, but because the competition will no doubt be on top of it.

There are two dimensions of software necessity — the bare minimum fundamental stuff that is requirement to do the job.  Then there is the the more specific aspects that allow you to do the job better.  These latter additions to your systems, small incremental changes and adaptations that transform a monotonous job involving a dozen steps and hide that complexity behind an intuitive interface — these are luxurious relative to the first iteration of the product.

Developing luxury
All software, if it is to survive, must be continuously developed, continuously improved and fleshed-out.  Are we developing luxurious software by making these improvements?  Or are we simply raising the bar of necessity?  Because, as time goes by, what was once thought of as a luxury, turns into a diminutive, core part.

If luxury is a definitive property of something whereby it goes above and beyond what the thing was designed to do, maybe luxury has no place in software development.  Remember, software is about simplicity — without simple solutions, we're stuck in an even repeating loop of frustrating experiences where we're trying to do something that goes beyond project scope.  Superfluous features, and conversely code, should be pruned.

But who's to say that creating luxurious software can't be reduced to simple improvements that aren't intrusive to the user or the developer?  We might even go so far as to say that luxury in software is created by removing things — by taking complexity and replacing it with simplicity.  Maybe instead of focusing on feature checklists, software projects should think about how the user will perceive it in terms of necessity.  A reductionist approach, removing superfluous constructs from the system, yielding something small in size and high in value.  The must luxurious software is that the fulfills the necessities, unlike other items in the world where luxury is the nice-to-haves.

Time To Refactor

Refactoring existing code makes better software.  Martin Fowler has championed this concept and as a result, developers look at refactoring code as an essential part of the process — not something to do on slow days.  Of course, idealizing over code design is one thing — making it a reality is another.  Unfortunately, refactoring doesn't happen for one reason or another — usually there isn't enough time.

But what about the programmers that have to stare this code in the face — day in and day out, watching unruliness assume control?  They're the ones who see the problematic consequences of letting ugly code grow and metastasize into other subsystems.  Programmers probably don't decide there is no time to for refactoring.

If there is no time to refactor code, when will it get done, if ever?  Understandably, features and enhancements are expected.  Expected yesterday.  Maybe there is a way for software developers to convince stakeholders that refactoring is the only option left.  Maybe we can treat, somehow, refactored code as a feature?

Code gains weight fast
Code gains weight at an alarming rate.  You might start off your hacking session with a small set of classes, maybe a couple utility functions.  It takes only a couple hours for this lightweight setup to transform itself into a bloated plate of spaghetti code.  Not because you don't know how to write elegant code, but because others are depending on you — it just needs to work.  If we wanted beautiful code the first time around, we're in the wrong business.  Refactoring is about improving existing code, not code that has yet to be written.


As programmers, we idealize solutions — before sitting down and typing it into the editor, we've at least, if only briefly, conceptualized a few alternative approaches to tackling the problem.  This isn't considered big, upfront design — it's more of a personal game plan to help spring us into action.  But, since this is only a quick mental note — not using real code and a real editor and not running real tests — chances are the some aspect of our plan is wrong.

The trouble is, we don't know that some aspect of our ideal implementation is invalid until it's concretely implemented and executed.  Sometimes our code doesn't even reach the point of execution before we've realized it doesn't fit.  Sometimes all it takes is a glance at the interfaces you're working with or the documentation of another component dependency to realize that there is a sudden change in plan.

This sudden realization means that the nice, natural flow that we had envisioned, the one that motivated us to get going in the first place, has been scrapped, ups the urgency of the situation.  Maybe the entire metal workflow isn't thrown out, but the calm confidence we had slowly dissipates.  We're probably not following a waterfall approach — so we need to adapt quickly — pound out something that'll work.  Otherwise, the forward motion of the entire project is jeopardized.

Fixes to sudden disruptions aren't difficult to implement, but they're not the most elegant or efficient solutions.  And so the code gains weight.  It works, but it's become heavier — in terms of raw size and in logical baggage — all within a few hours.

Features are released on time
Software projects have a brigade of developers, all with the attitude of making sure things get shipped when expected.  Even if that means not setting aside some refactoring time.  This process, this seemingly endless cycle of pounding out lines of code, is very rewarding.  Getting stuff done on time is a recipe for success in any software project as far as I'm concerned.

This is the habitual part of it for the programmers — who wouldn't want repeated success by continuously pleasing customers and other stakeholders by shipping early?  Don't fix what isn't broken, right?

And so life goes one — each cycle of the development process yielding a new and enhanced version of the software that exceeds customer expectations.  It's hard to change a good thing while it's good — making decisions to refactor code can seem unjustified.  Is there another way to squeeze in some code improvements that don't necessarily please the client directly?

Well, small incremental improvements can typically be tucked-into sections of the software system during the development process.  These are low-risk, low-effort improvements that don't have an immediate impact on code comprehensibility but instead have a concerted long-term impact.  The trick is — making these small incremental refactorings — is really only beneficial if they're a habitual task of each and every programmer working on the project — cumulative change for the better.  But this can be overridden, mentally, by that other little habit — satisfying the customer by shipping on time.

Forcing the issue
Two competing ideologies vetted against one another — the ship on time versus the refactor messy code — lead to one of two places.  Perhaps the worst thing that can happen is programmers get caught up in cleaning up code and as a result, deadlines for requested features tend to slip.  What we have in this scenario is some nice looking code and a frustrated group of stakeholders.  The product is better off now that the mess has been tidied — paving the way for future ingenuity — but how do you explain that to non-programmers?

The second outcome, and perhaps a good selling point for investing time in refactoring code, is a garbled mess that can't be updated with new features or enhancements to existing features.  There is an illusion that this will never happen because it takes a while to happen.  A misconception of those involved in software projects who don't write any code is that code is an immutable thing — once it's written, it's there and it should just work.  Ask any software developer and they'll tell you that any body of code is a living organism — one that must be fed and cared for.

So, forcing the need to refactor.  The best way to explain it to those involved in the project but do not understand the concept of refactoring code is to present it as a feature.  Or, sometimes even more effective, present it as a blocker for new, potentially interesting features.  This second approach takes some inventiveness, but is ultimately the most effective way to communicate the state of the code base to non programmers.  Think features they'd like to see and what would stop them from happening in terms of the current state of the code.

The reason this approach — let's call it proactive refactoring for features — works so well as a communication tool is that you're taking a deficiency in the fundamental code structure and presenting it in business objective.  Rather than trying to introduce the philosophy of refactoring code — because I can tell you right now they're not interested — you're taking a valid business concern and presenting it in a cohesive way.

One final suggestion on proactive refactoring for features — back your claims up with evidence.  Not a written words, but a quick little prototype that'll make very obvious the problem at hand.  Nothing says urgency like proof that a potential money making feature might not work in our product down the road.

What URIs Say About Resources

The web today is made up of boundless resources — each identified by their URI.  I favor the term URI over URL because it denotes identity.  So what information can we gather from the URI alone?  Are we better of calling it a URL since the mail purpose is to do a lookup?  I don't think so because the fact that URIs are used to look something up is implied knowledge — it's the identity of the resource we want to learn about.  But, can we attain this type of information from the URI alone or is it a meaningless question?  URI's should be designed to advocate foreknowledge of what the resource is.

Uniqueness
URIs are unique.  That is, there is a one-to-one mapping between a URI and the resource it points to.  There are, of course, exceptions to this rule.  For example, you might have a radio station web application that displays the currently playing artist.  During the artist's air time, they might have a artist/current URI that points to the artist's detail page.  Alternatively, there might be a single canonical URI associated with the artist's page — artist/123 for instance.

So in the case of the former — where the artist can have two URIs pointing to their page — there is no one-to-one mapping of URI to resource.  There might even be more than two URIs pointing to the artist's page — charts/top, for instance.  But these URIs are unique in that they're referring to one resource.  The underlying resource might change — the artist/current URI stays the same but the resource it points to will change frequently.

The artist/current URI is an example of a virtual resource — to the external agent, this appears to be where the resource lives.  But this isn't where the resource lives — it isn't it's canonical URI.  The URI artist/123 is static and probably will never change.  The virtual URI points to the canonical URI.

To better illustrate this concept, let's talk in hockey terms.  Imagine the center for the home team.  He is number 15.  So his canonical resource URI looks like home/15.  Now imagine that you want a URI for the home team player currently in possession of the puck.  Our star center has the puck — so we can represent this as a virtual URI — puck/control.  There isn't anything special about this URI — it just contains some logic that points to the home/15 URI.

Meaning
So it turns out that URIs carry some important information after all.  And this is what I'm trying to figure out.  Exactly how much information is of value to the reader of the URI?  In theory, every URI on the web could be some arbitrary string — CD4F2ACF4, for example.  It wouldn't matter because information is properly linked to other information.  The readers don't care what the URI is — they only care about the anchor text.

I think this might have some degree of truth behind it but the reality is that people do care about the URI and what it looks like.  I know I do.  In fact, before clicking on links, I find myself hovering over them to see where they go — trying to examine the URI to guess it's worth before I go there.  Mind you, I take a very active interest in URI design — so I doubt every single user will scrutinize — or even care for that matter — what a URI looks like.

But it turns out that even the most arbitrary URIs make subtle attempts to attach meaning.  Consider our earlier URI — artist/123.  What do we know about this page before visiting it?  Even if you're a lay user — you're probably able to guess that it has something to do with an artist.  We achieve two things with this URI — the vocabulary and the multiplicity.

The vocabulary establishes the kind of thing users can expect to see should they choose to follow the link — in this case, an artist.  The multiplicity is established in two ways.  First, we're explicitly choosing the term artist, not artists.  Second, the reader can see the trailing identifier — 123.

So the most meaningful piece of information in this URI is artist.  The arbitrary part, arbitrary from the reader's perspective, is both meaningless and important at the same time.  The arbitrary identifier assigned to the resource is an important part of the URI — it's what makes it canonical.  The number itself has no meaning to the user but it has utility in sharing that URI with others.

Evolution
It turns out that URI design has evolved quite a lot since the emergence of the web.  We've seen a lot of resources — immeasurable resources — created over the years.  This directly impacts our ability to create meaningful URIs for users.  If it were simply a matter of incrementing the resource count once a new resource is created, we'd be all set.  Unfortunately, that isn't true at all.  There are new types of resources that need to be created as applications and organizations evolve.  These new resource types are going to form an ever more complex mesh of relationships — links to other resources both new and old.

These new resource types — once invented to help solve the technological problems of the day — will also need virtual resources.  The virtual resources are the logic of the web — they're not real data, just pointers to other canonical resources that store the real information that external agents update and use.

Keeping URLs meaningful for users is important as available information continues to expand.  If we succumb to churning out completely arbitrary URIs, we're taking a step backward.  Likewise, the URI itself is real data that needs to be shared and passed around — so we must be careful to add meaning, but not too much.

Widgets And Frameworks

Web frameworks are important because today, the target user environment is the web.  Why is this?  Because the web means anyone can use your application.  There are no barriers to installing the software and no restrictions on the hardware.  All the user needs is a decent web browser.

The web application framework itself is relevant because it enables developers to write solid software by reusing common components.  Common patterns used in all web applications.  Without these tools supplied by the framework, nothing would ever get off the ground — a diffuse time to market for smaller development teams.

What patterns are used in web frameworks?  Database connectivity, URL mapping, authentication.  The list goes on, each framework providing their own flavor of these major components.  Each is considered a pattern because any web application needs to implement it.  This quickly burdens the developer.  The role of a web framework is to parameterize these patterns and make them accessible.

What role do widgets play in web application frameworks?  By widgets, I mean standalone Javascript toolkits that have a selection of cool user interface stuff.  This includes jQuery UI, Dojo, and ExtJS.  Making these Javascript widget tool kits — which are a framework on their own — integrate seamlessly into our web application frameworks is disjointed to say the least.

Not an easy fit
Web application frameworks don't give us all the front-end browser code necessary for a finished product.  If it were a matter of delivering only HTML and CSS to the browser, we'd be all set.  This isn't the case unfortunately.  Not if there is any kind of user interaction involved — which is almost certainly the case.  For the kind of experiences users have come to expect, developers need to employ Javascript or Flash technologies.

In theory, building these rich user experiences is fairly straightforward — alter the HTML templates and serve the additional resources — Javascript, Flash, etc.  This can be achieved without any headaches — granted the project is a small one.  There many resources to worry about, not as big a user base compared to something enterprise-grade.  Every aspect of the software is scaled back slightly — and so the cause for error is scaled back accordingly.  For example, having a very large Javascript file or dozens of smaller Javascript files can be hard to maintain.  Not on their own — but because Javascript is different — it depends heavily on the DOM.

So as far as the development process goes, we need sound approach to ensuring there is some degree of coupling between individual HTML templates and their corresponding Javascript modules.  The days of implementing a monolithic Javascript file are quickly leaving us behind.  Especially if given the challenges of managing external widget library dependencies.  Imagine an application with a hundred or more HTML templates — I can't think of an optimal method where a single Javascript module suffices.

Technically speaking, uniting widget frameworks and web application frameworks wouldn't be overly strenuous.  All this entails, and this is generally how they're put together now, is serving new resources and having your HTML content ask for those resources.  The place where widgets don't fit seamlessly into the web application framework is in the back-end the web application.  How does a given view or controller (whatever the framework terminology alludes) know about which widgets are used?  Are these widget configurations persistent?  For that matter, are they configurable at all from the web framework?  For me this is a problem — widgets are a big part of the overall picture — web application frameworks should accommodate widget frameworks.

Frameworks and forms
A crucial feature of any software with a web user interface is forms.  Forms collect user feedback.  They can be simple — a single input field and a submit button.  They can be complex — a registration form that nobody in their right mind would fill out.  Web application frameworks do a pretty good job with tools that both render and validate forms.

There is good reason for.  Web frameworks define the database schema for the application.  The database is populated by user input.  Forms.  There is an obvious connection between the two concepts — this is why frameworks take great care to supply the database with what it expects.

Take Django for example.  The Django framework has a forms module that'll assist with defining forms, the validation, and rendering the form widgets for the browser.  Not unlike how Flask uses WTForms to validate and present forms.  The Pylons project also provides developers with form processing capabilities — they require a little more effort than the other Python frameworks.

What really stands out is the connection forms and their individual input widgets have the framework itself.  They're an essential ingredient to the application — a formative piece of the database.  There is a mapping between the database schema and what is presented to the end user so they can populate it.  This fits perfectly with the ideal of the web application framework — to take a pattern and expose it as a cohesive API for the developer.  This concept is further exemplified by Django where we're able to point the form subsystem to a database model and let it figure out how the form should be rendered and validated.  This isn't so easy with more general widgets that aren't necessarily used for collecting input and storing it.

Widgets and data
If we're able to utilize the framework form processing tools to store data, how does the framework help us present data to the user?  Well, since a web application framework is only responsible for delivering the content to be rendered by the browser, all we've got is templates.  These assist in generating the markup necessary for displaying fancy Javascript widgets.

For instance, a common user interface pattern is to group logical categories using tabs.  Tabs are visually appealing and allow for intuitive navigation.  This is a top-level layout widget — meant to replace a typical sidebar navigation item.  Or consider a smaller widget — a progress bar.  This type of widget, unlike tabs, isn't a layout container.  It displays something specific about the application data.

What both layout container widgets and smaller application data widgets have in common is that they both depend on specific application details that reside on the server, within the framework.  The tabs widget needs to know what specific tabs to display and what content within each tab.  The progress bar widget probably depends on a computed number — derived from user data from the database.  So what is the preferred approach to displaying both widget types in any given application?

Navigational constructs are typically done in the template.  Maybe there is a nested div and a tag structure the Javascript widget is expecting — like the jQuery UI tabs widget.  However, the developer still needs to instantiate their widget when the page loads.  This means that the Javascript code needs to know about the template and what the rendered HTML output will look like.  This is somewhat error-prone as the developer will need to maintain dependencies between the Javascript widgets and the HTML structure they're expecting.

More challenging is the application user data presented by widgets — like a progress bar.  The progress bar needs the value to display — but where does it come from?  This might be a JSON response, returned from one of the application controllers.  Again, as with defining layout widgets, the problem lies in maintaining the connection between what the Javascript widget is expecting to see and what the framework delivers.  The problem isn't so much technical — it is more of a design problem.  One that leads to a loosely coupled system.  Too loose.

Solutions and challenges
Web frameworks already do a great job of taking common patterns on the server and making tools developers can use to efficiently develop code.  Might it be that web application frameworks have no business managing widgets?  Should this task be left to the templates and other static files?  I think there are patterns we're missing here that can be taken advantage of — it's still only an emergent thought.

One benefit to having the framework manage the connection between application data and widgets is the potential to be widget framework agnostic.  There are a lot of widget tool kits out there and a lot of similarities between them.  The commonalities across all toolkits — these are the interesting widgets.  The basic widgets that all web applications use in one form or another could potentially be integrated into web frameworks.

What might we do with the layout widgets to help standardize and improve the coupling between Javascript and framework data?  After all, layouts, if they can be standardized in some fashion, are a valuable asset.  You'll generally have a web application framework of choice and you'll almost certainly be creating more than one application.

I think layout widgets — stuff specific to the user interface and not necessarily what the application does — will be easier for frameworks to implement.  If the framework is able to introduce the concept of a widget, maybe storing it's configuration, then properly outputting the required markup and the required Javascript isn't a stretch.  For example, the configuration for a tabs widget would store what tabs are actually rendered, the contents of each tab, and should produce the necessary Javascript to instantiate it.

Widgets specific to application data — like a grid — aren't as easy.  This is because they're more toward what the application does.  Another challenge is that widgets generally depend on some data source like JSON that is generated from the user data.  This isn't so easy to stardardize on.  The framework would have to know about the data schema in addition to the visual configuration.  This means the application would have to store configuration that dictates what the widget will look like, where to get the data it needs, and how that data should be interpreted for presentation.  Not impossible, just more challenging.

Perhaps a good start to making web application frameworks a little more aware of patterns in how browser widgets interact with the framework itself is to discover patterns you're using.  How would you benefit if you're able to store some configuration that defines how and what the Javascript widgets do?  Every application is different — and so every combination of widgets used brings a new set of challenges.  This is how we make progress and is how the current features of frameworks came into being — by experimentation and pattern discovery.

Django Class-Based Views: A Design Perspective

The release of Django 1.3 introduced the concept of class-based generic views.  Prior to 1.3, generic views were written as functions.  What's the big fuss over class-based views?  What can they possibly offer that functional views can't?  Convenience, less code, and elegant design - a property that extends down through Django's core.   From a design perspective, class-based views in Django are a better way to implement common view patters with marginal effort than generic function views.  For me, having only used class-based views for a few months, I'm very impressed.  When I compare generic functions with generic classes, I'm amazed at the difference in quality.  This is one of those small enhancements that I think will change Python web development for the better.

Maybe a little history first, shall we?  First of all, what makes Django views so useful and easy to work with in the first place?  We can tell Django how to map HTTP requests to our view functions using URL configurations.  This is useful because we have URL definitions separate from the functions themselves - multiple URL patterns can be handled by a single view function.  The URL path segments and query parameters are passed to the function as positional arguments and keyword arguments respectfully.

Having a Python function handle each HTTP request sent to the application is intuitive, no two ways about it.  The job of the view function is to take the client's request, perform some business logic, and return a rendering context for the template.  For common scenarios, generic view functions can be passed to the URL configuration.  For example, listing specific objects or for a detailed view of a single object.  For this, generic view functions are perfect.  We don't need to write repetitive code that differs only slightly from view to view.

However, we can't use generic views for everything and this is where the Django developer needs to step in and write some custom stuff.  Stuff that'll refine a query or pass additional context to the template.

Prior to Django 1.3, where generic class views were introduced, generic functions were customized by passing an info dictionary to the URL configuration.  This isn't an ideal coupling since the view now depends on additional data being passed to the URL definition.  For instance, if I have a URL where I want to return a list of objects, I can use a generic view that does just that.  However, to customize the query, I need to pass additional information to the URL definition that tells the view how to alter the query.  So if I wanted to reuse this same view, with the same query modifications, I'd have to duplicate the additional generic view info.  Not that this is painfully tedious, just error-prone.

Class-based generic views aims to solve the trouble of binding view-specific data to the URL, and to that end, I think they've succeeded hugely.  The new class-based approach introduces the concept of inheritance.  This method of extending views is less susceptible to problems with configuring URLs which depend on additional information.  Plus, extending generic views is simply elegant from a design perspective.

For example, we can extend the ListView class provided by Django to render a list of objects from our model.  At the most basic level, all we really need to tell ListView about is the model to select from.  Say we have a Book class.  We can define ourselves a new BookList class view for returning lists for books.  We're overriding the default model value of ListView.

I find this superior to the decorated customization approach.  With decorators, we're wrapping the view function with additional logic.  For instance, depending on the request context, HTTP headers, GET values, the URL, and so forth, we'll execute the logic of the generic view differently.  Nothing is wrong with this except for the fact that it doesn't sit well with doing things generically for a database view.  If I'm writing a list or detail view for things stored in the database, I want as little customization as possible if I'm using generic views.  This is a key point - the views are generic because you're supposed to give them only as little information as possible.  Just the bare minimum required to produce the appropriate template context.

With the new class-based approach, it's easier to leave the defaults alone, only overriding what's necessary.  The model name, for instance.  Or if you need more control over the resulting query set, overriding the get_queryset() method.  It's best to leave the common stuff to the generic view framework.  This is it's selling point, after all.  Things like automatically finding the template based on the model name.  If you absolutely must change something about a view, there is an attribute or a method you can override.  If you find that you're extending the Django generic views and they're becoming large, chances are you shouldn't be using them.  That is, if a method you've overridden on a Django generic class view looks nothing like a basic customization, you should think hard about why you're using a generic view in the first place.  Maybe you're better off writing a regular, non-generic view.  There is nothing wrong with this approach - that's why standard, function-based views exist - for writing business logic.

So from a design perspective, what do class-based generic views bring to the table?  A cleaner, more elegant mechanism for displaying database data.  In addition to decoupling the customizations that go along with generic views from the URL configurations, class-based generic views embrace inheritance as a means to use default behaviour and override only when necessary.  Often the generic behaviour is enough, just tell the view about the basic necessities, and you're off to the races.  Django offers a plentiful selection of built-in class-based views to choose from, each of which can be extended cleanly.

Server Side Embed

In this day and age, our web experience wouldn't be nearly as interactive without the objects embedded inside most pages. What is the purpose of embedding objects inside a web page? Shouldn't HTML take care of this for us? It should, especially now that HTML5 is ever more mainstream with each passing day. Flash, on the other hand, is the standard for video, or anything fancy user interactions - transitions and such. Are embeddable objects for the web, good or bad? They're so pervasive that we can't simply ignore them, even if we don't like them. From a design perspective, embedded objects seem like a good way to reduce coupling by offloading some resources such as video. Its easy to use resources from other services, like YouTube. With social networks abound, embedding objects is an important concept for building web pages in a social context. How can our stuff on the server benefit from the embed tactic?
 

Let's take a closer look at why we embed things in our web pages – videos, menus, widgets and the like. Videos are obvious – websites want to show videos and we can't do this with HTML alone. HTML5 has the video element, but even then, I think we're still embedding conceptually because the video data isn't markup language. Likewise, an iframe might be considered an embedded object – even though its data is HTML, we still need to fetch it from an external resource, one outside of the page. Social networking widgets, too, are loaded from outside the page. I consider stuff that isn't directly contained in the page an embedded object. This works for me logically, because I can classify everything else as external, even though it may reside on my own server. Everything else, the paragraphs, headings, and so on, are a tightly-integrated unit that depends on external data.
 

What does embedding objects in HTML pages have to do with low-coupling? Well, if we can think of our web pages as a skeleton structure with slots for objects, we can reuse a lot of our HTML, consequently, CSS too. These object placeholders can put whatever we need in them – flash, HTML, our own widgets. That's really all we really need to worry about with this design, fill the hole with a web object. This is different from traditional programming languages where we're filling in the blanks using objects and polymorphism. Here, you've got to make sure that each object has the interface the slot is expecting. Not so much with HTML. You can embed whatever you want, the worst outcome being a shoddy looking site. So can we take these ideas, these embeddable objects and apply them to our web applications that live on the server?
 

As an idea, of course we can. In fact we already do this, we just use different terminology. On the server, we've got components. This is how the functional aspects of our application interchange. Replaceable authentication components, graphics libraries, you name it. There is something that can step in to do the job of a misbehaving or under-performing server object fairly trivially. I say fairly trivially because it isn't always so easy. The server-side is very different from the browser, worlds apart.
 

Lets say we're designing a simple brochure site. It doesn't rely on the server in any way other than to deliver web pages. So the big requirement is that it displays some information, maybe has a mouse-over effect here and there. A project like this only requires HTML, CSS, and Javascript. We need a web server, but don't have to do anything with it aside from starting it – we don't have to program it. Our little web site, in the browser, has three things, three languages, to worry about. This can be handled easily by one person on a small scale. Even when it comes to replacing the embedded objects, there is no learning curve because there is nothing to objects in HTML. Usually just a URL. Or maybe some cut-and-paste object code with a parameter. Easily manageable.
 

The server part of an application is a different story because there is just so much involved with serving HTTP requests. We have to make sure the user is authenticated, make find the appropriate request handler, process the request in a timely manor, and perform database queries. All of this while making sure we're fault-tolerant because unlike the browser, an HTTP request has the potential to break the entire application for every other user. This is why we have web application frameworks, to handle the common stuff with serving dynamic content. So how do we embed objects, objects that extend the capabilities of our application, into the framework?
 

Django immediately comes to mind for me, when thinking about embedding on the server. Partly because I really enjoy using it and partly because it has the best example I can think of. I'm sure there are plenty others too. In Django, we've got middleware and applications, both ready to embed in our projects. Middleware is how we inject logic into each HTTP request served. Sorry, embed logic. Middleware inspects the request before it is handled and after the response has been generated by the handler. This relates to HTML embedding in that we're able to take third-party objects and use them for our purposes, just as we would copy and paste a YouTube video.

Embedding things into larger applications, be it an HTML user interface, or something on the server, is a powerful concept. Embedded objects are simply part of the larger component architecture, where stuff can be swapped out. I think the embed concept for severs has a close parallel with the kind of embedding we do in the browser. Its a powerful idea, it helps with designing loosely-coupled systems for the web. It helps with delegating the responsibilities of a feature to the appropriate party. And that's it, really. Its just a mode of thought when putting together an application.

Features Don't Freeze Themselves

How do we know when its time to stop adding new features to our software? When do we take off our creative hats and put on our cautious stability testing hats? Once we meet the functional requirements of the project, that's when. The term used to describe this dividing line between the chaos of development and the order of a finished product is feature freeze. Yep, features are literally frozen in time. They don't change or evolve from this point forward, until the release is out. After releasing, we've got a clean slate, more or less. Its time to start adding new features, and expanding on those previously released. So my question is this: is the feature freeze phase really that straightforward to define? By this I mean, can way really say with confidence, based on the project requirements, that we'll know ahead of time when we're going to stop making new code? I would argue not, as would most developers because impudent predictions are discouraged in software development. This problem is more pronounced when we're building things that have ambiguous requirements, things like frameworks.

The rationale behind feature freezes is self-explanatory for the most part. There has to be a point during development were we devote our efforts to quality assurance. Of course, we do this even during development too – modern approaches to software development involving test harnesses for each feature. Even this is limited, speaking to the system as a whole, how it will be used in the wild by real users? The incremental develop and test approach cannot cover all cases. So we have to stop development, and try our best to destroy our code, automating as much of the process as possible. We pick and prod and poke, systematically dismantling what we've built until we can't find any weak points. This is the best way to build reliable software. This is also the ideal scenario - not exactly reflective of what we face in reality.

Think about software requirements for a client project. When have you ever built software based on a single set of unchanging requirements? That's what I thought, never. Requirements change and so does everything else. By everything else, I mean the driving force behind the requirements. The client wants a new widget because it'll make a new department at their organization function more efficiently. They want a new social widget because it is the brand new thing the simply cannot live without. They want the colours slightly different, due to nothing more than their mood that day. Who knows why? The important thing to remember is that initial requirements are nothing more than a general prototype of what the finished product might look like. Requests for change usually don't arrive until well after it would have been convenient for us to implement.

You'd think this whole implement the requirements thing would be a little easier when building a product that solves a more general problem. For example, shopping cart software, income tax reporting software, image editing software. The list goes on – things that are general enough that more than a small group of people might find it useful. Maybe a better example is the Linux kernel. You'd think that solving a general problem would make the requirements a little more concise. The opposite is when building stuff based on a contract for a company. Usually a web application of one flavour or another. These have more niche things that need building. This is why you were hired to build them in the first place – to execute a precise vision. Consumer software on the other hand, usually doesn't cater to the specific problems of any one organization. And this is the challenging part when it comes to devising requirements for such a thing.

There is no such thing as a universal software product that will solve every problem for every person on the planet. Not even for a specific problem domain is this true. There will always be a deficiency, a missing feature, or maybe the user interface is just not that friendly. This is why alternatives exist for every kind of consumer software system. Linux isn't for everyone, nor Windows, nor Mac. Users have their own reasons for choosing one over the other. Different software that strives to solve the same problem excels in different aspects. Some operating systems are more user-friendly than others, and some are more secure, and some are used for the sake of familiarity, why change? Again, no software caters to everyone. That doesn't always stop us from trying though.

Programmers are perfectionists. Not only does the code need to look beautiful, it must work as expected, with no bugs. More than that, we also care about the usefulness of our software, more than some might think. Nobody wants to write code that doesn't help somebody solve a problem. This is considered a failure, and this is never fun, we've all been there before. Successful projects have at least a handful, maybe a couple hundred people using it on a daily process. Ideally, users recommend it and praise its usefulness and how they simply could not live without it. I know I have a list of a dozen or so tools I'd be lost without. This is always great to hear. However, we also hear about those who struggle with certain aspects of the system or outright stopped using it for some reason or another. Hearing these things about code you've sweated over are hard to ignore. Maybe the last feature freeze wasn't such a good idea after all, I mean, there are people out there disappointed with our product!

And so it begins, we've now got to start the task of expanding our system's capabilities to accommodate. Built to please. This is the cycle, add new features, expand existing features. Accommodation. The problem is, the cycle never ends, and what you're left with is usually something bloated that looks nothing like the initial vision held so long ago. How do we break the cycle?

Loose-coupling is part of the solution to the feature bloating problem. Large systems that have grown due to the sheer number of features it provides aren't necessarily loosely-coupled. Can you download and install a smaller version of the software? For example, some applications let you configure the pieces you'll need before installing and using it. This isn't possible if your system's components are tightly-coupled. However, even this approach might not suit all users because they may need these negated components down the road. Can they add the missing pieces? Or do they need to repeat the entire download and install process? I think the only real solution is to develop independently installable components.

If your software can be broken down into individual components, there aren't so many dependencies that force features on users that don't need them. Building independently installable components, and designing loosely-coupled code, however, are two very different things. Loosely-coupled code is design discipline. Independently installable components is a deployment discipline. There is some degree of overlap though, because your design needs to support variable dependencies. For example, the application still needs to work if there is no sorting component found – it just wont sort stuff. As always, there needs to be some sort of core foundation, the essential code necessary to bootstrap the application. This is where the logic to determine what features are available live.

It might seem like overkill to divide your software into tiny installable components, one for every feature of the system. It might seem that way, even scary at times, what happens if the component installation fails for some reason? This isn't any different than installing the whole package as one big opaque box – buggy code is buggy code. Its better to think from the users perspective – maybe the majority of features we've implemented in the system are overkill to them. If they don't want them, they don't need them. So my advice is this – implement feature components as much as possible. Implement 400 of them if you need to, just don't ship them as one unit that cannot be decomposed.

Interface Connection Thinking

Software object systems, are a collection of objects, connected by relationships. Typically, objects are connected by structural relationships. I can think of a dozen real-world analogies when I'm writing code. When I think of these analogies, I'm thinking about the physical structure of the buildings, cars, engines, books, anything really, that inspires my software design. This is why object-oriented software uses the object as the fundamental unit, not the behaviour. We don't focus on the behaviour of our software when we're designing it. At least it isn't the first thing we consider when starting a new project. I don't think “what are the major behavioural components of this system”? The first thing that comes to mind when building from scratch is something along the lines of “what are the fundamental data structures I'll need to make this work?”. Maybe this approach is backward.

Emphasis on the data structures taking precedence over how the software works – the moving parts of the system – is hard to overcome because we're so used to making analogies to real-world objects and their properties. This is an essential feature of our brain – if you cannot distinguish between dangerous objects and irrelevant objects or beneficial objects, your chances of survival are slim to say the least. We differentiate between objects in the real-world based on their attributes – the things we can see. We recognize objects by nothing more than their physical attributes – their shape, their colour. My desk I work at everyday stands out relative to every other desk in the office because of its attributes. The monitor, the coffee mug, the unsorted pile of paper – all things that make the desk unique and recognizable.

This isn't to say that we don't make analogies with regard to how things function, the way they move or produce output. My desk, of course doesn't do anything very interesting, or anything at all for that matter. It just sits there – I have no real need to compare what it does to anything else. My car, on the other hand, has several interesting characteristics other than its less-tangible attributes. I can use these to make comparisons - to other cars, or other objects. The car moves – hopefully in more than one direction. It has a maximum speed and it turns. I could go on, but that would be overkill. Clearly, we're capable of making behavioural analogies too. The question is, why do we favour the structural when it comes to software design?

As we know, software is a host of objects connected to one another. I'm not sure there is as much a distinction between structural and behavioural aspects as we might think. The structure of software objects is like glue - it prevents an objects from becoming unraveled and going rogue in the system. When we're thinking abstractly, we're thinking that structure equals order. If we could only build the perfect foundation, we could add the required functionality on top. Everything will work better than it would had we not built a solid base. I'm not sure this is the correct view to take.

My reasoning is simple – interfaces describe what an object does, and suggests some structural attributes the object might have, based on its name. Interfaces are concrete in their behavioural descriptions and tentative in their structural descriptions. This is because interfaces give all the necessary details for implementing the interface methods – the behaviour. If the interface has a good, descriptive name, you can glean some potential attributes that support the implementation. The objects attributes aren't part of the interface contract whereas the operations are. This is important, I think, for at least some degree of implementation freedom. Whats really important with interface connections is that clients get what they expect from their suppliers.

Suppose we've got an IReader interface that requires a read() method be implemented by any objects that provide the interface. The read() method is supposed to return data. How it does this, exactly, is up to the implementation. Maybe the data is read from a file, maybe from an attribute stored in memory, maybe from a network connection. These are all implementation concerns, not that of the interface. The interface serves as a connector, putting the emphasis on what that implementing object does. The client object knows what it needs, in the case of the IReader interface, it needs an object that can read data.

So how does thinking in interface connections result in better software? I think it takes the focus away from the structural properties of objects and places it on the behavioural descriptions. Objects still have attributes, even when we're emphasizing the interface connections. However, these are more closely tied to the implementation than the design. Connecting objects by what they do is how you connect objects by design instead of by implementation.

Statistical Objects

Software is really good at keeping statistical records. We can write code that stores raw data we want to keep track of. We can then display this data in a user friendly way, maybe as a chart of some kind. In addition, the stored raw data can be mined for trends, and other anomalies. Imagine trying to function in a modern scientific discipline without this capability - software that aids in statistical analysis. It would be nearly impossible. There is simply too much data in the world for us to process without tools that extract meaning for us. This same complexity phenomenon is prevalent in modern software systems. External tools will monitor the externally-visible performance characteristics of our running system for us. Inside the code, however, is a completely different ball-game. We really don't know what is happening at a level we can grasp. Logs help - they're good at reporting events that take place, things like simple tasks, errors, and values for debugging. But the traditional style of logging can only take us so far in terms of the true nature of how our software behaves in it's environment. Object systems should retain this information instead of broadcasting it to a log file and forgetting about it.

Statistics software is different from software statistics. Software statistics is data about software itself, things like how long it takes to process an event or respond to a request. Running software can track and store data like this in logs. Why do we need this data? The short answer, we need it to gauge characteristics, the otherwise intangible attributes of our software exhibits during it's lifespan. The system logs, in their chronological format can answer questions like “when was the last failure?” and “what was the value of the url parameter in the last parse() call?”. Further questions, questions about an aggregate measure of quality or otherwise, require a tool that will parse these logs. Something that will answer questions such as “what is the average response time for the profile view?” or “which class produces the most failures”?

These latter questions need a little more work to produce answers. But first, are runtime characteristics the same thing as application logs?

Well, yes and no. Logs are statements that we're making about an occurrence in time, an event. This is why logs are typically timestamped and what makes them such an effective debugging tool. Something went wrong at 10:34:11? The error logs just before then say something about permission problems. I now make a few adjustments, perhaps even modify the logging a little, and problem solved. Characteristics of a running system, on the other hand, equate to qualities that cannot be measured by a single event. The characteristics of running software changes over time. We can say things like “the system was performing well yesterday” or “two hours ago, the disk activity spiked for five minutes”.

Software logs are like a newspaper. Not an individual paper, but the entire publication process. Events take place in the world and each event gets logged. We, as external observers, read about them and draw our own conclusions. Software characteristics are more like the readout on car dashboards that tell you what the fuel economy is like over the past several kilometres. This can answer questions such as “how much money will I need this week for gas?”.

Its not as though we cannot log these characteristics to files for the consumption of external tools to analyze and provide us with meaningful insight. We can, but that doesn't really suit the intent of logging events. Events are one-time occurrences. Characteristics, or traits, is something that is established over time. Our system needs to run and interact with its environment before anything interesting and be accumulated and measured. The question is, how is this done? If we want to measure characteristics of our software, how it behaves in its environment over time, we'll need an external observer to do it for us, to take measurement. External tools can give us performance characteristics or point to trends that cause are software to fail. These things are limiting in some respects because they say nothing about how the objects inside the system interact with one another and the resulting idiosyncrasies.

In a system composed of objects, wouldn't it be nice to know how they interact with one another? That is, store a statistical record of the system's behaviour at both an individual object level and at a class level? This type of information about our software is meta-statistical – stats that only exist during the lifetime of our software. Once it's no longer running, the behavioural data stored about our objects is meaningless because this could change entirely once the software starts up again. If we can't use a report generated by a running system to improve something, say, our code, or development process, or whatever, what value does it have?

For example, suppose we want to know how often an instance of the Road class is created. We might also be interested in how the instance is created – which object was responsible for it's instantiation? If I want to find out, I generally have to write some code that will log what I'm looking for and remove it when I'm done. This is typical of the debugging process – make a change that will produce some information that we'd otherwise not be interested in. This is why we remove the debugging code when we're finished with it – its in the way. Running systems tested as being stable don't need the added overhead of keeping track of stuff like which object has been around the longest or which class has the most instances. These statistics don't seem valuable at first, but we add code to produce it when we need it. When something goes wrong, it certainly comes in handy. Maybe we don't need to get rid of it entirely.

Maybe we want to use statistical objects in our deployed, production systems after all. Maybe this will prevent us from having to piece together logic that helps us diagnose what went wrong. Logs are also handy in this sense, for figuring out what happened leading up to the failure. Recall that logs are system occurrences, a chronological ordering of stuff that happens. We can log as much or as little as we please about things that take place while our software is running. But, too much information in the log files is equally useless as not having enough information to begin with.

The trouble with storing statistical data about our running system – data about software objects in their environment – is the overhead. If overhead weren't an issue, we probably wouldn't bother removing our debug code in the first place. In fact, we might build it into the system. Code that stores interesting information. Insightful information. Developers dream of having the capability to query a running system for any characteristic they can think of. Obviously this isn't feasible, let alone possible. To store everything we want to know about our running system would be an exercise in futility. It simply cannot be done, even if we had unlimited memory. The complexities involved are too great.

So what can we do with statistical software objects that speak to meta properties of the system? Properties that will guide us in maintenance or further modifications to the software in the future. Its simply, really. We only store the interesting meta data on our objects. Just as we only log events we feel are relevant, we only keep a record of system properties that are statistically relevant to a better performing system. Perhaps a more stable system or any other unforeseen improvement made as a result of the meta data being made available to us. This way we can be selective in our resource usage. Storing things that don't help us, doesn't necessarily make sense, although, the usages sometimes wont reveal themselves until its too late and you don't have the data you need and have a heroic debugging effort to deal with.

For example, say you're building a painter application, one that allows you to draw simple shapes and move them around. You can keep track of things like how many Circle and Rectangle objects are created. This falls into the domain statistics, however, because this is what the application does. This information is useful to the user, potentially, but not so much the developer, or the application itself. It is more a characteristic of the user than the software. But what if we knew how each shape was instantiated? Did the user select the shape from a menu, or did they use the toolbar button? Perhaps these two user interface components have different factories that create the objects. Which one is more popular? How can we, and by we I mean the software, exploit this information to function better? Using this information is really implementation dependent, if used at all. For example, the painter application could implement a cache for each factory that creates shapes. The cache stores a shape prototype that gets copied onto the drawing canvas when created. Armed with meta-statistical information about our system, we can treat one factory preferentially over another, perhaps allocating it a larger cache size.

The preceding example is still reflective of the domain itself. Sure, the implementation of our software could certainly benefit from having it, but what about problematic scenarios that are independent of the domain? For example, disk latency may be high in one class of objects, while not as high in another class. Again, this is does depend on the user and what they're doing, but also on factors external to the software, such as hardware or other software processes competing for resources. Whatever the cause, we give our system a fighting chance to adapt, given sufficient data. Sometimes, however, there really isn't anything that can be done to improve the software during runtime. Sometimes, external factors are simply too limiting, or maybe there is a problem with the design. In either case, the developers can query the system and say “wow, should definitely be running with more memory available” or “the BoundedBox class works great, except when running in threads”.

Of course, we're assuming we have the physical resources to store all this meta data about our software, data that highlights the running characteristics of it. We might not have the luxury of free memory or maybe writing to the disk frequently is out of the question. In these situations, it might make sense to have the ability to turn off statistical objects. You could run your software with them turned on in an environment that can handle them. When it comes to deploying to its live, production environment, shut off the extraneous stuff that causes unacceptable overhead. More often than not, however, there are more than enough physical resources in today's hardware deployments to handle the extra data and processing power required by statistical objects. If you've got the space, utilize it for better software. The reality is, as our software grows more complex, we'll have no choice but to generate and use this type of information to cope with factors we cannot understand.

Implicit Encapsulation

Understanding large data structures in software is hard. Using abstract entities helps us understand what we're building by hiding information about the structure that we aren't interested in. This is what encapsulation is all about. Software objects are simplifications of some idea, so, anything. The cohesiveness of abstractions in a software system, how understandable they are, what information they expose, the impact they have on other abstractions in the system - we don't consider things things once we've hacked something together that works. Of utmost importance is that it works, not how comprehensible the classes are. How do we measure abstraction quality? Encapsulation is a design principle, one that allows us to remove irrelevant aspects from the problem at hand, which makes it hard to objectify. We can look at some trade-offs of various encapsulation approaches. For instance, some programming languages allow us privatize object attributes - hide them from other objects. As developers, have an amplified ability to hide irrelevant details that aren't beneficial to us. How valuable is explicitly changing the visibility of an object attribute so as to hide it?

What makes an attribute irrelevant to the outside world? Not only irrelevant, but dangerous if exposed due to misuse by other developers. Subscribing to this attitude when it comes to information hiding is somewhat useless - your code is always going to be abused no matter how you try to protect it. We can safeguard against misuse to some degree by hiding attributes as you can't change what you can't see. Once I've marked my attribute as private, it is as though it never existed. If we wanted to get fancy with how our abstraction is discerned, we could implement and endless combination of public, private, and protected visibility settings. This type of configuration, we don't want to be stuck with. I can see the code, so I can see the attribute marked as private in one way or another - it isn't as though I'm completely ignorant of it's existence. I just can't use it there is usually no apparent reason as to why. What makes an attribute irrelevant to the outside world?
Private attributes inaccessible to other objects exist as part of the internal structure of the object. They give the object it's character, even if we can't see it from the outside. For instance, public attributes can be derived from private ones. Say I have a Product class with two private attributes - sellerCost and profitMargin. A public attribute, buyerCost, could then derive it's value from the two private attributes. The outside world only cares about the buyerCost, not how it was computed. The two private attributes are encapsulated - nobody can see them. Attempts to access sellerCost or profitMargin are met with failure. Where do we set the value of these private attributes?

The constructor is a good place to set attribute values. Passing object attribute values to the constructor isn't the same as modifying the attributes after instantiation. Think of it this way - creating an object is more than just saying "create a new product". You create objects by saying things like "create a new product with a seller price of $15 and a profit margin of 25%". This creates a product instance, different from those with a $500 seller price and 16% profit margin. If we're able to change these values after the object's existence has been established, we're in effect creating a new object. This is an abstract idea, you can obviously change attribute values all you want once an object is created as long as it isn't static. If we know we're supposed to set attributes in the constructor, we know that we're using an object that relies on initial values that don't change throughout the duration of it's life. Knowing this leads to better understanding of the system under construction because we can see the expected values used by derived attributes.
Objects change state in a running system, meaning, an object's attributes change value. Otherwise, we'd have a completely static system that just creates objects and doesn't do anything interesting with them. In the spirit of encapsulation, one approach to changing the state of an object is by using setters. Setters are nothing more than simple methods that set the value of a private attribute. There are also getters - methods that return the value of a private attribute. All we're doing by adopting this methodology is forcing developers to take a scenic route to storing and retrieving object values. Why not just read the attribute directly? Why not assign a value to an attribute directly? If you have a method that does this for you, you're not improving the encapsulated design of your code. Remember, the idea is to hide data that is irrelevant to the outside world, not to impose an unnecessary data conduit. When the state of an object changes or when data is read from an object, you may want to trigger some event. This is easy to do with setters and getters - set the data, trigger the event. However, this isn't within the scope of encapsulation - if setters and getters provide you with convenience, then by all means, go for it. Just don't assume that you've hidden all irrelevant attributes properly.

We've established that the visibility of an attribute doesn't necessarily dictate the level of encapsulation an object exhibits. It is up to the developer who designs that class to implement an outer wrapper that the outside world can see. Is this possible without explicitly saying so in the code? Can I implement a class that when instantiated, provides an adequate interface to fulfill the object's responsibilities? An interface says more about information hiding than the attribute visibility does. We can explicitly hide stuff all we want, but the interface, the contract says how it'll be used. No developer in their right mind is going to say "how can I abuse this object as much as possible by playing with it's innards even though I shouldn't". We really don't care to know how things work under the hood. We like to think about it as it just works. If the desire to tinker was just too strong, we'd write our own code to do the same thing. Reinventing the wheel is so last decade. When change happens, when inane requirements arrive, the provided level of encapsulation no longer provides the basic necessities of the system. Developers need to start playing out of bounds - we need to rethink what is relevant and what isn't.

At this point, we need to step back and say "hey, looks like a lot of hacking is going on with the Logger class, I think we'd better rethink visibility there". But this never happens. Developers aren't going to say to one another how much a decision to make something private has made their lives miserable. No, it just goes unspoken - privacy is irrevocable. The same predicament holds true for publicly visible attributes - are they forever available to anyone interested or at one point or another does it make sense to hide them? I've always worked under the assumption that visibility is a "set in stone" type of ideology. The reason is simple - if you have something public and you suddenly restrict access, you're asking for trouble. Obviously this is a lot of work to go and fix because we'll no doubt find many subtle issues even after we find an alternative way of doing things since we no longer have access to the weight of the product. You take an integer field that your using somewhere in your application, suppose, innocently enough, your just reading it because you need to perform some calculation. Imagine that. Now this simple act of hiding the integer value, because it is causing problems somewhere else in your code, is now creating a new problem. Now for the other side of the coin - pulling the curtain and displaying something that we didn't even know existed. Great, now I don't have any problems doing what I need to do. I'm free to read any attribute values, I can use them to compute whatever I want and I don't have to worry about picking and choosing or about implementing workarounds. But what about the other problem, the whole thing about developers having access and abusing it? Wouldn't this just break things entirely since I can't trust anyone to do things right and only the most minimalist interface feasible is ever given out to developers? Not exactly. This is actually much safer to do because having something visible, waiting to be read, updated, or deleted entirely for that matter, will not break anything. This is contrasted with taking access away - big problems here. And can we actually plan for this kind of change? We're not exactly going to have error handling for invisible object properties. This chore is for humans to manage.

Another way to think about visibility with regard to object design is file system permissions. These are a lot more fine-grained than those of attribute visibility. With file system permissions, we have the ability to say that someone can read a file, but can't update it. Someone can execute a file while others can't. The number of combinations that can be used on just a handful of files is staggeringly complex. No wonder we don't have something like this in code. Its just trouble waiting to reek havoc. How would that work if we were to think about doing something more fine-grained, like file system permissions? Who would permissions be assigned to? Other software objects? Developers? Would permissions be granted on classes, or individual attributes? Suppose we could manage complexity at this level - a little intimidating isn't it? The problem here is that we're taking some of the creativity out of software design. And this is part of the problem with even simple restrictions like private vs public - we need at least some level of freedom to break things by playing with the internals of the map class. Mistakes will be made, no doubt, but this is part of the process. Object design isn't some black magic that will eliminate the burden of trial and error.
Complexity aside, let's revisit the idea of interfaces. They're more powerful for designing an encapsulated structure than visibility is, or so I'm claiming here anyway. If the contract designates what an object is supposed to do, and not how, we should be content with this as our restriction on how we go about deciding who can access what. Can interfaces guide our decisions regarding access policies? I would certainly think so. Remember, interfaces appropriate the visible attributes, not the hidden ones. How can I be sure that hiding the size attribute of my file class is a good idea until I see how the instances interact with others in the system?

If we don't worry about explicitly hiding information, we save time and effort. We're not concerning ourselves with what makes sense to expose, and what doesn't. In Python, we don't specify the visibility of an attribute or operation. It feels like a weight has been lifted when we adopt this attitude. It feels like I've put up a sign on my class that says "use at your own risk". Even in languages such as Python, there are feeble attempts made to make data private. One such method is prefixing the attribute name with an underscore. This doesn't actually reflect the verity of encapsulation. All we're doing here is sending a visual cue to human readers from the source code. This isn't necessarily wrong, it looks bad, but isn't wrong. I think it sends the wrong message however, because, using an arbitrary marker to signify an out of bounds entity, only signifies privacy, not why. It is probably better to use a descriptive name, one that makes readers think - I bet that attribute is private.

So what is the benefit to explicit data encapsulation in object-oriented software? Information hiding is the key idea that allows us as developers to take a concept and transform it into something more abstract. It is the inner details we often don't care about when we encounter something useful. In, the real world, the products we use on a daily basis have replaceable components. They also have internal parts that we're blissfully ignorant of. Had these inner pieces of an object's core been exposed, we'd risk damaging it. This is why we want to hide these details. This is the secret sauce that makes the thing work. It is the nature of the object. So applied to software, this idea works well - we don't want to expose the innards of our software components to other developers. I'm sure that enforcing these privacy characteristics isn't necessary. It is better to use a contract, this means defining an interface for you're object, either explicitly, or implicitly. Remember, interfaces do not describe the inner-workings of software objects. Interfaces are contracts and are only capable of describing the externally-visible behavior and or data of software objects. Using interfaces as the blueprint for visible properties, instead of explicitly hiding information is more descriptive for developers and thus more valuable.

Exceptions From Within

How do you properly raise and handle exceptions? This seems like a straightforward design problem. Accompanying exception handling in your code are questions about depth. For example, function a() raises an exception. Function b() calls a(). Should b() handle the exception? Or, should the exception continue outward in the call stack? The distance in the call stack, between the exception being raised, and the exception being caught, or handled, is up to the developer. This includes raising exceptions from within the exception handler itself.

Are these concerns a matter of coding-style? Or, can we establish a pattern that helps us decide where to raise and handle exceptions? Defining a generic exception handling pattern is hard to do because anything has the potential to raise an exception - intended or otherwise. From a functional perspective, exception handling is all about preventing unexpected events from disrupting your program. From a coding-style perspective, exception handling is all about knowing exactly where any given exception originated. Both are hard to do.

Picture the low-level functions in your code. The atomic functions that don't call anything else. These functions will, at one point or another, raise exceptions. The exceptions raised here will propagate upward in the call stack. Think of an exception as signal that notifies the rest of your application something went wrong. The signal is received at each point in the call stack. The only way an exception will stop propagating upward is if a handler for that exception type lives at that point.

Now, picture a higher-level function. A function that calls other, low-level functions. If this higher-level function knows about the exceptions that might be raised by the functions it calls, it can define a handler for them. This is the how exceptions are handled from a context outside that of which they are raised. All the handler needs to know about are the types of exceptions that might be raised. This is a key exception-handling concept - they are classes and therefore represent a type of thing that can go wrong. Any number of problems may cause an IOError to be raised. The handler only cares that it is an IOError. Anything else will continue upward in the call stack.

An exception handler has two components. The first, a try block, attempts to execute something. The second, an except block, conditionally runs if the first block fails. Is it plausible for us to raise exceptions from within the try block? Imagine an exception handler that handles a TypeError. If the try block itself raises a TypeError, the except block will handle it and the exception signal will stop propagating.

Raising exceptions from within the try block can reduce code verbosity. If I call a function that may raise an exception, you call that function inside a try block. Inside that same try block, you may call another function that doesn't raise an exception. For instance, the function might not return what you expected. In this case, you want your except block to run. It wont, however, because no exception was raised by the function call. Rather than alter the low level function, you can check the result and explicitly raise the exception that enables the except block.

Although this use method of exception handling is useful, it doesn't always feel right. Exceptions are indeed better suited for propagating outward in the call stack instead of being raised and handled in the same statement. Another drawback is that you are potentially blocking exception handlers even higher up in the call stack that might take a different approach to handling the same type of exception.

Using Python Generators

Python 2.2 saw the introduction of the yield statement. This statement returns a generator from a function or method. Returning a generator is like returning a list. You can iterate through the result in a loop. The difference between returning a generator and a list is the flow of control. When a method yields data, it also yields control flow temporarily. The method will resume control once the generator's next() method is called. When a method returns data, control flow is permanently returned. Generators aren't a replacement for returning data. They are a tool for Python developers to return a data stream.

Let's create a simple example system to get a better feel for what generators are all about. We'll design a book catalog that searches book files. The format of each book file is a serialized Python dictionary. The book fields, or dictionary keys, are the book title, synopsis, and cover image. Using generators where a simple list is sufficient can be avoided. During it's lifetime, a generator method goes through several states. These states help us envision generator properties, such as control flow and the overall responsiveness of the program.

Before we embark on our book catalog design, we should make an analogy. A Unix pipe is a data channel between two processes. The first process writes data to the pipe while the second process reads data from the pipe. Python generators are similar. Instead of processes we have a method that writes data to the generator and an iterator that reads data from the generator. The iterator that reads data from the pipe is a for loop. Each loop iteration is executed when data is made available by the generator. Generators, used as a loop operand, behave the same as lists or tuples. The only difference is the flow of control.

Our book catalog application should have an optional sorting component to sort search results. For this, we will create a Sorter class with a sort() method. This method accepts a list of books and returns a sorted list. Python lists have a sort() method that will sort the elements in the list, so we'll use it instead of reinventing the wheel. Once the list has been sorted, should Sorter.sort() return a generator? Before you decide, ask yourself if the return data is available. Does the method use a pipe and filter approach? Are we iterating over some data set, manipulating each element? Does the method produce a stream of data, or a monotonous piece of data? Our Sorter.sort() method isn't any of these things and will simply return the sorted dictionary.

The next book catalog component is the most important piece of the puzzle. Searching for books. A Filter class with a filter() method will handle this. This method accepts book iterator and filter string parameters and returns a book iterator as the filter result. Returning a generator from this method is a good idea because the return data isn't immediately available. This is because we're iterating over a set of books. We filter each book by checking if the supplied string exists in the title or description. If so, the book is yielded. This method produces streaming behavior because other objects that invoke the Filter.filter() method can begin reading from the returned generator before all data is available.

How does all this interleaving data and control flow work? Let's take a look at the states a generator method goes through during it's lifetime. The method starts in a running state. This is where the method computes data to send to the generator. Once data is ready to yield, the method goes into a yielding state. This state isn't active for very long. It is only active while the data is being written to the generator. Finally, the method goes into a frozen state. The method is frozen so that it may resume its flow of control once data has been read from the generator. The method will then enter the running state again.

The final component of our book catalog is a FileReader class. This class has a read() method that loads all books files and creates the corresponding book dictionaries. Each dictionary is then sent to the generator returned by this method. Now that we have all our book catalog components, the search work flow is easy. The FileReader.read() method yields a book dictionary. The Filter.filter() method searches the dictionary for "MyBook". A match is found and the book is sent to the generator. The user interface, which invoked Filter.filter() displays "MyBook" in the search results. FileReader.read() resumes with the next file in the directory. This chain of generators produces a stream of data and a responsive search. Think about a catalog with 5000 books. If one of the first 100 books matches the criteria, the user sees this book before the search has completed.

Designing a simple book catalog program has shown us that generators add overhead to methods when they aren't used to create a data stream. If your method operates on individual elements of an input data set and produces another set, use a generator. This is a pipe and filter approach where the generator is the pipe and the method logic is the filter. Our example illustrates the effect data streams can have on the responsiveness of some behaviors, like searching for books. Adding multiple threads of control to an application can also increase the responsiveness but using generators is a more intuitive, data-centric approach.