How do you automate the delivery of solid software? Software without bugs, software that fulfills what the customer has envisioned? It's difficult to determine how software is going to achieve it's role when dropped into problem domain battle field. Adroit human beings — the software development team — collaborate and work closely with the customer to work all this out.
Typically, this process never ends — even long after the software has been deployed and has become a possession of the customer. We'd like for our software to work well enough that it can handle subtle mistakes in configuration — in other words, the unexpected. Most experienced computer users have had experiences where they know ahead of time the software is going to fail as a result of their actions. The nice surprise these folks get is when the software unexpectedly adapts to the situation and might even offer the user some advice.
These advanced features — little nuggets of exceptionally usable software don't come naturally — not without a lot of testing. Back to automating the development process — the repetitive pieces that are better performed by unit test bots. We, as software developers, are constantly pushing the limits of what it means to be stable — how can we find our weak points and design a test that will cover them when we want to step through the testing procedure?
The question I have is this — can you devise creative automated unit tests without manually testing the software yourself?
Static software design
When I talk about static software design, what I'm actually referring to is not the design itself, but the methodology used to generate a design. In the end, what we're looking for is something that satisfies the requirements of our customers. The customers rarely glimpse into the process that produces their code. So if a development team follows the waterfall approach — if we're to assume that the requirements are Newtonian and unchanging, that'd be fine by them.
As we all know, the waterfall approach has a pretense of completeness about it. We're assuming, under this mode of operation, that nothing about the project will change. I'm sure there have been a few instances of this in the real world — hello world programs, for example.
Static software design means that we're not incorporating the emergent system properties into the final product. During development, we're bound to make discoveries about our approach, the problem domain, or even the technology itself. We don't have robots that'll write code for us yet — this means that humans still play a pivotal role in the design of software. And where there are humans, there are false assumptions.
Is this necessarily a bad thing? No, because if there is something unique within our species, it's our ability to problem solve and adapt quickly. There are simply too many variables in software design to fully automate the process — and even if we could take humans out of the driver seat when it comes to optimal programming language use, we've still got humans supplying the input.
The iterative approach to software development is really nothing more than a means to organize the natural learning that takes place during evolution of the system. Each iteration has new input. The first iteration has what the customer envisions — each subsequent iteration has our previous attempts at prototyping the system along with new problems we've introduced for ourselves. This is as good as it gets — we not only improve the software system we're building, but as developers, we're building our own repertoire of software development knowledge. You don't get this with a static software design methodology.
Dynamic software testing
Write the tests first, build software that pass those tests. An elegant solution — perhaps an obvious solution to building quality software in as little time as possible. It's like when you're figuring out a problem mentally — it helps if you have a set of smaller problems — questions you can explicitly ask yourself before answering. The same goes with software — it's much easier to write code that'll pass small tests than it is to write software that'll answer some grand philosophical question.
This is indeed a great starting point — an atomic means to get your feet wet with each iteration of the product life cycle. As a bonus, you've now got a suite of tests you can continue to apply throughout the development of the software.
But here's the thing — the tests you write for your software are like a little software project on their own. The tests are the nurturing parent that makes sure the child is able to make it in the real world without walking out onto the highway and getting run-over. Thus, if our software is continuously evolving, than so must our unit tests. As we write our tests, we're going to discover what works and what doesn't. But as we learn and improve the software, so too must the tests be improved.
As the system grows, we'll need to introduce new tests that aren't necessarily related to the overall requirements of the system. Perhaps there is a something a little less tangible — something related to a technological dependency that caused problems during development. These are useful tests to pass and are impossible to collect up-front.
So if tests are also evolutionary, they should become an integral part of the software development process. Incorporating automated unit tests into the project isn't anything new — what I'm suggesting is that they're treated as the parent of the software product and follow the same evolutionary course. Just as static software development isn't a tool humans can readily use, neither are static up-front unit tests. New unforeseen scenarios need tests. These don't reveal themselves till several iterations into the project. And, old tests should eventually be culled — much later on. Mature software products should have sophisticated unit tests that supply unique input as a result of evolution.
Showing posts with label testing. Show all posts
Showing posts with label testing. Show all posts
Wednesday, October 12, 2011
Thursday, October 8, 2009
Self Assuring Code
The notion of self assuring code sounds a little like a spin on writing code that incorporates well-written exception handling. This is probably true for the most part but if exception handling can be taken up a notch, developers can build programs that are resilient beyond expectations.
One way to write self assuring code is to do just that; write exceptions handling code for just about every possible exception that might be raised. Exception handling works great for handling any type of exceptional occurrence such as type errors. Custom exceptions are another common construct in many exception handling implementations. The custom exception classes typically inherit from the primitive error types of the implementation language. What is useful about this approach is that the developer is extending the notion of what constitutes an exceptional occurrence.
Something that even the best exception handling cannot do is give the same type of feedback to the developer that is brought about by quality assurance that is brought about by human test users. This is especially effective if the users have never used the software before. This is because there is no bias involved. These users haven't had questionable experiences in certain parts of the application and are not lenient if it just barely works.
Is it even possible then, to have self assuring code? Can developers take exception handling to the next level and build it into the code? Think of trying to find a really difficult bug. What does the developer do? They put debug log messages in places that don't necessarily make sense. But as most developers know, it is these messages, these just by change debug notes in strange places that often end up solving the problem.
The question becomes, how elaborate must these exceptions become? Do they try and predict future failures best on how well the current execution is going? Maybe. This whole idea is very context dependent. But experimentation with creative exceptions might be worth exploring.
One way to write self assuring code is to do just that; write exceptions handling code for just about every possible exception that might be raised. Exception handling works great for handling any type of exceptional occurrence such as type errors. Custom exceptions are another common construct in many exception handling implementations. The custom exception classes typically inherit from the primitive error types of the implementation language. What is useful about this approach is that the developer is extending the notion of what constitutes an exceptional occurrence.
Something that even the best exception handling cannot do is give the same type of feedback to the developer that is brought about by quality assurance that is brought about by human test users. This is especially effective if the users have never used the software before. This is because there is no bias involved. These users haven't had questionable experiences in certain parts of the application and are not lenient if it just barely works.
Is it even possible then, to have self assuring code? Can developers take exception handling to the next level and build it into the code? Think of trying to find a really difficult bug. What does the developer do? They put debug log messages in places that don't necessarily make sense. But as most developers know, it is these messages, these just by change debug notes in strange places that often end up solving the problem.
The question becomes, how elaborate must these exceptions become? Do they try and predict future failures best on how well the current execution is going? Maybe. This whole idea is very context dependent. But experimentation with creative exceptions might be worth exploring.
Tuesday, January 6, 2009
Python benchmarks
Just for fun, I decided to run some Python benchmarks that test the lookup time differences between list, tuple, and dictionary types. The list performance seems to come out on top every time. This is confusing to me because I hear that tuples are supposed to be faster because they are immutable. Here is the test I used:
I'm running this on a Intel(R) Core(TM) 2 CPU T7200 @ 2.00GHz machine. I wonder if my test is flawed.
from timeit import Timer
test_data1=[1,2,3,4,5]
test_data2=(1,2,3,4,5)
test_data3={0:1,1:2,2:3,3:4,4:5}
def test1():
v=test_data1[2]
def test2():
v=test_data2[2]
def test3():
v=test_data3[2]
if __name__=='__main__':
print 'Test 1:', Timer("test1()", "from __main__ import test1").timeit()
print 'Test 2:', Timer("test2()", "from __main__ import test2").timeit()
print 'Test 3:', Timer("test3()", "from __main__ import test3").timeit()
Monday, December 8, 2008
Distributing unit tests.
In this context, I'm referring to distributing unit tests along with a software package, as opposed to executing unit tests in a distributed fashion (which is an interesting topic in its' own right). Unit testing has proven to be an essential development artifact to ensure all use cases are executed correctly. I wish I could say all open source projects distribute unit tests as a component of the package. This is simply not the case. Several projects do, however, include a testing facility which is usually composed of unit tests.
Why, if the authors of these software packages spend time writing all these unit tests, should they be distributed along with the software? After all, they write these unit tests for their own testing. Sometimes an entire testing team is responsible for this task.
I would say that no matter how big a testing team any given project has, they are never going to cover all possible corner cases. That goes without saying (but I'm saying it anyway). Custom operating environments, even some other piece of installed software, could cause unexpected behavior that is in no way handled correctly be the software in question. At least having unit testing available in these situations can offer some clue.
Why, if the authors of these software packages spend time writing all these unit tests, should they be distributed along with the software? After all, they write these unit tests for their own testing. Sometimes an entire testing team is responsible for this task.
I would say that no matter how big a testing team any given project has, they are never going to cover all possible corner cases. That goes without saying (but I'm saying it anyway). Custom operating environments, even some other piece of installed software, could cause unexpected behavior that is in no way handled correctly be the software in question. At least having unit testing available in these situations can offer some clue.
Labels:
artifact
,
opensource
,
package
,
testing
,
unittest
Wednesday, November 26, 2008
Enomaly ECP update
Just this week, I've started writing some new unit tests for Enomaly ECP. I decided to take a different approach than the previous unit tests. These new tests are designed to test the RESTful API of the software. The new testing module, which I'm using to test version 2.2, establishes a connection with a running ECP instance and makes several HTTP requests to the API. The results for each test pass or fail based on the HTTP status, the internal error code returned, and ensuring that modifications actually took place.
Another nice little tool I've incorporated into the new unit testing is sptest. It gives us some nice colourful output.
This shift in testing focus will hopefully yield a more robust API for the Enomaly ECP platform that can be verified with test results. I hope to actually ship this new testing module with 2.2 so users can run the unit tests for themselves.
Another nice little tool I've incorporated into the new unit testing is sptest. It gives us some nice colourful output.
This shift in testing focus will hopefully yield a more robust API for the Enomaly ECP platform that can be verified with test results. I hope to actually ship this new testing module with 2.2 so users can run the unit tests for themselves.
Thursday, October 30, 2008
Don't test your own code
Well, test your code but don't say it is production ready after testing it. If a formal unit testing procedure is not in place for the project, at the very least, another human needs to test your code. This other person doing the testing doesn't even need to be a developer (if the code is fronted by a GUI).
You can spend hours on end testing your own code and find nothing wrong with it. Pass it off to the developer next to you and he is able to break it in under ten minutes. I obviously don't quite understand the psychological underpinnings of why these biases occur when testing your own code, I just know that they do. Perhaps it is a matter of motivation. Put yourself in role of tester. Someone gives you some code and says "here, break this". Whereas testing your own code may result in more work on your part. Maybe it is easier to fix things that need to be fixed as opposed to if I don't see it, its not broken.
I've actually thought about this while testing my own code and thought about previous testing sessions. I recalled scenarios where I figured good enough==not broken. I'm sure anyone can think of these mishaps if they try hard enough. The good news is that it is unavoidable. Try all you want, you alone will not eliminate your own bugs. Let someone else test your code.
You can spend hours on end testing your own code and find nothing wrong with it. Pass it off to the developer next to you and he is able to break it in under ten minutes. I obviously don't quite understand the psychological underpinnings of why these biases occur when testing your own code, I just know that they do. Perhaps it is a matter of motivation. Put yourself in role of tester. Someone gives you some code and says "here, break this". Whereas testing your own code may result in more work on your part. Maybe it is easier to fix things that need to be fixed as opposed to if I don't see it, its not broken.
I've actually thought about this while testing my own code and thought about previous testing sessions. I recalled scenarios where I figured good enough==not broken. I'm sure anyone can think of these mishaps if they try hard enough. The good news is that it is unavoidable. Try all you want, you alone will not eliminate your own bugs. Let someone else test your code.
Tuesday, August 5, 2008
User interface testing
Although there are several tools available today that enable the automation us user interface unit testing, they are not ready to use as the only testing procedure of any given application.
There are several issues that these tools fail to catch. Mainly, the work flow errors. Your application could pass the GUI testing framework with flying colors and yet, once your application is human-accessible, it fails colossally. This is because the slightest misinterpretation by the testing framework as to what the work flow should look like (work flow in this context is the sequence of user interface actions, not application logic). This is not to say that the user interface testing frameworks are not capable of fully automating the interface testing, it is a question of effort involved. A simple user interface validation test done by a human may take an hour to perform where designing a test may take much longer.
In the long run is it worth it? Maybe. If your application is user interface centric (some would argue that all applications be designed using the interface as a starting point), it might make sense. In this case, the user interface designers will most likely be designated to this task. They will not likely care about application work flow.
What if you are in a situation where the user interface takes a "back seat" to other "more important" tasks? Well, this is bound to happen. After all, a user interface with no application behind it isn't terribly useful. Perhaps what is needed here is two sets of use cases. One for the application, which is independent of how external actors interact with the system. And, one for the user interface that describes the interface cases in much detail while keeping the application cases vague.
There are several issues that these tools fail to catch. Mainly, the work flow errors. Your application could pass the GUI testing framework with flying colors and yet, once your application is human-accessible, it fails colossally. This is because the slightest misinterpretation by the testing framework as to what the work flow should look like (work flow in this context is the sequence of user interface actions, not application logic). This is not to say that the user interface testing frameworks are not capable of fully automating the interface testing, it is a question of effort involved. A simple user interface validation test done by a human may take an hour to perform where designing a test may take much longer.
In the long run is it worth it? Maybe. If your application is user interface centric (some would argue that all applications be designed using the interface as a starting point), it might make sense. In this case, the user interface designers will most likely be designated to this task. They will not likely care about application work flow.
What if you are in a situation where the user interface takes a "back seat" to other "more important" tasks? Well, this is bound to happen. After all, a user interface with no application behind it isn't terribly useful. Perhaps what is needed here is two sets of use cases. One for the application, which is independent of how external actors interact with the system. And, one for the user interface that describes the interface cases in much detail while keeping the application cases vague.
Subscribe to:
Posts
(
Atom
)