Open Source: The Just-Do-It-Yourself Argument
A few of my fellow developers and I were recently tasked with bolstering the test suite which runs against our commercial framework. As I discussed in part two of The Summer of Bloated Software, HttpUnit, one of our shop’s favorite tools for testing, simply could not do the job due to the unusual way in which our commercial framework generated form fields. This meant that several important use cases could not be tested with HttpUnit, and that we were going to have to have someone manually test them.
First off, human unit tests are hard (without brainwashing) to automate, and secondly, the word manual is painful to any programmer’s ears – we are lazy and hate such exercises in brutish grunt work. We would rather write a script to do that grunt work for us.
Yet, poor HttpUnit, which is very script-like in its nature, was failing to meet our needs. This got me thinking. None of us wanted to do any manual work, so we could either write a patch for HttpUnit, use another HttpUnit-like framework, or write a new framework altogether. I immediately knew I did not want to write a patch for HttpUnit. Diving into someone else’s code takes time and patience, and at the moment I did not have a lot of either. So, I decided to start looking around the Internet for some HttpUnit-like alternatives. The first, and probably most readily available, replacement that I found was HtmlUnit, which uses the HTML document itself for performing unit tests, whereas HttpUnit makes use of the response. While I am certain that HtmlUnit’s approach has its merits, HttpUnit was nice and familiar, and I didn’t want to stray too far from it. I continued my searching but found nothing that looked promising. I was left with the final option. Write my own framework.
This brings us to another one of the hot-topic issues surrounding open source. The “just-do-it-yourself” argument. This can be seen as either a wonderful boon, or a terrible burden. It can be a burden when you are relying on a piece of open source software, and you need a feature that no one is interested in adding (including yourself). However, it can also be a great boon when you are out of alternatives, and you need something that works, in which case open source tools can give you what you need to start implementing a solution.
Having done some preliminary research, and finding no suitable replacement frameworks, I decided I would simply write my own HttpUnit-like framework. I knew I needed more control over the post data that was being sent to the server than HttpUnit could provide, so I chose to use an Apache Commons project called HttpClient to handle my getting and posting. I also need a tool for parsing HTML so I could turn the server’s response into something more useful from a testing perspective. For this task, I first tried using Jtidy, which I had limited success with. Ultimately, it was not sufficient to do the parsing I desired. Next I tried a Source Forge project called HtmlParser. The name says it all. It parses HTML! It also uses a marvelously simple approach to parsing HTML through the use of extensible filter classes! Brilliant! HtmlParser proved to be an easy an effective tool for parsing the HTML the server returned to me. I then set to work creating a framework around these two tools, which made writing tests with our custom framework as simple as writing tests with HttpUnit. By combining the parser and the client together, along with my overlaying framework, I finally had an easy way to create posts to the sever, but with a level of control that HttpUnit couldn’t offer us.
So, in this particularly instance, the “just-do-it-yourself” argument turned out to be a boon. However, the complexity of the test framework I wrote was not inherently great. It was simple enough that it only required the connecting of several solid, existing open source tools. A larger project may have proved more troublesome; however, in this instance the usefulness and flexibility of open source seemed to show itself once again! What kind of experiences have you had as a developer working with open source? Have you ever had an open source project blow up in your face, or create a major road block?