It’s Time We Stopped Rewarding Projects with Crappy Documentation: Open Source is No Excuse

In the last month, I shared my experiences with Spring Batch—not all positive—which led me to ponder the political correctness of criticizing open source software. Time to move on.

This week I’ve been writing unit tests for Java. In the past I’ve always used JUnit for no other reason than it’s the first such unit testing framework I learned. When doing repeated tests, you have a couple of options.

Firstly, you can create a single test that loops across a list of inputs and expected results. For example:

public class RepeatTest {
  private final static List TESTS = Arrays.asList(...);

  public void testData() {
    for (TestData data : TESTS) {
      assertEquals(doSomething(data), data.getExpectedResult();

You get the idea. This of course is coarse-grained.

Wherefore Art Thou, Documentation?

I had a recollection from years past that you could create a parameterized test set up in JUnit (although I had to google to figure out that’s what it was called) so I went looking for info on it.

Just take one look at Seriously, take it all in. I have never in my life seen a site so woeful for a framework that is as commonly-used and allegedly mature as JUnit. Considering I’ve seen Apache projects whose documentation borders on the criminally negligent, that’s really saying something.

Take Log4J as a prime example. I will say that the introduction to Log4J is quite reasonable. The problem? That’s all there is. Sorry, let me retract that: there is documentation. You just have to pay for it. I’m not sure how much influence Ceki Gülcü—the author of that book and the guy who started Log4J—still has (he has since moved on) but if anyone who was running a project (and thus deciding to a large degree what the priorities are and who does what) then if they were also selling documentation to that same project it in the very least has the appearance of a conflict of interest.

But back to JUnit. Consider the documentation. The "cookbook" is only a cookbook in the sense that “tear lid and microwave on HIGH for 8 minutes” on the back of a frozen dinner can be considered a recipe. There is an example of what I was looking for in the Javadoc for @Parameterized but the real answer came from the blogosphere, notably Writing JUnit 4 parameterized tests with varargs data and a few others.

I’ve previously evaluated TestNG as a unit testing framework. Although there didn’t seem to be anything wrong about it there wasn’t anything really compelling either. Test dependencies and the like just seemed complicated to me and not how I tend to use unit testing. Plus you generally find people know JUnit works and people know how to use it. Well, the basics anyway: @Before, @Test and @After, which covers 95%+ of cases.

It’s worth nothing that TestNG has extensive support for parameterized tests.

It’s the Documentation, Stupid

That’s not a compelling reason to switch though but I’ve now found one: documentation. I am gushing like a schoolgirl over TestNG’s documentation. There are examples of how to integrate into different IDEs—not just Eclipse (my pet peeve as a diehard IntelliJ IDEA user). The user guide is detailed and extensive. There is even integration documentation for Ant and Maven as well as a migration guide for JUnit (here is a comparison).

Documentation alone is reason enough I will in future choose TestNG over JUnit without exception where I have a choice.

Some open source projects have excellent documentation. I consider Spring documentation to be the gold standard of OSS documentation. Even Spring Batch, which I criticized, has good documentation. It’s not perfect. It’s coverage in some areas is a little light and it could use some more examples but overall it’s pretty good. TestNG documentation is excellent.

This will inevitably invite several counterarguments.


This one is predictable. Some will argue that the source code is sufficient documentation. Bollocks to that. While at some point it is inevitable you will end up reading or stepping through the source code of any framework or library you use for any non-trivial purpose, the fact that you have the source code is no substitute for high-level documentation that describes the overall architecture and design principles as well as how to get started and how to do common tasks.


Another popular defence of poor OSS practices is you can get involved and do it yourself. While theoretically true it is typically completely impractical. For one thing, before you can document something you have to know it. How do you learn it without reading the (non-existent) documentation?

Depending on the size and complexity of the project in question, it might takes anything from days to months or even years to get up to speed (eg the Linux kernel). Even spending days getting up to speed on something that isn’t documented is typically time you don’t have because you have a job you’re trying to get done.

It’s also unlikely to happen. Once you figure out how to do something, what are the chances that someone will then turn around and spend a few more days writing up some half-decent documentation? Chances are they’ll be so frustrated by their efforts to date it’ll simply be time to move along to the next problem.

It’s Free

This is the one that bothers me the most. It comes up in any criticism of open source and is little more than a justification for laziness. The idea seems to be that you can’t complain if it didn’t cost you anything.

Oh but you can.

You see, many people approach open source projects as a means of self-promotion. They simply want to make a name for themselves. That’s fine. I have no problem with that (I am after all writing a blog). The problem is it’s not about you.

Depending on your job, your company and your jurisdiction, if you are a consultant of some kind and thus dealing with people who don’t know a lot about software development you will typically have a duty of care to that party. You are making representations regarding your professionalism, capabilities, skills and/or experience. That other party is making has legal recourse for negligence (or even fraud) should you misrepresent yourself or fail to fulfil your duties.

When you create a library or framework or just some significant piece of code that you decide to share you are making representations—implicit and/or explicit—regarding it’s efficacy. What’s more your users will typically know a lot less than you.

What you should remember when you hand out “free” software is that if that software doesn’t work as advertised, it’s unreliable or the documentation is non-existent or misleading then you are doing it’s users harm. While you may not be legally liable for such deficiencies you have an ethical responsibility to ensure what you deliver does what it says it does.


Open source is no excuse. It costing nothing is no excuse. Like doctors, we programmers should first do no harm. We don’t have an Hippocratic Oath but that doesn’t mean we shouldn’t do what’s right.

I for one am tired of accepting mediocre libraries with documentation that is coming Real Soon Now [tm] (“Under Construction”). We should not tolerate second-rate offerings. Reward projects like TestNG who have done the right thing by us.

I don’t care if writing documentation is not fun or sexy. I care that you can explain how it works and provide me with some confidence that it does what you claim it does. Otherwise I’m just not interested anymore.


Anonymous said...

How about offering $50 or $100 to the developers as an incentive to sit down and write proper docs? You could volunteer to edit it as well.

Juergen Brendel said...

On one hand, the stuff is indeed free. Whoever provides it can chose to document it (or not) any way they see fit.

On the other hand: Just vote with your "wallet". In the case of open source this means: Don't use stuff that's not documented.

Lack of documentation is a terrible thing and I usually don't bother putting up with undocumented software. If people want their project to be known and used then they are shooting themselves in the foot by not providing docs as well.

Anonymous said...

OMouse: How about developers being professional and documenting their F'ing code. I for one, am sick going through source (open or not) which is not documented.

That practice is doing only half the job.

Imagine if a surgeon decided that it was too time consuming to give a patient post-op care instructions. Instead telling them RTFM (M being a medical book). Ok so bad analogy b/c most people pay large amounts of money to that surgeon (directly or indirectly).

Now what if that surgeon was giving away his (or her) services for free. Would RTFM still be acceptable? Or would there be outcry?

liamoc said...

I will share a personal philosophy.

If you have the energy to complain about something, you have the energy to change it.

The reason I got involved with OSS development in the first place is because I didn't want to be the person who whined about things not working in my software, and doing nothing myself to help it. I didn't want to be a complainer: I wanted to be a changer.

Changers do mostly the same things as complainers, but they go just a little bit further. If they want a feature, they'll put a good argument based on practical use cases for it, or they'll look at the source and try to effect the change themselves. If there's a bug, they'll try to isolate the cause, pick up common symptoms, try to boil it down to something reproducible. If something doesn't quite work, they'll try to find out why it doesn't work. If there's something they personally want, they ask themselves if the main users would appreciate it too. Complainers do none of these things; they only get upset when their requests get turned down.

When it comes down to it, people need to ask themselves: am I a complainer, or a changer?

Raoul Duke said...

@complainer, or changer?

complainer. i mean, as if i have any free time?

Anonymous said...


I will share a personal philosophy.

If you have the energy to design, release and then support software (on a forum/mailing list/IRC channel) for the next ten years, you have the energy to write documentation.

Depending on other people to get so frustrated with your poorly-designed and poorly-documented software that they write it for you is clearly a sign of a lack of commitment. If you are so incapable at English that you cannot even make an attempt at documentation, at least put out a request for technical writers and simultaneously offer to answer any questions that they have. Good documentation writers are not expected to be experts at using a product. They're expected to be given proper specifications and scenarios from which they can draft proper documentation.

Developers get upset when users complain, but the reality is that developers wouldn't see the importance of the deficiency without that complaint being voiced again and again. Priorities are very different between the users and developers of a product. Value your feedback, instead of deriding it because they're not doing your job.

Remember, it doesn't matter if it's free as in freedom or free as in beer if your software has a negative net value to the consumer. Free manure is good for the people that want it, but to the rest of us it just stinks.

Anonymous said...

I've probably written free code you've used.

You have an ethical responsibility to send me money. And a pony too.

Anonymous said...

Why didn't you write some documentation you miss instead of this blog post?

Anonymous said...

It's a vacuous argument you present. A library will be used (among other things) if a) it is easy to figure out by itself, or b) it has good documentation. The fact that some 3rdparty tool is very well used means that one of the above is true. If it wasn't true, it wouldn't be used much. The fact that JUnit is used heavily means that the documentation or ease of use is good enough.

Anonymous said...

Wherefore art thou != where art thou

Anonymous said...

Kamatsu:"If you have the energy to complain about something, you have the energy to change it."

That is absolutely incorrect. Complaining takes about 10-30 seconds to write a rant. Fixing it requires in depth knowledge of the software.

I am no fan of poorly documented software, but its the Open Source Project owners prerogative to fix their own documentation.

We are using open source software based on its quality, and documentation is part of it. If the overall software was that bad, you wouldn't even bother to use it.

Unknown said...
This comment has been removed by the author.
Emanuel said...

Voted-up at DZone for truth.

William Shields said...

As I pointed out, writing documentation requires a certain level of expertise with the software.

And as for me personally writing documentation, I've written 5-6 tutorials for Ibatis that cover a lot of the features I feel are inadequately covered by the official documentation. Not that the Ibatis documentation is bad. It's just feature-incomplete.

There are different ways of contributing. For me to date that has involved answering 1000+ questions on Stackoverflow and posts for this blog.

The idea that you can't criticize something where you could theoretically do it yourself (given enough time) is ludicrous.

Mark said...

How about contribute the documents to the open source community by yourself?

Not only "DIY" and use it privately, but also join and help to complete the project?

And, it will be more convincing.

William Shields said...


1. Yeah because spending the months or years to become sufficiently expert to write the documentation for EACH library that has woeful documentaiton is practical; and
2. Your argument is basically analagous to "You can't criticize the government because you can always become president and change it".

Mark said...

I wasn't argue your points, I just kindly try to ask you join and contribute to the community.

As a open soucre developer for past 5 years, I saw a lot people always blame everthing, when I ask they to contribute .... no one like/want to :)

Just a few people can contribute their time, works or money ...

So, If you really want to ask these open source communities to deliver something you want, no matter documentes or functions or anything. Just donate some money to ask or do it yourself and contribute...

And, it will be more positive.

Or, just use another library :)

It's all up to you.

Matt said...

There's nothing wrong with offering a critique of a piece of open source software. However, it does seem a little churlish is to gripe and whinge about it if it's not as good as you want. Essentially, you are criticising people for not having done work, for free, that *you* want. You have no such entitlement.

People do open source for a bunch of reasons, but primarily it's to scratch their own particular itch. And then they share the code. If you happen to find it useful, great.

Again, critique != complain.

Pointless Researcher said...

Do it yourself! Really

Hal Helms said...

Isn't "free" software wonderful? The idea that open source is great because it's free is what leads to this -- and so many other -- situations. It's old school thinking that says "Money is scarce; time is abundant". All this could be solved if there was a financial incentive -- i.e. pay -- for software. But as long as the fetish of "free" software persists, we will pay and pay and pay in time.

Unknown said...

Really. Thumbs up man.

I would like to expand my point though. I'm a big fan / user of free software. And lots of times I've been "overwhelmed" by the lack of documentation. As said in the article if you want to write good documentation, documentation that's WORTH reading you need to know what you're talking about. The only person who can do that is the person who writes the code.

It's my point of view, feel free to criticize. But I really believe in that and I'm having lots of hard times trying to convince my colleagues and friends to write good documentation. It's not that hard really.

Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Chris said...

Having a lack of documentation for your code is shooting yourself in the foot in my view, as you will end up (if it's successful) having to answer a tonne of emails or forum posts. A few walkthroughs and API docs is about completeness and knowing you've finished your project properly.

Doing documentation for an entire-library after it's complete is also really tedious. Doing it piecemeal when you finish each class gets rid of that.

shevy said...

It is because Java is over-verbose.

Devs hate to write docu if they need to spend as much time as on the code before...

Anonymous said...

I agree with you, time to boycott oss that doesn't have good documentation. Its almost ubiquitous in the Ruby community, someone announces an amazing new lib, that revolutionizes programming, and writes itself while you sleep.. Ah, but wait..they were too busy adding new features to write any documentation. Or better yet, just look in the RDoc.. (which is what the poor excuse that 90% of them call documentation) Seriously, screw that.

Anonymous said...

Some developers don't realize that their project is more likely to be adopted if the documentation is more thorough. We are all busy - people don't have time nor energy to figure out how your stuff works.

Essentially, if you want the project to be *used*, documentation is almost THE key.

However, it IS difficult to spend hours writing documentation, since our free time is precious, and we like to use it for writing code, eh.

Anonymous said...

I remember reading about a feature in Blender (the water simulation I think) where someone was saying "I'm happy to write documentation for this. Can I have your test files? Certainly you tested it, but I can't even figure out how to start describing the parameters. Please?"

If you're not going to document stuff, at least don't be aggressively secretive. :-)

Post a Comment