Something is rotten in the build!

Don’t you just hate it when something breaks the build of your software?  Build problems create extra, unplanned work. They always seem to happen when you are rushing to solve some other problem. Often, the build breaks for some factor that is unrelated to anything you are doing at the moment. So it can be rather frustrating to have to drop what you are doing, dig into what caused the build to break, fix that problem, and restart the build (hoping you have only one problem to resolve).

I ran into this issue today, while rebuilding some open-source packages. A few months ago, we updated our copy of OpenSSL to version 1.0.0, but we didn’t rebuild all of the packages that depend on it. As it happens, when I went to rebuild the subversion client to pick up a bug fix, I found that the subversion build was broken due to the fact that OpenSSL 1.0.0 introduced some incompatible changes. It didn’t take me long to find the fix, because this issue had been corrected months ago, and a simple web search found both the bug report and the fix. But it was still rather irritating to run into this problem when I was just trying to rebuild the product.

I don’t have a brilliant suggestion to make here. But I do have an observation, which is that machine time is now plentiful. Maybe we should simply rebuild all of our software every night. That will catch build problems when the issues are still fresh in everyone’s mind.

Also, just in case you have some manual steps in your build process, deciding to rebuild frequently will be a good incentive to eliminate them. Manual steps are a common source of errors.

Software testers tend to dislike rebuilding software once they have tested it. They see it as an opportunity to introduce new defects. I like to have enough confidence in my build tools (compilers, linkers, makefiles) that I can rebuild at any point in time. Given that we are working in high-level languages, we must have confidence in our build tools.  But there is room enough for both views in a phased development process.

If you have suggestions on how to manage software development to minimize the chances of breaking the build, or of minimizing the effort required to correct build problems, please add your comments to this post.