A recurring theme in my software development life

28May10

Apparently I need to learn to search better, since I really tried this time.

In order to notify our Buildbot of changes in our source repository at Bitbucket, we’ve been manually pulling from Bitbucket to a repo on a development system here.  The local repo had a changegroup hook that utilized the standard buildbot.changes.hgbuildbot hook.  This worked fine, but it was tedious, so I decided to work on making Bitbucket trigger Buildbot itself.  Bitbucket offers various services that can run on commit, but no direct Buildbot notification.  But, they have a generic POST service that’ll send a specially formatted payload to an HTTP address of your choosing.  I did a quick Google to see if anyone had already written a connector.  Finding nothing and figuring it’d be an easy task, I set about writing my own.

The result is bb2bb, which (in its original form) was a FastCGI daemon serving a WSGI application (since nginx and flup make that setup easy) that reused a bunch of code from hgbuildbot.  I also employed some modules I’d used before (like WebOb), and it came together after most of a day of work and testing.   Unfortunately, after I put it in production I found a problem: after 2 successful commits and subsequent buildbot change notifications, the application would stop sending notifications.

Until this point, I had been blissfully ignoring the inner workings of Twisted since it’s a bit complex and I didn’t want to take the time to learn it when it only seemed vaguely necessary for what I was trying to do.  Unfortunately, I discovered that I couldn’t start and stop a Twisted reactor from within my WSGI application like I was trying to do.

Somewhere after a few hours of trying to wrap my head around Twisted so I could fix this, I happened to take a look at the list of files in Debian’s buildbot package and discovered the following:

% dpkg --listfiles buildbot                                                                                                                                                                                                                                          /usr/lib/python2.5/site-packages/buildbot
/.
/usr
/usr/share
/usr/share/buildbot
/usr/share/buildbot/contrib
/usr/share/buildbot/contrib/svnpoller.py
/usr/share/buildbot/contrib/bb_applet.py
/usr/share/buildbot/contrib/README.txt
/usr/share/buildbot/contrib/svn_watcher.py
/usr/share/buildbot/contrib/generate_changelog.py
/usr/share/buildbot/contrib/git_buildbot.py
/usr/share/buildbot/contrib/run_maxq.py
/usr/share/buildbot/contrib/fakechange.py
/usr/share/buildbot/contrib/hg_buildbot.py
/usr/share/buildbot/contrib/darcs_buildbot.py
/usr/share/buildbot/contrib/svn_buildbot.py
/usr/share/buildbot/contrib/bzr_buildbot.py
/usr/share/buildbot/contrib/arch_buildbot.py
/usr/share/buildbot/contrib/github_buildbot.py
/usr/share/buildbot/contrib/viewcvspoll.py

Hmm… a github to buildbot connector?  I checked it out and (no surprise) it does for github exactly what I was trying to do with Bitbucket.  So I spent another hour integrating my existing code (in the process, removing all those things that I didn’t need (like flup and WebOb) because I learned that Twisted already has the functionality) and an hour testing it.  Once it was all working, I went to modify setup.py to reflect all these changes and I decided to change the name to bitbucket_buildbot.py to match the stuff already in contrib.  At this point it struck me to Google that name.

ARRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRGH.

Someone had already done exactly what I just spent all this time doing, and it was committed in Buildbot’s contrib/ directory.  It wasn’t in my buildbot package since my version was older.  It, like mine, is even based on github_buildbot.py.

So I just spent almost 2 days doing something that’d already been done because I didn’t look in the right place with the right name.  Oh well, it’ll sit there on Bitbucket as a monument to wasted effort. =P

Advertisements


One Response to “A recurring theme in my software development life”

  1. 1 John Chadwick

    Thank you for documenting this. I was just about to make the same mistake.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: