Invoking Buildbot builds from CLI

Post by Jalmari Ikävalko

So, recently I decided to try out Buildbot, a continuous integration tool with emphasis on programmability and flexibility.

After setting up basic building procedures, I wanted to integrate Bitbucket into the mix for triggering these builds when a commit has been pushed. From some reason, I couldn't get Bitbucket polling to work, while BuildBot's www hooks seemed to be using a legacy API. Being the practical person that I am, I decided to sidestep my problem instead of fixing it!

..But I wound up spending the better half of the day with that sidestep. I figured that I could put up a simple HTTP server, which listens to POST requests from Bitbucket commit change hooks and invokes Buildbot build procedures from the command line. Easy enough?

Well, since I am a stubborn idiot who rather continues following his intuition than reads actual documentation with any focus, it took a while to get it working.

I'm documenting my poor solution here just in the case that someone ever has a similar problem.

In the end, my builders looked like this:

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a
# what steps, and which slaves can execute them.  Note that any particular build
# only take place on one slave.

# Create steps for each repository in the project
api_build = BuildFactory()

for repo in repos:
    repo_url = "git@bitbucket.org:ourUserOrCompany/" + repo + ".git"
    api_build.addStep(steps.Git(
        name = repo,
        workdir = repo,
        repourl = repo_url,
        mode='full',
        method='clean',
        submodules=True))
    api_build.addStep(steps.ShellCommand(
        command=["npm", "install"],
        workdir=repo))
    api_build.addStep(steps.ShellCommand(
        command=["npm", "start"],
        workdir=repo))

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="api",
      slavenames=["staging-slave"],
      factory=api_build,
      builddir="api"))

We have multiple repositories for a single project. There's overhead in doing it like this; In reality, building and running each repository should happen inside its own BuilderConfig, but for convenience's sake, I did it like this.

Then, for schedulers:

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In t
# case, just kick off a 'runtests' build

c['schedulers'] = []
c['schedulers'].append(
    _schedulers.SingleBranchScheduler(
        name="all",
        change_filter=ChangeFilter(category="api"),
        treeStableTimer=2,
        builderNames=["api"]))
c['schedulers'].append(schedulers.ForceScheduler(
        name="force",
        builderNames=["api"]))

Here we have the usual ForceScheduler and a SingleBranchScheduler, which is triggered on any changes in the category "api".

Finally, our changesources:

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.  Here we point to the buildbot clone of pyflakes.

c['change_source'] = []
c['change_source'].append(changes.PBChangeSource(port=9999,
    user='manual',
    passwd='password'))

Here, we have a PBChangeSource, listening to any connections on the port 5024.

To trigger a change, I set up a simple web server. It listens to connections and upon receiving a POST message, triggers a build. It looks like this:

#!/usr/bin/python
import time
import BaseHTTPServer
import subprocess

HOST_NAME = 'my.doma.in'
PORT_NUMBER = 9000

class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
        def do_HEAD(s):
                s.send_response(200)
                s.send_header("Content-type", "text/html")
                s.end_headers()
        def do_GET(s):
                """Respond to a GET request."""
                s.send_response(200)
                s.send_header("Content-type", "text/html")
                s.end_headers()
        def do_POST(s):
                """Respond to a POST request."""
                s.send_response(200)
                s.send_header("Content-type", "text/html")
                s.end_headers()

                if (s.path == "/build/api/my_secret_key"):
                        print("Starting to build API..")
                        subprocess.call("cd .. & buildbot sendchange --master localhost:9999 --auth manual:password --who buildbot -C api", shell=True)

if __name__ == '__main__':
        server_class = BaseHTTPServer.HTTPServer
        httpd = server_class((HOST_NAME, PORT_NUMBER), MyHandler)
        try:
                httpd.serve_forever()
        except KeyboardInterrupt:
                pass
        httpd.server_close()

Now the URL http://my.doma.in:9000/build/api/my_secret_key can be given to services such as BitBucket or GitHub to call when ever there's a commit, thus triggering a build.

The important part here is the ChangeSources and PBChangeSource. With PBChangeSource set as it is, you can trigger a changeset to propagate through BuildBot via CLI by calling buildbot sendchange --master localhost:9999 --auth manual:password --who buildbot -C api. The last parameter - -C api - sets the category of the changeset. Then, Schedulers are configured to listen to changes in the category "api". When one such change is spotted, the builders are called.

So here it is, how I set up triggering builds in BuildBot from BitBucket or GitHub by using the CLI. If you happen to be new to BuildBot, instead of looking at this post, you ought to be checking out Buildbot Tutorial and after that, Buildbot in 5 minutes!