Splitting a big job into smaller jobs

Added by Kohsuke Kawaguchi, last edited by Kohsuke Kawaguchi on Jun 27, 2008  (view change)
Hudson
Home
Downloads
Mailing lists
Source code
Issue tracker
#
Documents
Meet Hudson
Use Hudson
Extend Hudson
Plugins
SCMs
Triggers
Build Tools
Build Wrappers
Notifiers
Reporters
Ext. Site Integration
Uploaders
Misc
Others
Container-specific Notes
Site Map

A build is normally a fairly sequential process, and in a big project, a full execution can easily take hours. While one could bring such a job on Hudson, a long turn-around time to the result tends to reduce the value of continuous integration. This page discusses a technique to cope with this problem.

The idea to is to split a big build into multiple stages. Each stage is executed sequentially for a particular build run, but this works like a CPU pipeline and increase the throughput of CI, and also reduces the turn-around time by reducing the time a build sits in the build queue.

In this situation, your earlier stage needs to pass files to later stages. A general way to do this is as follows:

  1. An earlier stage archives all the files into a zip/tgz file at the end of the build.
  2. Tell Hudson to archive this zip/tgz file as a post-build action, take a fingerprint of it, then trigger the next stage.
  3. The first thing the next stage does in its build is to obtain this bundle through the permalink for the last successful artifact, then unzip it. Do keep this archive file around because we'll take a fingerprint of it here, too.
  4. The build proceeds by using the files obtained from the earlier stage.
  5. Tell Hudson to fingerprint the zip/tgz file. This allows you to correlate executions of these stages to track the flow.

If you have more than 2 stages, you can repeat this process. In some cases, this "zip/tgz" file would have to contain the entire workspace. If so, the next stage can use the URL SCM plugin to simplify the retrieval.

issue #682 keeps track of an RFE to more explicitly and better support this use case. Please feel free to add yourself to CC, vote, and comment on the issue.

Labels

  Edit Labels
(None)