Introduction to the arch patch queue manager

Copyright 2003, 2004 Colin Walters, Walter Landry. Permission is granted to copy, distribute and/or modify this document under the terms of the GPL.

Revision History
Revision 1 2003-10-16
First version.
Revision 2 2003-11-15
Updated for release 0.3.2.
Revision 3
2003-12-23
Updated for new commands
Revision 4
2003-4-18
More updates for new commands

What problem does it solve?

The idea is simple. You have a project with a number of developers. With a revision control system like CVS, it's obvious that the project code will be kept in a single repository, which all the developers use. You really don't have much of a choice.

But arch is fully distributed. You want to take advantage of those features, allowing your developers to commit while disconnected (say while they are travelling with a laptop), easily create their own temporary branches without affecting the main repository, and more. To accomplish these things, each developer needs to have their own arch archive.

This then raises a question - where is the project? One solution is to pick a specific developer to perform the task of merging in the other developer's code. That developer's archive becomes the canonical one for the project.

However, there is a better way. The main idea of the arch patch queue manager is to have a special archive which is managed entirely by the patch queue software.

How it works

You create the distinguished archive to be managed by the patch queue manager. Then, every developer branches off of it. When a developer has reached a milestone and wants to sync up the main tree with their archive, they submit a merge request.

Essentially, this just leverages arch's "star-merge" functionality to perform the actual merge.

Handling conflicts

One question that arises at this point - what happens if there's a conflict? If two people submit merge requests, the second could conflict with the first. Obviously the patch queue manager is not an AI; it can't automatically resolve all conflicts. So the simple solution is to just to inform the submitter of the second patch of this fact, and reject the merge request. They can then sync up their tree with the main branch, and resubmit the merge request.

Running tests

However, textual conflicts aren't the only kind of conflict; you could also have _semantic_ conflicts, where the patches don't touch the exact same portion of code, but taken together they break it. One way to help solve this problem is to run a testsuite before every commit to the main archive. ArX has precommit hooks to cover this, and the arch patch queue manager implements its own precommit hooks for other arch implementations. You can actually do pretty much anything you want inside this hook.

Security

You have two choices for submitting merge requests; one option is to send them via email. arch-pqm supports GPG-signed messages for security.

If it can be arranged for developers to all have write access to the same filesystem (via NFS/SFS, over sftp, etc), then you can have them simply drop merge requests into a special queue directory.

Setup

Requirements

  1. Python 2.3 (or Python 2.2 and the python-logging module)

  2. GNUPG (optional, but highly recommended)

Compilation and installation

  1. Use the normal configure, make, make install commands to install the software.

  2. Next, you need to create a ~/.arch-pqm.conf configuration file. This lists the location of the queue, the path to the GNUPG keyring, whether or not to verify signatures, etc. In addition, the arch-pqm.conf file also specifies which archives are valid to merge into. This is not checked for all commands.  In the future, it will also allow specifying a keyring per-archive, and some other nice things.

    Please see the example file as a starting point.  Note that if you want to use an arch implementation besides TLA, you must set the tlapath argument in .arch-pqm.conf.

  3. Set up arch-pqm to process requests.  If you are using a shared filesystem, then you can just drop requests into a the queue directory.  If necessary, set up arch-pqm to process requests through email.  You will likely want to use procmail. If you have an account dedicated to processing merge requests (a good idea), you could use this sample .procmailrc entry:
    :0:
    | arch-pqm --read

    If you don't have a dedicated account, you will have to pick a specific Subject: or other header to use to differentiate merge requests from your regular email.  This will scan and verify any emails that come in.  You will need to populate the patch queue's GPG keyring.

    To actually perform these actions, you need to run

    $ arch-pqm --run

    That simply iterates through the pending merge requests in order (by mtime), and processes them. One way to run it is via a cron job.

  4. Create any precommit or postcommit hooks, and my-id.

Almost everything else can be done using with arch-pqm actions.

Using arch-pqm

Actions

There are currently nine different actions that arch-pqm can do.  The first five commands

  • make-archive ARCHIVE-NAME ARCHIVE-LOCATION
  • register-archive ARCHIVE-NAME ARCHIVE-LOCATION
    or register-archive -d ARCHIVE-NAME
  • archive-cache-revision ARCHIVE/REVISION
  • archive-uncache-revision ARCHIVE/REVISION
  • tag FROM-ARCHIVE/FROM-REVISION TO-ARCHIVE/TO-REVISION

function exactly the same way as the corresponding commands in arch.  Note that you must fully specify both the archive and revision.  The patch queue manager does not, in general, have a default archive.  It does have an id, which you can set with my-id
  • my-id First Last <email@example.org>
For merging, use
  • star-merge FROM-ARCHIVE/FROM-REVISION TO-ARCHIVE/TO-REVISION
This performs the merge and commits it to the archive, using the Subject: line of your merge request as the commit message.

To create a new branch
  • create-branch FROM-ARCHIVE/FROM-REVISION TO-ARCHIVE/TO-REVISION
and to create a completely new line of development
  • create-version ARCHIVE/VERSION
Again, the commit message will be the Subject: line of your merge request.

So the first two commands you will use are probably my-id and make-archive.  If you are just starting development, you will then use create-version.  If you have already started development, you will use create-branch.  The developers then branch off of the patch queue manager's branch.  When the developers are ready, they will merge back with star-merge.  You will use archive-cache-revision and archive-uncache-revision to speed up get's and free up space. Finally, you will use tag to mark a release.

Submitting a request via a shared filesystem

If you can provide all of the developers with write access to the same filesystem, then you can simply have them place request files in the queue directory, using a small shell script such as this:

#!/bin/sh (echo "From: $(arch my-id)"; echo "Subject: $1"; echo; echo
star-merge "$(arch tree-version)" "$(cat {arch}/+upstream)") >
/usr/src/arch-pqm/patch.$(date +%s)

Save that as arch-submit-merge. Then you would use this script by saying:

$ arch-submit-merge 'fix lots of bugs'

Other actions should work as well.  In the future, we would like to have better support integrated in arch for this kind of thing.

Submitting a merge request via GNUPG-signed email

If your site is widely distributed, or giving shell accounts to all developers doesn't work well, you can also use GNUPG-signed email. Here's how you could modify the script above:

#!/bin/sh
echo star-merge "$(arch tree-version)" "$(cat {arch}/+upstream)" | gpg --clearsign | mail -s "$1" "$2"

You might use the above script by saying:

$ arch-submit-merge 'fix bugs' pqm@example.com