PatchHandlingUnificationRFC

Patch handling unification IRC logs and ideas

This is a wiki page to try to summarize ongoing IRC chatlogs to find a way to unify patch handling extensions and tools (mainly mq, attic, shelve and record).

Attempt at capturing the situation today

Today, we have a number of overlapping extensions for managing patches on top of Mercurial repositories: mq, shelve, attic, and pbranch. It would be desirable to prune this list, optimally down to just one. However, we likely don't want to lose any of the main functionality offered by the extensions today:

mq

Goals:

• UI close to quilt (http://savannah.nongnu.org/projects/quilt/)

• Maintain multiple, usually related patches.
• Typically for later inclusion upstream.
• Or for different local configurations (guards).
• Import patches from downstream developers.

Strong points:

• Patches, while applied, are normal revisions and can be emailed etc. as such.
• Can go back and forth between patches.
• Can reorder patches, as long as they don't interdepend.
• Can fold multiple consecutive patches into one.
• Has guards to conveniently enable/disable patches.
• Can easily promote patches to regular changesets.
• Can convert regular changesets into malleable patches.
• Can be used directly in your normal work repo.
• Directly supported on bitbucket.org.
• Patches are stored as files, so they can be viewed and edited as such.

• hg strip to strip unwanted history (dangerous, too).

• Is an established extension shipped with Mercurial.
• Is documented in the Mercurial book.

So-so points:

• Can collaborate on patches by versioning patch files.

Weak points:

• Collaboration is not straightforward (merging patches - yuck).
• Base revision is not clearly defined.
• Rebase is complicated (or used to be?).
• Updating patches is destructive, so it's easy to shoot yourself in your foot. Especially when distributing pending changes to their proper place.
• Updating a patch is a special operation. So needs special tool support (qrecord, qct).
• Actual intended patch dependencies are not explicit. Always assumes incremental patches.

Ideas to fix it

• records the parent/base revision of a patch.
• qsplit Splitting patch (or/and interactive options for qrefresh).
• integrated interdiff (diff of diff stool) for patches repository versioning.

pbranch

Don't call it p(atch)branch call it f(eature)branch

Goals:

• Maintain multiple, usually related patches.
• Typically for later inclusion upstream.
• Be better than mq for
  • collaboration and long-term patch maintenance,
  • traceability of changes to patches, and
  • distribution of changes to patches.

Strong points:

• Can go back and forth between patches.
• Can fold multiple consecutive patches into one.
• Maintains an explicit DAG of patch dependencies.
• Each patch has a clearly recorded base it officially applies to. So you always know which patches are already rebased, and which are not.
• Rebasing on upstream and on changes in base patches are normal Mercurial merges.
• Can collaborate on patches using normal Mercurial commands.
• Is non-destructive.

• Updating a patch is a normal commit. Tools like record, crecord, TortoiseHg etc. can be used.

• Can email patches directly.
• Has extensive documentation (Mercurial book style).
• May reuse merge conflict resolution (could be included in resolved).

So-so points:

• Can export patches as mq patch series.
• Needs sound understanding of multiple branches/heads and merge flow.

Weak points:

• Leads to intractable history in the pbranch clone (all these automated merges).
• "Applied" patches are not regular changesets (patches are slightly filtered diffs between branches).
• Patches will likely need to be linearized before submission to avoid potential fuzz when committers try to apply them. Currently, such linearization is permanent.
• Reordering patches can be non-trivial (backout or rebase needed sometimes). Better to start out with proper dependencies in the first place.
• Must be used in a throw-away clone of the main repo, unless you intend to eventually strip the patch branches.
• Not included with Mercurial. Tests and documentation currently not according to Mercurial's standard.

Ideas to fix it

• Adds a pfinish command to handy create the final changeset (using lbranch or garbage collected bookmark).
• Prevents pbranch managed branches to be pulled/pushed.
• Allows cherrypicking a single patch(branch).

shelve

Goals:

• Put aside a single piece of unfinished work.

Strong points:

• Simple to use.
• Some tool integration.
• Can be used directly in your normal work repo.

Weak points:

• Hunk selection always interactive.
• Not included with Mercurial.
• It's a subset of attic

attic

Goals:

• Maintain multiple unrelated, unfinished strands of work.
• Allow passing unfinished work on to colleagues.
• Easily and quickly store unfinished work away so you can come back to it later.
• Import singular, unrelated patches from downstream developers.

Strong points:

• Easy to use.

• Can untangle multiple unrelated strands (hg shelve --interactive).

• Can be used directly in your normal work repo.
• Patches are stored as files, so they can be viewed and edited as such.
• When working on a patch, it is right in the working copy (no refreshing changesets and such).
• Works well alongside mq/pbranch.
• Is non-destructive (doesn't touch history at all).
• Finishing a patch is a normal commit.
• Stronger unshelve as the base revision of a shelve is stored and used during the unshelve process.

So-so points:

• Can move patches back and forth to mq.
• Can share .hg/attic dir as a repo on its own, thus sharing patches

Weak points:

• Not included with Mercurial.
• Can get rejects when picking up a patch when base repo has evolved.
• Only deals with one patch at a time.

lbranch

Goals:

• Get multiple clone feature with a single working directory frontend.

Strong points:

• Hidden commit

Weak points:

• Can not share branches

So-so points:

• XXX

Ideas to fix it:

• Integrated collapse stuff.

bundle overlay

Goals:

• looks like lbranch but without actually cloning the repository.

Strong points:

• Hidden commit.

Weak points:

• XXX

So-so points:

• Nested overlay.

Ideas to fix it:

• XXX

List of use-cases

To aid in making a good decision about the way that the functionality of these tools can be merged, several use-cases are listed here (please contribute new ones if you have one in mind that is not yet listed):

• Barack is working on a single-branch repository, and realizes that he wants to try to experiment with coding feature A, without sharing any of this work yet. He signals his intent to hg, and proceeds to work on feature A, making several commits. After some time, he realizes that he won't get far until feature B is first implemented. He issues a command that starts a new branch parallel to the 'feature A' branch, and makes a few commits. When feature B is finished, Barack rebases the 'A' branch onto the tip of the 'B' branch, and continues his work on feature A. When he's happy with feature A, he cleans up the history of the B+A branch, and either makes it public, or merges it into a public branch.
• Jan has a repository on his machine with various non-public feature branches in it, and wants to locally (i.e. on his machine) clone the repository so that the clone contains all of the non-public branches. He also wants to make another clone which does not contain any non-public branches.
• Sarah has been working on several branches, both public and non-public, and decides to strip off a portion of one of the non-public branches. She continues working on this truncated branch, and then realizes that she stripped away some work which she really needs. She runs an hg command which launches a history browser, allowing her to see all the work she's done for the last 30 days, on both public and non-public (permanent and non-permanent) branches. She chooses a changeset, and a diff is computed against the working directory. She is given the option to apply any of the hunks/lines in the diff to her working directory.
• Eduardo has a repository he uses on his work computer with various public and non-public branches. He makes this repository available via hgweb, and restricts the access so that only he and his friends Renaldo and Maria have push/pull access to the repository. Renaldo is only given access to pull/push public branches, whereas Eduardo and Maria are (via an hgrc setting) given the option to add a flag when pushing/pulling so that they can push/pull both public and private branches. When Eduardo or Maria eliminate/restructure a private branch, and push with the "both public and private" flag, the private branch on the remote repository is also eliminated/restructured.
• Marty is using an OSS tool and finds a bug. He goes on the tool mailing list and learns that the list community submits patches inline in emails. After some encouragement, Marty grabs the source and writes a patch. After posting the patch somebody suggests that he split the patch up. So, Marty resubmits a new set of patches. At this point another person suggests a better way to solve Marty's problem and asks Marty to share his repository so that they can collaborate on the patches. A short while later the two submit a much improved patchset and an upstream developer says he is applying the patches.
• Jim is a full time developer for an application with an active mailing list based developer community. Somebody on the list found a bug and has written a patch for it, but it isn't up to the standards of the current codebase yet. Jim replies with a suggestion for improving the patch and says that in its current form, the patch will not be accepted. At the same time Jim runs his mbox-patch-importer and declines the new awaiting patch. Shortly later a better patch set comes along and Jim says he will apply these new patches. Around the same time, somebody else provides another patch set that Jim says isn't ready yet. At this time Jim reruns his importer and accepts the approved patches and declines the others.

Tentative proposal

Hidden branches/heads

Add a special kind of commits heads/branches that don't get transfered during pulling/pushing/cloning. Let's call it 'hidden' branches.

They'd get transfered just when requested explicitly or when converted to normal public heads/branches.

A possible way to mark those heads/branches is using an additional extra field, similar to how closed branches are done. Removing that extra field would "unhide" the head and the branch it tips.

An alternative implementation would avoid keeping the 'hidden' property in extra and use instead some tagging for bookmarks.

In some way, this kind of branches would be similar to git topic branches, or a way to do an hg clone/pull/push -r exclusion.

Applying hidden branches to unify mq/pbranches/attic/shelve

This new kind of branches allow for a safe playground zone without resorting to full separate clones. As they can be used without propagating its changesets to other clones they're suited for history altering changes like strip, rebase, qfold or just in-progress work that's not ready to be published.

With that, the mercurial queues, pbranches or attic/shelve stores could live in the main repo instead of outside repos and this would also ease rebasing or full 3-way merging. mq, pbranch or attic would be just different ways to interface with evolving changesets.

An outline of how mq, pbranches, attic or shelve could be implemented is given:

attic

attic's shelve could move the working dir changes to a commit in a 'hidden' branch and unshelve would just merge back those changes to the working dir without (optionally) commiting them. shelves could be still named and versioned (bookmarks and/or making it a named branch would do the trick). unshelve --delete would be just a strip of the hidden branch that keeps the patch

pbranches

pbranches could be also used without the need to clone the main repo to keep tentative patches, as the pbranches could be of the hidden kind.

mercurial queues

mq could use a hidden branch to store its patches info that gets rooted at the point (changeset) where the queue is needed. When time passes and new changes are added it could be possible to rebase the mq branch to a more recent changeset and use the full rebase machinery.

Alternative approach: Overlay repositories

Instead of using hidden branches, another possibility would be using overlay repositories, with similar behaviour.

Alternative approach: Tasks

Use tasks instead of hidden branches.

Alternative approach: Plans to make attic a superset of mq

The use cases for the existing patch handling extensions could be reduced to two different feature sets / models. One that allows the handling and management of patches and its conversion into and from commits, and another one that implements a way to collaboratively develop patches.

The first feature set would merge the capabilities of mq, attic and shelve, while the second one would correspond to pbranch. Naturally, the two extensions implementing these feature sets should be highly integrated with each other.

This proposal tackles the first part, as of right now attic is already a superset of shelve and I (BillBarry) have plans to extend the functionality of attic to become a superset of mq.

General interaction with patches

Interaction with patches can be viewed from different angles:

* from how they interact with the repo history and the working copy contents: they can be working copy changes or mutable changesets.

• As working copy changes, patches don't affect the repository history and are just a way to store changes using a name so they can be applied or unapplied later. They're used like that in the existing shelve and attic extensions.

• As mutable changesets, the existing changes are solidified as changesets in the repository history, so they can be rebased, merged, pushed or pulled... This is how Mercurial Queues view patches.

* from how they are created: they can consolidate existing changes in the working copy or used to stash away changes those changes.

• Using patches to consolidate changes in the working copy is useful when trying to evolve those changes or when they correspond to a changing base content which needs some adaptation in the form of additional patches, and giving these changes a name is very convenient. This is how Mercurial Queues generate its patches, taking them from existing content to later manage in a convenient way.
• Stashing away changes allows a more convenient workflow, avoiding having to mess with unrelated changes, so they can be temporarily put appart to concentrate on a different aspect and later can be reapplied. The existing shelve extension uses patches like this.

* patches can be a single set of changes or stacked, either in series, with a given order, or as unrelated changes.

• These series allow reordering of patches or partial application, allowing guards.

Once patches are created, there's also the need to refresh its changes, apply or unapply them to the working copy or repository history, and delete them.

Conceptual user interface

Patch creation:

• Create named patch and keep related changes to the working copy
• Create named patch and stash away related changes to the working copy

Patch management:

• Apply changes from named patch (or series of patches) to the working copy
• Unapply changes from named patch (or series of patches) to the working copy
• Apply changes from named patch (or series of patches) to the repo history
• Unapply changes from named patch (or series of patches) from the repo history
• Set guards for a patch

Patch transformation:

• Refresh changes belonging to the named patch
• Transfer content between patches (rename, fold, split, join, rearrange...)

Patch removal:

• Delete this patch name or named series of patches

Patch status:

• Status information (changes, guards,...)
• Diff

Simplified proposed user interface for attic

(none of these commands necessarily have their final names yet)

• Patch creation

  • store - Create named patch and keep related changes in the working copy: