Changesets Evolution - development page.

This page is intended for developer

For a user perspective have a look at the ChangesetEvolution page.

1. Contributing

The simplest way to help is to grab one of: list of easy bug

There is also a multiple well defined topic that where idea exist but needs an implementation

• Rebase could make more use of obsolescence marker:
  • detection that part of the rebase set is already in the destination
  • warning about divergence creation.

• Obsolescence markers exchange (no really we have idea waiting to be implemented)

• Prototype for a command bringing changeset back to life.

There is more complicated part that requires attention too.

• Better storage//loading//cache for markers (depends on marker exchange)

• Solving the problem of Topic Branch

• Handling moving changesets around with the "plan" concept,

• in memory merge (helps a lot of troubles resolution throught evolve)

• Wide transaction capability
  • Continue/stop/abort for all command including evolve
  • In transaction content non exchangeable.
• Making evolve capable of solving all troubles that user can encounter
• Computing UI message about troubles from event who happened during the transaction.

2. Roadmap

Current status:

• Alpha Stage,

• Beta Stage,

• Release Stage,

2.1. Stage A

Changeset Evolution is currently at Alpha Stage. If won't eat people data, but only handful of people knowns how to get out some situation..

2.2. Stage B

All the "core" features should be here and somewhat work. Some will be sluggish and unpleasant to use as the focus is not yet on user interface.

We can gather the base-line expectations for this stage in 4 groups.

3. Troubles Resolution

Users need to know that he can trust the tool to offer him what it needs and bring him out of trouble. We have already cover a good share of that work with a descent hg evolve command handle most common case and a large set of small commands to each operation. Yet there is some area where the user is still left with no "simple" option.

• Evolve should have predictable result (--rev options and co),

• Evolve should be able to somewhat handle all possibles troubles (including
  • divergence and bumping)
• Evolve should have a working --abort and --continue (and --drop?)
• We must have some command to "bring obsolete changeset" back. (issue4851)

4. Displaying Evolution History to the User

Now that we (almost) have all the brick to build a clean evolution history, there is a lot of way we could easily expose it to the user. This is critical to ensure the user have some awareness of the feature and is able to understand what is happening in case of troubles.

• Having "hg evolve --list" displaying troubles changesets, trouble affecting
  • them and details of the diagnostic

• Basic display of precursors/successors available in hg log

• Some highlight of obsolete//unstable node in default log (color?)

5. Troubles Prevention

There is multiple cases were we know that an action from the user will create troubles (most notably divergence) preventing the user to do so in the first place (in the same philosophy) as phase would be very useful,

• Having rebase skip changesets with successors in destination,
• Having a generic mechanism to run "validation" before any "editing" of a
  • changesets,
• requires a special flag or config to create divergence locally.

6. Low level Utilities

Currently, the "obsstore" is not really "fixable" having tool available to very advanced users would be good.

• "hg strip" should have a way to strip related obsolescence markers,
• "hg strip" should have a way to strip a whole obsolescence Lineage.
• Some debug command to strip arbitraty markers?

6.1. Stage C

At this "beta" state, the UI and experience will not be easy/pleasant enough for normal user, but advance user of Mercurial should find their mark. We may consider shipping it with Core Mercurial with an experimental flag.

Blocker to beta release:

• Obsmarkers exchange should be good enough to define a final storage format for markers.
• On disk format should be stable in the foreseeable future and fit performance//exchange need.
• Evolve should be able to solve (or provide a way to) all troubles that a user may encounter, especially divergence,
• Evolve should have predictable result (--rev options and co),
• Evolve should be abortable (wide transaction ?),

• Performance impact should be reasonable,

6.2. Release Stage

A which point we can merge changesets evolution into core.

• UI offering a Solution to the N² markers creation when editing history (TopicPlan),

• Commands set defined enough to be freezed for backward compatibility

• No race condition when exchanging with server (bundle2 + repo layout allowing atomic transaction),

• No impactful Performance Regression (including efficient exchange),
• Concrete plan to handle high volume of markers (archiving or something),

6.3. Related Concept

Other concept not directly involved in Changeset Evolution and closely related for technical or user experience reason.

• AtomicRepositoryLayoutPlan: To reduce race condition around new evolve

• metadata

• WideTransactionPlan: To help ability to have "pure" abort/rollback/undo,

• FeatureBranchesStruggle: To help organise all your developpements branches,

• TopicPlan: A possible solution to the above,

• RevsetOperatorPlan: To offer proper way to travel through the

• evolution-history,

• PlanHistoryRewritePlan: Factorise and empower all the common history rewriting operation,

• ExperimentalExtensionsPlan: To move the evolve extension into core,

• CommitCustodyConcept: Top level way to follow and sign draft changeset.

7. In progress Features

7.1. Using Obsolescence Marker during Rebase

There is two big issues with rebasing a set containing obsolescence changeset:

• It is easy to create divergence
• You get a lot of conflict when rebasing an obsolete stack on it's successors version.

To enable the current implementation set the config experimental.rebaseskipobsolete to true.

Current progress:

• rebase can skip obsolete changeset when rebased on successors,

• same logic as above, handling prune,

• same logic as above, handling split,

• ability to detect divergence creation and bail out,

• ability to rebase set with obsolescence inside the set (rebasing both precursors and successors) without creating divergence,

• official config to control these two behavior (either in one or two config)

• config on by default.

7.2. `hg evolve --list`

This is also tracked in https://bz.mercurial-scm.org/show_bug.cgi?id=4845

Evolution can compute a lot of information out of the obsolescence graph. It already use it to compute troubles and automatically solve them. Yet this information is not easily accessible to the user.

We would like to gain a basic version of hg evolve --list that would provide all available information on current troubles.

7.2.1. Output Proposal

• CHANGESET
  • - TROUBLE: REASON - TROUBLE: REASON
  CHANGESET
  • - TROUBLE: REASON

More concrete example:

5633576f2df1: unbundle: cleanly abort on unknown ...

• unstable: parent 485eb2a02a2b is obsolete bumped: precursors ec2662b9629d is public divergent: precursors 4a8e21578e14 have another succcessor 84dcc37b1272 divergent: precursors 4a8e21578e14 have another succcessor 2a3010ba6f52

We want to ensure this is implemented through the 'formatter' API so we gain structured output easily "eg -Tjson"

7.2.2. Command Line API

We probably want

• --rev to allow select revision to display

• --unstable/--bumped/divergent to restrict the trouble we display.

7.2.3. Details on Each Trouble Cases

Here is some idea for data we could provide in each cases. This provide an handy list of case in the process.

7.2.3.1. Unstable from Obsolete Parent

• reason: unstability comes from parent being obsolete
• Source: What is this obsolete parent,
• Evolution target: What the evolution will turn the parent into (beware of pruned changeset)
• We should mention problematic split,

7.2.3.2. Unstable from Obsolete Parent

• reason: unstability comes from parent being unstable
• Source: What is this unstable parent,
• parent needs to be evolved before that one.

7.2.3.3. Unstable Merge

• If both parent have obsolescence/troubles, we must issue two entries

7.2.3.4. Bumped Changeset

• Point at the public changeset
• Mention if we need a rebase or not
• ADVANCE: Is there a commit message change

7.2.3.5. Divergent Changeset

• Point at the other divergent changeset
• Point at the "greater common evolution ancestors" of divergence
• Says if this GCEA is known locally
• Point "greater locally known common evolution ancestors" in addition
  • otherwise.
• mention if we need a rebase or not,
• The "divergent with a split" case…

7.2.3.6. Divergent with the other side public

This is actually one of the most common divergence scenario. We probably need a dedicated name for this.

• We should mention that the other side is public/immutable.

7.2.3.7. Visible Obsolete Changeset

Should list them with a mention of why they are still visible?

7.3. obsmarker support in `hg strip`

We would like to be able to strip obsolescence marker related to a changeset when stripping it.

We want:

• (some) way to disable it ("nice way" can come later),
• the obsmarker to be in the backup bundle,
• A mode where we strip all markers between the stripped one and first known
  • precursors (default?)
• A mode where we strip all precursors too (and associated markers) (beware of
  • untargeted descendant of precursors, cf drophack).

The "obsmarker in the backup bundle" will requires some work because there is no other place storing obsmarkers in an on disk bundle. However, all infra should be in place as we already exchange obsmarker over the wire through a bundle.

In the same go, you can consider adding obsmarker support to hg bundle through the "variants" part of the type.

7.4. `hg evolve --continue/--abort/--drop`

Currently evolve implement a super basic state format that does not allow a proper "multi step" behavior.

7.4.1. About `--continue`

Current issues

• does not remember the currently evolved set (just finish the current one)

• does not work for --bumped or --divergent

We need to update the format on disk to allow these.

7.4.2. About `--abort`

This does not exists but it would be useful.

• We need to know the evolved set,
• We must also record and remove obsmarker we created in the process.

7.4.3. About `--drop`

Currently, user can "abandon" his evolve at any time. leaving half of the thing evolved and the other one still unstable. That is a pretty cool feature we want a way to keep. Adding a flag could be a way. openning the way to similar feature in other multi step command.

Name is not frozen, this is the first time we mention it.

Maybe we just want hg up -C to be that, but it seems a bit too easy to run by mistake.

8. Archived Topic

8.1. Obsstore Format

Markers are stored in an append-only file stored in '.hg/store/obsstore'.

8.1.1. V1 (current) Format

(see in line document for latest data)

8.1.1.1. quick summary

• <number-of-successors(=N)><metadata-lenght(=M)><bits-field><precursor>(<successor>*N)<metadata>

• B, I, B, 20s, (20s*N), s*M

8.1.1.2. longer explanation

The file starts with a version header:

• 1 unsigned byte: version number, starting at zero.

The header is followed by the markers. Each marker is made of:

• 1 unsigned byte: number of new changesets "N", can be zero.
• 1 unsigned 32-bits integer: metadata size "M" in bytes.
• 1 byte: a bit field. It is reserved for flags used in common
  • obsolete marker operations, to avoid repeated decoding of metadata entries.
• 20 bytes: obsoleted changeset identifier.
• N*20 bytes: new changesets identifiers.
• M bytes: metadata as a sequence of nul-terminated strings. Each
  • string contains a key and a value, separated by a colon ':', without additional encoding. Keys cannot contain '\0' or ':' and values cannot contain '\0'.

8.1.2. V2 (current) Format

8.1.2.1. motivation

There is two extra information we would like to see in a second version of the format:

• date: There is currently *always* a date in the meta data. So storing it explicitly would be more space efficient. It would also open the way to quickly access the date for sorting purpose (no use case yet but not crazy to think about it)
• parents: When a changesets is pruned (obsoleted, no successors) we needs to records its parents. This is necessary to link the markers chain to the push/pull operation it is relevant to.
• We may want to extend the bit field to 2 bytes. We currently use 1 and can see use case for 3-5 others (tracking the type of changes introduce by the rewriting (desc, patches, content, etc) so we are running short
• We may also want to explicitly store the username of the marker's creator are they will always be ones. however there is no need for quick access of such information so it could stay in the metadata field.

8.1.2.2. possible change

Date:

• The date will be a 64bits float (for seconds since epoch) followed by a 16 bits integer (time zone)
• It will make sense to put the date in front of the markers. that would give markers sorting some semantic.

Parents:

We have multiple option for storing parents:

1. Having an explicite field similar to successors (one byte to know how many parents, then parents)
2. Having an explicite field but store the number of parent in the bit fields (since we never have more than 2 parents)
3. Using the successors field. Having negative number of successors mean it is a prune.

Option (3) is the most space saving but prevent use to store parent information for more changesets if needed in the future (We do not have a final exchange plan yet).

Option (1) and (2) takes 2 to 8 bits more than (3) but are more flexible.

bit field

If we extend the bit field to 2 Bytes, it makes sense to use option (2) for storing parent.

8.1.2.3. Proposed Format

• <date><timezone><number-of-successors(=N)><metadata-lenght(=M)><bits-field+nb-parents(=P)><precursor>(<successor>*N)(<parents>*P)<metadata>

• d, h, B, I, H, 20s, (20s*N), (20s*P), s*M

The P number would be hidden in the bit field. We need to store 4 possible values here: 0 parents, 1 parent, 2 parents, ø parents information stored. Possible assignement is 00, 01, 10, 11. this let both 0 parent and ø parent info to be 0 module 3.