Mercurial sprint 5.6 (notes from https://mypads.framapad.org/mypads/?/mypads/group/octobus-public-5d3rw470w/pad/view/mercurial-sprint-gb2akv7pm)
Agenda: https://docs.google.com/spreadsheets/d/1-PhQvYRmUk2LRrpKZg32fktAsfBRN93WC1GKCnDK7YY/edit#gid=0
Day 1 (Thursday 05)
Attendees:
Anton Shestakov
Augie Fackler
Charles Chamberlain
Danny Hooper
Georges Racinet
Gregory Szorc
Jeff Sipek
Jörg Sonnenberger
Kyle Lippincott
Manuel Jacob
Martin von Zweigbergk
Mathias de Maré
Matt Harbison
Pierre-Yves David
Pulkit Goyal
Raphaël Gomès
Sushil Khanchi
Thomas Klausner (from NetBSD)
Valentin Gatien Baron
Yuya Nishihara
Dropping Python 2.7
Who gets hurt? / Who is using 2.7?
OpenVMS
TortoiseHg (Corporate Windows users)
Random bytes/str issues
Windows installers complicated
macOS DMG issues
Repository structure is a mess
CentOS 7
Action: change our RPMs to use Python 3
Windows
Non-ascii filenames are problematic
pip install pager doesn't work in msys
make install doesn't work
WSL is wonky
NTFS per-directory case sensitivity flag in WSL
Filesystem I/O performance issues
Kalithea dropped Python 2 support
hg-subversion
It is probably out of date; not ported to modern Mercurial
FreeBSD builds hg-subversion for Python 3. Unsure if it works.
hg-fastimport: Jörg going to look at the porting
Conclusions:
2.7 blockers are working TortoiseHg MSI and DMG installers
Come back to this later today to decide!
Other extensions that do not (fully) support python 3.x:
hglist (https://alastairs-place.net/projects/hglist/) (Does not work with current mercurial anyway)
Mozilla version-control-tools repo (https://hg.mozilla.org/hgcustom/version-control-tools/, https://mercurial.selenic.com/wiki/MozillaPushlogExtension)
hgnested (https://pypi.org/project/hgnested/) (Does not work with current mercurial anyway)
Testing Approach
https://www.mercurial-scm.org/wiki/IntegrationTestsPlan
Yuya: lots of tests don't care about UI; internals can be tested without invoking `hg`
Should we adopt pytest?
Python ecosystem is using pytest
Alphare: As long as we have a "code style" for automagic fixtures, I'm +100 for using pytest
Complexity of current tests both for performance and maintainability, revset tests used as example, consideration for the bad HTTP tests -> Jörg and Georges will look at that
Conclusion:
Georges to send patch series vendoring pytest and demonstrating use/integration with run-tests.py. Fort at least 1 test to demonstrate how it works.
Command Namespaces
https://www.mercurial-scm.org/wiki/5.6Sprint#More_command_.22namespace.22
debug*, perf* commands
We don't want users running debug* commands
Question: create a new namespace (name TBD - possibly "admin") where we consolidate some debug* and regular namespace commands to aggregate "repo administration" commands. Upgrade, rollback, verify, debugstrip, etc.
Proposal: Add - between namespace name and command
Additional namespaces possible (for example one for experimental commands)
Conclusion:
Change canonical commands to use - between words; automatically register non-hyphen aliases to existing commands
Also convert arguments to use -???
Some users might also prefer to use _
New namespaces are good idea
Use "maintenance-" for repo admin / maintenance commands?
IRC / Chat Futures
IRC is ancient. Should we ditch IRC?
We are losing potential contributors by using IRC.
Matrix and Zulip are interesting.
Replacement needs:
persistence (no need to always be online)
anonymous access or very very very easy registration process (e.g. using already existing 3rd-party account or credentials from something in our project infrastructure, like phab)
bridge to/from IRC
Conclusion: Let's play with Matrix with IRC gateway -- see https://app.element.io/#/room/#freenode_#mercurial:matrix.org
Website
Raphael has volunteered to modernize website; big problem is modernizing content
Tortoise domain expiring in 2021; unsure of future
The wiki is a graveyard; no review process; no commenting
Replace wiki with Sphinx-based documentation?
Day 2 (Friday 06)
Tags
Usability issue: hg clone -r tag doesn't get the tag, just the tagged changeset
Scaling issue: consider every head
Weird conflict resolutions using non-stable ordering, merge conflicts in .hgtags are annoying, and other bugs
Alternative implementations are used by some companies:
Jane Street uses a dedicated repo for tags
Nokia also has a separate approach
Using the opportunity that changing the hash function will break all old clients anyway and changing the tags system together with the hash transition
We could store the tag information inside the changeset
Implementing some sort of caching mechanism for side-data (extra?) will fix performance issues for new tags, but also topics, etc.
We need to take into account people not using evolve
Tag namespaces for monorepos (also maybe narrow/sparse information)?
Experimental VCS (Martin) called "JJ" (placeholder)
Written in Rust
Compatible with Git repos
Separated into two lib and cli crates
All filepaths are UTF-8 (Rust String)
Meant as a client for Git repos with revsets, evolution, templates, etc.
Split between ChangeID and Revision ID (follows the change across amends)
Commands:
describe: to change metadata
close: equivalent to commit?
diff: equivalent to diff
co/checkout: equivalent to checkout
prune: equivalent to prune?
rebase: equivalent to rebase
restore: equivalent to revert
merge: equivalent to merge
op log: remembers all operations you've done to the repo
--at-op flag to see the state of the repo at any given operation
op undo/op restore
working copy changes stay where they are, need to "rebase the working copy" to bring them along on some other revision, like hg update does by default
unresolved conflicts can be committed, and the resulting revision can be used further. Unresolved conflicts are materialized as conflict markers on checkout. Reordering commits can result in intermediate conflicts but jj knows the overall diff is the same as before and so the final rev has no conflicts
allows conflicts to resolved collaboratively, rebase doesn't have to store state
merges are represented as the diff from the merged parents, so you can rebase them and add/remove parents
op log is motivated to support replication-friendly repository format
Discussion about what could be included into mercurial
backwards compatibility would hold us back? but revlogs are not really stable, the command line interface and network protocol are the parts that are most stable
Treating commits like working copy
https://www.mercurial-scm.org/wiki/RevisionAsWDirPlan
Turned into a discussion about ambiguous --rev meaning
Idea: --into <rev> for describing revision to mutate on; --from <rev> for revision to retrieve from
List all existing commands that take --rev (and also others) to check if the new flags would apply to existing commands
hg status -r doesn't map to --rev
Use different commands for very different user-facing results instead of adding a (non-advanced) flag to an existing command. (like histedit/rebase -i)
Replication Friendly Storage
We aren't replication friendly today
We don't use fsync properly today so not even POSIX filesystem compatible today (e.g. NFS)
We need to formalize storage interfaces and have continuously tested alternative storage backend to keep it working to unblock viable alternative storage backends.
Day 3 (Saturday 07)
--auto-shelve flag
What happens on conflict? And conflicts with untracked files
Depends on the next discussion (phase-based shelve)
Phase-based shelve
Using this, we re-use commit manipulation features already in hg instead of having a "side-channel" for shelves
internal phases are 95% ready: missing the upgrade/downgrade repo (removing all internal except shelves, etc.)
Performance needs to be worked on and understood before making the switch
Removes the need for stripping the intermediate stages
State of Rust in hg
effort to have a rust executable, rhg, that should become eventually the start point, and which run python for the complex commands (which is most of them)
would include chg, so it's much friendlier to enable, and could even be enabled by default
we'd like to have status in pure rust, but where do we stop?
concern about rust as a hard dependency in BSD. Thoughts from previous sprints would be no hard dependency on rust, however some (non-mandatory) features/performance work could be rust only
Sometimes it's a bit more complicated, as the persistent-nodemap feature is faster than equivalent functionality in C+python, and while it's possible to use a python-only hg with persistent-nodemap, it's much slower than the equivalent with C+python.
Greg claims that people with big monorepos should be running on archs well supported by rust (objections about powerpc cpus). May not even need a pure python implementation for performance-only changes. People can clone a repo to remove a rust requirement on a dirstate, for instance. Wouldn't work with nfs repos.
The network protocol must not require rust, but the local disk format can, and probably should, as it's better use of resources.
Jörg concerned about significant performance degradation for people not using rust
We don't really want much new C code at this code, especially one that's not fuzzed
augie makes the following claim "if we assume we ship pyoxidized binaries on mercurial-scm.org, a format that's used by default by that executable should have a python implementation", not clear if people agree.
To address the conflicting requirements of: "1. hg repos can be copied around and work, 2. work with reasonable performance if you don't have rust, 3. be fast when you have rust", one option that's suggested would be that hg init may add requirement specific to the hg version you're running (so persistent-nodemap for instance if you're running rust), but there would be `hg init --universal` or something to prevent this installation-specific requirements
Greg wants the default experience to be the fastest that the client can provide, because default matters etc
Conclusions:
Mercurial will not require Rust on any platform
Rust is an optional requirement; Mercurial features can be implemented in Rust; But end-user functionality (e.g. workflow hg commands) will not require Rust.
The Mercurial experience on large repos without Rust may not be great: this is due to the limits of performance with Python
The official Mercurial distributions will ship with Rust code / features on platforms where Rust is supported
hg init produces the best possible experience (read: set of enabled requirements) for whatever the running hg client supports
If there is a Rust feature supported that provides benefit, it is enabled by default
There is implicit BC here where we no longer guarantee that an hg X.Y can read a repo on machine B that was created on machine A
If a Mercurial install supports N>1 implementations of given functionality (e.g. Rust versus Python dirstate) and 1 implementation is slower than the other, it should use the faster one by default and possible emit a warning and/or require a config option to opt in to the known slower implementation.
We don't have a blanket policy of whether a non-Rust implementation of a feature is required
We don't require a 2nd implementation at initial implementation time.
At feature enable-by-default time, we have a formal review of whether a 2nd implementation should be required.
Having the ability to downgrade the repository format is greatly preferred.
Releases
mostly automated (build, sign the release, new version on bugzilla, ...), although the release notes are not. Augie doesn't have time for the latter
maybe we should write to the release-notes files in the hg repo more. May be able to make phabricator suggest editing the file if it's not already modified
unclear if api changes notes are useful. One could imagine a link to a revset on the hg repo that would show the commit with "(api)" in the description instead.
Mercurial layered architecture
Gradually moving towards a "core" and "consumer" split: the core should never output anything user-facing, and be responsible for most of the logic
discussion about ripgrep support (meaning it knowing about hgignore), since code for that is implemented in the rust code, which burntsushi should accept
In general splitting building bricks into crates (for the Rust case) like graph traversal, graph splitting, ignore support, etc.
Rust-cpython
we use rust-cpython, but py03 seems to be the most popular option now, so we may want to switch over at some point
the split between hg-core and hg-cpython means the mercurial rust is ready for this kind of switch
we should check at some point with mark thomas what the concerns are about memory-safety in py03
pyoxidizer also uses rust-cpython, but greg wants to switch at some point. He had to patch rust-cpython to remove some assumptions that didn't work in the case of embedding the runtime and things into a single executable. May have to do the same thing with py03.
Should some extensions be on by default
rebase: yes, it was discussed in a previous sprint but hasn't happened. Didn't follow the bit about various levels of obsolescence markers or something
share: yes, with pulkit's safe-share
clean: if we can make it not wipe by default, so perhaps interactive by default (but not interactive if the purge extension is loaded, for compatibility)
strip: don't remember what we said here, but I think the previous sprint said we should have hg debugstrip
Dirstate format mentioned in the rust discussion
it's a tree format ( so we have sorted order). It contains full paths for files, at the leaves. Writes can edit the tree without rewriting the whole file (by appending edits, similar to persistent-nodemap), but eventually the file gets rewritten from scratch when too much edits have accumulated.
went into a lot discussion about how the new status works, how some parts will require mtime for directories that get updated at the right time with high precision (which are going to be gated on some runtime check, under the assumption that a single checkout is not spread across multiple filesystems). What kind of paths are stored in the file (the same bytes as in the manifest, I believe).
Day 4 (Sunday 08)
Combined revlog
The general idea is, with a not too big amount of work, to add a new field with a hash of the filename to reduce the number of filelogs
Optimizes copying a file if both are in the same revlog
Numbers: NetBSD repo (4.1GB for store) (I missed the part for filelogs...)
We need an efficient index (so, the current persistent nodemap) if we want to combine revlogs
Greg: we need strong interfaces for storage internals, a second format that's tested in storage, at which point it should be much easier to change repo formats
We need to take into account strip performance
Take alignment of revlog entries into account (current are exactly 64 bytes long, which happens to be a line of cache for modern processors)
Maybe separate strip into two commands, one that just drops the index, the other does the actual "lose my data" (essentially something like "prune" and "gc")
Using MMap more
Reduces memory pressure by sharing kernel pages
Has some speed drawbacks in rare cases, most of the time it's a big win for large repos
It helps low-RAM users because less data is copied twice
The main issue is that memory access can return a SIGBUS if the file has been truncated, so just don't truncate unless you're doing a vacuum or you really really need to. Indirectly you can also have a random prefix for the data, keep that in the docket file and write to a separate file, then rename if you need to truncate
Alternatively: signal handler for SIGBUS ==> They're very hard to get right in Python. Any reading to a truncated file would return zeros only. But it's probably very hacky
We need an abstraction for using/not using mmap
Performance
Looking back at recent work:
Clone needs 40% less memory (RSS) now
Sparse-revlog improves performance in highly merged history
Persistent nodemap is a lot faster (if you use Rust). On one particulary large repo latency dropped from ~400ms per side for each pull/push in discovery code to basically < 10ms.
Copytracing is getting faster, hasn't yet landed. The Python version works fine and is reasonably efficient, the Rust implementation is much faster and might be necessary for very large repos. Numbers are fuzzy, but multiple cases that were in the multiple seconds range (20s+) are down to <1s now.
Status has gotten about twice as fast with Rust in the general case. In pure Rust (rhg), the current POC code is much faster (25x times with a 8c/16t computer in Mozilla-Central)
chg: performance difference (C is less resource-intensive for now), and aside for fixing small existing bugs, it should work
Stable-range work can help with many different performance issues
Some French researchers are interested in general fixing graph issues
Issues:
Evolution problem of building the stable-range caches
Clone performance: a lot of time is spent on metadata (changelog/manifest) and seems too slow
In general commands are slower than they should be because of a threshold effect
Having many heads makes multiple things slow: discovery, branchmap, ??
Better datastructures for evolve/topic are needed, but this is not worrying, just something that needs to be done
Valentin: the main issues are things that are in the works (files, cat, status...), but some things are in consideration like:
using transaction numbers for pull instead of discovery
performance with narrow for the server-side
hg update is parallel but slow, slower than single threaded hg cat of the whole repository in rust. So maybe some optimizations could be done with writing files and the parallel work is bottlenecked by Python
hg bundle is extremely slow
Packaging TortoiseHg
Packaging for Windows is broken, restart from scratch
Host dependency components on m-s.o, instead of random developer repos? e.g. merge modules, Qt/PyQt/sip/Qscintilla source tarballs, etc
Mac packaging is a bash script
TortoiseHg is a blocker for Python 3 migration (as previously said). If we merge the two build repos into the main thg repo, Greg will do the work.
Heptapod CI could gain Windows and Mac runners in the next few months
Signing packages is important, the Mac story is more complicated, because you need a Mac computer to sign it
tortoisehg.org is still not in the hands of the Mercurial project and will expire in 2021
add sitecustomize.py or equivalent to core hg binaries like thg packages have, so extensions are a simple `foo =` for all binaries on all platforms when using `pip install --user foo`
Need a mechanism to install packages in this manner with PyOxidizer. Ship a standalone python.exe?
Dirstate cache vs source data
General discussion about whether the souce information in the dirstate (parent revisinos, hg add, hg remove, hg rename, hg forget) can be separated from the cache part of the dirstate (things like mtime)
Such a file would be tiny the vast majority of the time, just 40 bytes
It would apparently avoid repository corruptions due to google repos being edited on multiple machines
Virtual sprints in general
the next couple of sprints will probably be virtual as well
general discussion about people talking over people. Maybe we should have a convention for people to signal they want to speak
some people like virtual sprints because they are easier to attend
not every discussion should be done at the sprint, people can also talk between themselves in between sprints
post-sprint note by jeffpc: it looks like Zoom has breakout rooms that the host can enable - I just did a quick test and it seems to work quite well