Largefiles extension

/!\ As of Mercurial 2.0, this extension still has a number of rough edges

Large binary files tend to be not very compressible, not very "diffable", and not at all mergeable. Such files are not handled well by Mercurial's storage format (Revlog), which is based on compressed binary deltas. largefiles solves this problem by adding a centralized client-server layer on top of Mercurial: largefiles live in a central store out on the network somewhere, and you only fetch the ones that you need when you need them.

Contents

  1. Status
  2. Overview
  3. Configuration
  4. Design
    1. The largefile store
    2. The local repository
    3. The user cache
    4. Implementation details
  5. See also

1. Status

This extension is distributed with Mercurial 2.0 and later.

Author: Various

2. Overview

The largefiles extension allows for tracking large, incompressible binary files in Mercurial without requiring excessive bandwidth for clones and pulls. Files added as largefiles are not tracked directly by Mercurial; rather, their revisions are identified by a checksum, and Mercurial tracks these checksums. This way, when you clone a repository or pull in changesets, the large files in older revisions of the repository are not needed, and only the ones needed to update to the current version are downloaded. This saves both disk space and bandwidth.

If you are starting a new repository or adding new large binary files, using largefiles for them is as easy as adding '--large' to your hg add command. For example: 

$ dd if=/dev/urandom of=thisfileislarge count=2000
$ hg add --large thisfileislarge
$ hg commit -m 'add thisfileislarge, which is large, as a largefile'

When you push a changeset that affects largefiles to a remote repository, its largefile revisions will be uploaded along with it. Note that the remote Mercurial must also have the largefiles extension enabled for this to work.

When you pull a changeset that affects largefiles from a remote repository, nothing different from Mercurial's normal behavior happens. However, when you update to such a revision, any largefiles needed by that revision are downloaded and cached if they have never been downloaded before. This means that network access is required to update to revision you have not yet updated to.

If you already have large files tracked by Mercurial without the largefiles extension, you will need to convert your repository in order to benefit from largefiles. This is done with the 'hg lfconvert' command: 

$ hg lfconvert --size 10 oldrepo newrepo

By default, in repositories that already have largefiles in them, any new file over 10MB will automatically be added as largefiles. To change this threshhold, set largefiles.size in your Mercurial config file to the minimum size in megabytes to track as a largefile: 

[largefiles]
minsize = 2

or use the --lfsize option to the add command (also in megabytes): 

$ hg add --lfsize 2

The largefiles.patterns config option allows you to specify specific space-separated filename patterns (in shell glob syntax) that should always be tracked as largefiles: 

[largefiles]
patterns = *.jpg *.{png,bmp} library.zip content/audio/*

Note: the patterns syntax shown here is probably incorrect, please try hg help patterns to see if it fits better, in particular *.{png,bmp} seems not to work, whereas re:.*\.(png|bmp) get things done as expected.

3. Configuration

Configure your config file to enable the extension by adding following lines: 

[extensions]
largefiles =

4. Design

This section explains how largefiles works behind the scenes. If you're just adding/modifying/committing/pushing/pulling in a largefiles repo, you shouldn't have to read this section (although it can't hurt). But if you are setting up or administering Mercurial with largefiles, this is essential reading.

4.1. The largefile store

A largefiles store, typically on a centralized server, has every past revision of every largefile. Each largefile revision is identified by its SHA-1 hash, and all interactions with the store take one of the following forms.

largefiles stores can take one of two forms:

4.2. The local repository

The local repository has a largefile store in '.hg/largefiles' which holds a subset of the largefiles needed. When largefiles are downloaded from the central store, a copy is saved in this store.

4.3. The user cache

largefiles in a local repository store are hardlinked to files in the user cache. Before a file is downloaded we check if it is in the global cache, hard-linking to the local store if we find it.

4.4. Implementation details

Each largefile has a standin file in .hglf/, which is tracked by Mercurial like any other file. The standin contains the SHA-1 hash of the largefile contents. When a largefile is added/removed/copied/renamed/etc the same operation is applied to the standin. Thus the history of the standin is the history of the largefile.

For performance reasons, the contents of a standin are only updated before a commit. Standins are added/removed/copied/renamed from add/remove/copy/rename Mercurial commands but their contents will not be updated. The contents of a standin will always be the hash of the largefile as of the last commit. To support some commands (revert) some standins are temporarily updated, but changed back after the command is finished.

A Mercurial dirstate object tracks the state of the largefiles. The dirstate uses the last modified time and current size to detect if a file has changed without reading the entire contents of the file. (Unfortunately, the use of dirstate limits largefiles to 2 GB. This will hopefully be fixed after Mercurial 2.0.)

5. See also

There are a number of older extensions for managing large files. This extension is a descendant of the BfilesExtension and is now the recommended way to handle such files. Alternatives are BigfilesExtension and SnapExtension.

CategoryBundledExtension