EOL Extension

This is considered a feature of last resort.

Automatic management of EOL conversion.

1. Status

This extension has been distributed with Mercurial as of version 1.5.4.

Author: Martin Geisler martin@geisler.net

2. Overview

Different platforms have different conventions for representing line endings in text files:

• Windows traditionally uses CRLF (\r\n, carriage-return followed by line-feed). The default text editor on Windows, Notepad, only understands CRLF. Command line tools and redirection also use CRLF.

• Unix and Linux traditionally uses LF (\n), though many tools can handle CRLF as well.

• Older versions of Mac OS used CR (\r), but Mac OS X is Unix and uses LF. This extension does not support the old CR format.

As a result of these differences, when people using different operating systems are working on the same repository, there is a real risk that someone's text editor will change the line endings in the files that they edit. If they commit these line ending changes, then unfortunately the diff for their commit will be trashed, and hg annotate/blame for the touched files will be useless.

The eol extension solves this problem by allowing a native (canonical) line ending type to be specified for the repository, and by converting seamlessly between the repository's and the user's native types as the user checks out and commits changes to select files. In this way users get to work with files using their native line ending type while the conversion process makes sure the wrong line endings aren't committed to those files. The extension also allows certain files to have a fixed line ending type both internally in the repository and when checked out (useful for Makefiles/.vcproj files, for example), and for others to be treated as binary files.

3. Per-user configuration

Each user needs to enable the eol extension on their system. If the repository uses the eol extension and some users do not have the extension enabled, then those users will have problems - see Problem 2 in the Troubleshooting section below. (And if shared repositories don't guard against bad line endings being pushed, then those problems will likely be shared with other users - see the hooks section for setting up push guards.)

3.1. Enabling and configuring the eol extension

To enable the eol extension add the following lines to an appropriate configuration file:

[extensions]
eol =

You should probably add this to $HOME/.hgrc (Mac/Linux) or %USERPROFILE%\.hgrc (Windows) to enable the extension for all repositories with a .hgeol file at their root that your user works with, but there are other Mercurial configuration files you can use. (Since only repositories with a .hgeol file will be affected, and if you have any such repositories then the eol extension really should be enabled for them, then you should know what you're doing if you choose a different file.)

The eol extension can be further configured per-user by adding a [eol] section like:

[eol]
native = CRLF
only-consistent = False

This section only accepts these two values. The native value can be assigned either LF or CRLF, and tells the eol extension to use this line ending type as the user's native type instead of their actual operating system's type (probably you should not set this). The only-consistent value tells the eol extension whether it should only do line ending conversion for files with consistent line endings (that is, files containing \r\n OR \n line endings, but not both). The default value is True, on the assumption that the user knows what he or she is doing if a file has mixed line endings.

3.2. IMPORTANT - after enabling the eol extension

You can skip this section if you do not have any preexisting repositories on your system (with a working directory checked out) that will have been affected by enabling the eol extension. In other words, if you do not yet have any repositories that have a .hgeol file at their root and that are affected by the configuration file that you enabled the eol extension in.

If you do have any affected repositories, there shouldn't be many of them. (Since everyone really should be enabling the eol extension as soon as they get their first repository containing a .hgeol file.) However, for the affected repositories that you do have, it is very important that you remove then re-check out ALL the version controlled files in their working directories. For each affected repository, do the following...

If there are local changes in the repository, save the changes (to a patch file, or whatever), then use hg revert . to remove the local changes from the working directory. Now remove all version controlled files from your working copy using hg co -C null (do not use hg remove for this, that is not what you want). Next check everything out afresh using hg update -C tip (replacing "tip" with a different revision if appropriate). The files in your working directory will now have the "correct" line endings. Finally, if you previously saved any local changes, reapply these now.

If you don't carry out these steps, then you have a real risk of encountering Problem 1 detailed in the Troubleshooting section below, possibly at some later date when you've forgotten about enabling the eol extension.

So what's actually going on here? The issue is that the eol extension only converts the line endings of working directory files to the user's native type when (a different) revision of those files is checked out into the working directory. If you have an affected repository with a working directory that was checked out prior to enabling the eol extension, then the files in that working directory will have the same line endings that the files are stored with internally. If this happens to be different to the line endings the files would be checked out with after enabling the eol extension, then Mercurial is going to be unhappy and you are going to get confused (problem 1, below).

4. Configuring a repository using the .hgeol file

The line ending type that the eol extension gives files as they are checked in/out of the repository is configured using a version controlled file called .hgeol at the root of the repository (version controlled so everyone uses the same rules!). If this file is missing, the extension will not convert any files. The .hgeol file can contain two configuration sections as detailed below, but before we get to that first read this...

4.1. IMPORTANT - before adding/changing a .hgeol file

So as to avoid confusion during the checks that you should make AFTER adding/changing a .hgeol file (see below), it is important that you remove any local changes from the working directory of your repository BEFORE adding or changing the .hgeol file.

4.2. The [repository] section

The .hgeol file can be given an optional [repository] section to set the repository's native line ending type (its canonical type). The default type is LF, but if you want to change the type to CRLF (the only other supported type) then add the following to your repository's .hgeol:

[repository]
native = CRLF

4.3. The [patterns] section

The meat of the .hgeol file is in the [patterns] section. This section contains patterns that tell the extension what line ending type should be used internally in the repository vs in the working directory for files that match a given pattern. There are four supported types for patterns. Those types are:

• native - use the repository's native type internally, and use the user's native type (their operating system's native type, or the type they configure for themselves) in the working directory

• LF - use LF both internally and in the working directory, regardless of the user's native type

• CRLF - use CRLF both internally and in the working directory, regardless of the user's native type

• BIN - do not convert line endings (the eol extension will never touch files containing NUL bytes (\0) so creating rules of this type is not usually necessary for genuinely binary file types)

Note that BIN (do nothing to line endings) is Mercurial's default behaviour; it is only needed if you need to override a later, more general pattern.

An example of a [patterns] section is:

[patterns]
** = native

This example will translate all text files to native EOL, or more precisely, all files not containing NUL characters will use native EOL.

Another example of a [patterns] section is:

[patterns]
**.py = native
**.txt = native
**.vcproj = CRLF
Makefile = LF
**.jpg = BIN

This example specifies the following rules:

• all Python files (**.py, a recursive glob pattern) and .txt files (**.txt) should be checked out to the working copy using operating system native EOLs (\r\n on Windows, \n on Mac OS X and Linux). Internally the repository will store these files using LF by default, or the type specified in the [repository] section of the .hgeol file (see above).

• all Visual Studio project files (**.vcproj) should be stored internally and checked out in Windows format. This rule applies equally to all operating systems.

• the root Makefile should be stored internally and checked out in Unix format. Again, this rule applies equally to all operating systems.

• all JPEG images (**.jpg) should be treated as binary and left alone. The eol extension will never touch files containing NUL bytes (\0) so this rule is not strictly necessary for JPEG files, but you may sometimes find it necessary for other file types.

Note that the first matching pattern is used, so put more specific patterns first.

For more information on pattern formats see hg help patterns or the File Name Patterns section of the man hg man page.

4.4. IMPORTANT - after adding/changing a .hgeol file

After adding or changing a .hgeol file it is very important that you check whether Mercurial reports line ending changes to files that you didn't touch, and that you do this BEFORE pushing the .hgeol changes out to other users. If you don't resolve any such changes before pushing the new/changed .hgeol file out to other users, then those users are going to see the same unexpected line ending changes as uncommitted changes in their working directories, which is really going to confuse and annoy them.

Checking for any such changes is as simple as running hg status or hg diff. However, MAKE SURE THAT THE EOL EXTENSION IS ENABLED for you first (see the per-user configuration section) or you won't be able to see any such changes, whereas users that DO have the extension enabled will see them after you push (assuming any exist)! If no changed files are reported, then you're good to go.

If Mercurial does report unexpected line ending changes, you may want to understand why; this happens if the repository is storing preexisting files internally with line endings that the new/changed .hgeol file now says are the wrong internal line ending type. When working copy files are compared to their internal copy the eol extension converts the line endings to the (new) internal type first, and if the new and old internal line endings are different, then as far as Mercurial is concerned the files have changed, since, well, they have.

There are two options to clear up such line ending changes; you can either reconsider the changes you made to the .hgeol file, or else you can commit the line ending changes that Mercurial is reporting with a message along the lines of "convert line endings to new canonical type". If you do decide to commit such line ending changes then be sure to push those changes at the same time as you push your .hgeol changes!

5. Troubleshooting

Problem 1: You get the error message 'abort: local changes found' or 'local changed ... which remote deleted' but hg diff does NOT show any changes. (hg status should still show which files Mercurial considers to have been modified though.) You can get these messages if a file matching a 'native' pattern in the .hgeol file contains line endings that are not your operating system's native type. This is likely to happen if you failed to carry out the IMPORTANT step above after enabling the eol extension. To recover, if you have any local changes, first save them to an mq patch using hg qnew tmp; hg qpop. Now do something like hg co -C null; hg co tip. This will update all tracked files in your working directory to have the correct line endings. If you had any local changes, then finally reinstate them using hg qpush; hg qref -X .; hg qpop -f; hg qdel tmp.

Problem 2: You get the error 'abort: local changes found' and hg diff DOES show line ending changes, but you didn't touch the affected files. This will happen if the repository's INTERNAL copy of the file(s) shown in the diff has the WRONG (non-canonical) line endings according to the .hgeol file. This can happen if you/someone else did not follow the IMPORTANT section above that should be carried out after adding/changing the .hgeol file. It can also happen when someone who hasn't turned on the eol extension commits the wrong line endings to the repository. See the IMPORTANT section mentioned above to understand the issue and your options.

Problem 3: You get the error 'abort: inconsistent newline style in ...' when trying to commit: either you need to make the newlines in the file consistent yourself before committing, or else you should add the 'only-consistent = False' detailed above to your hgrc to have the eol extension do it for you.

6. Hooks

See hg help eol.

7. Migrating from win32text to eol

In earlier versions of Mercurial, the conversion of line endings was done with the Win32TextExtension. Please see the page MigratingFromWin32TextToEol for a guide for migrating from win32text to eol.

CategoryBundledExtension CategoryHowTo