Type checking is an important part of making Mercurial compatible with Python 3. This page describes the workflow, recommendations and gotchas.
Setup
Mercurial source code files have to remain compatible with Python 2 for the foreseeable future. This is why we are storing type annotations in external .pyi files (called "stubs"). We then reapply them using retype for the purpose of type checking using mypy.
Types are stored in a root directory called, well, types. The output of the retype command lands in an .hgignore'd directory called typed-src.
Type hints
Never used type hinting with Python before? Start here: http://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html
You will need Python 3.6. Download mypy. If you're a flake8 user, upgrade to the latest version and install the flake8-pyi plugin.
Creating a new .pyi file
The simplest way to do it is to start with the stubgen script available in the mypy GitHub repo. It is crude but gets the job done. It recreates the basic structure of the given source file in the form of a .pyi file. Note that this .pyi file is missing most annotations, and the ones that it **does** add are likely wrong (in most cases defaulting to the Any type). Your job is to fill the blanks in with actual typing information.
Our .pyi files describe Python 3.5's type system and standard library. Not Python 2.7 because the purpose of this exercise is to find all the places in which the code currently doesn't play well with Python 3. Not Python 3.6 because the initial version of Python 3 that Mercurial is targetting is 3.5. That being said, syntax-wise, you are recommended to use variable annotations in .pyi files since they are cleaner and type checkers and retype use them anyway.
Speaking of .pyi files, you might have noticed that they are syntactically valid Python, with the single difference that they natively support forward references (in other words, you can specify definitions out of order without consequence). Stylistically, .pyi files are not meant to follow PEP 8. Instead, they conform to the typeshed coding style.
Mercurial ships with mercurial.pycompat, a library providing a number of compatibility bridges between Python 2 and Python 3. While this library is imported implicitly at runtime (via a magic import hook), we don't do that for the purposes of type checking. So, if needed, add the following import in your .pyi files:
from mercurial.pycompat import delattr, getattr, hasattr, setattr, xrange, open
retype
NOTE: This tool is very new. If you hit any issues, open an issue on GitHub or contact LukaszLanga directly.
From the root of the hg repo run:
$ retype --hg --quiet mercurial
A retype run does the following:
- for each source file ...
find the corresponding .pyi file in ./types and ...
reapply typing information from .pyi to the source, writing the result to typed-src/;
if retype is executed with --hg, also translate all "native" string literals into bytes
Re-application of types requires copying typing imports and alias definitions from the .pyi file, which is why line numbers will no longer match original source code. It retains its original formatting though.
Some design principles:
- it's okay for a given .pyi file to be incomplete (gradual typing, baby!), this will only generate warnings
- it's okay for functions and classes to be out of order in .pyi files and the source
it's an error for a function or class to be missing in the source
it's an error for a function's signature to be incompatible between the .pyi file and the source
it's an error for an annotation in the source to be incompatible with the .pyi file
Type checking
This is what you came here for, right? Once sources are translated in typed-src/, run:
$ cd typed-src/ $ mypy .