|
Size: 3718
Comment: Add commands which already works on Python 3.
|
Size: 4071
Comment: update test pass rate; add a couple of things to be investigated
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 5: | Line 5: |
| This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3. Our current aim is to support Python 3.5. | This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3. Our current aim is to support Python 3.5+. |
| Line 8: | Line 8: |
| `hg version`, `hg debuginstall`, `hg init`, `hg files`, `hg manifest`, `hg log`, `hg diff`, `hg export`, `hg status`, `hg summary`, `hg config`, `hg identify`, `hg update`, `hg commit`, `hg branches`, `hg bookmarks` works on Python 3 without using any out of core extensions. These won't work for you if you have out of core extensions enabled. There are certain things which don't work yet with these commands like revset, templates. | Most of the basic and daily usage commands work with out of tree extensions being disabled. More than 98% of test suite works with Python 3. The lists of passing tests can be found at [[https://www.mercurial-scm.org/repo/hg-committed/file/tip/contrib/python3-whitelist|python3-whitelist]] It is very likely that an alpha or beta Python 3 release will be there in first half of 2019. Out of tree hg-extensions still don't work. There would be several unicode issues. |
| Line 15: | Line 19: |
| 1. It adds b'' in front of string starting with "'" or '"' and not having any u'', r'' or b'' in front. | 1. It adds `b''` in front of string starting with `'` or `"` and not having any `u''`, `r''` or `b''` in front. |
| Line 18: | Line 22: |
| 4. Converts argument of *attr and encode, decode to unicodes by adding u''. | 4. Converts argument of *attr and encode, decode to unicodes by adding `u''`. |
| Line 22: | Line 26: |
| * Due to everything is unicodes by default in Python 3, and we need to rely on bytes, we have [[https://www.mercurial-scm.org/repo/hg/file/tip/mercurial/pycompat.py|pycompat.py]] which contains hacks related to various functions of `os` module on Python. | * We deal with bytes internally, we have [[https://www.mercurial-scm.org/repo/hg/file/tip/mercurial/pycompat.py|pycompat.py]] which contains hacks related to various functions of `os` module on Python. |
| Line 26: | Line 30: |
| * We are also adding r'' at some places to make it a raw string. * There are currently two tests which are based on Python 3 compatibility and few checks in our linter [[https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-check-code.t|test-check-code.t]] to make sure new patches include things from `pycompat.py`. 1. [[https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-check-py3-compat.t|test-check-py3-compat.t]] : This test was used initially to fix a lot of things, not very much helpful now. 2. [[https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-py3-commands.t|test-py3-commands.t]]: This test lists commands which actually works on Python 3. If you want an updated list anytime, the test is the best place to look for. |
* We are also adding `r''` at some places to make it a raw string. |
| Line 35: | Line 34: |
| == How to start == | == How to contribute == |
| Line 37: | Line 36: |
| * You can always set up a virtual environment and run Mercurial in it, but we have a easier way around. 1. Clone the [[https://www.mercurial-scm.org/repo/hg|main repo]] or [[https://www.mercurial-scm.org/repo/hg-committed/|committed-repo]] and say it's in folder name `hgrepo`. 2. Have Python 3.5 installed and say you have it in variable `python3.5`. 3. Run `hgrepo$ python3.5 hg version`. That must work, if not check that you should not have out of tree extensions enabled. 4. Now you can run any hg command this way and test if it's working or not. |
* Pick a failing tests, run `python3 ./run-tests.py <test-name>` and try to fix the exceptions raised. Pure-python tests are sometimes easier to port, but often need to be ported to use unittest first instead of our legacy testing system. The first step in migrating such tests to Python 3 involves [[https://www.mercurial-scm.org/repo/hg-committed/rev/11d128a14ec0|porting to unittest]], followed by any necessary followups to fix issues on Python 3. A list of tests that probably still need this work done can be obtained by running `comm -23 <(hg files 'set:tests/test*py - grep(unittest)' | sed 's$tests/$$') contrib/python3-whitelist`. |
| Line 44: | Line 41: |
== Things need to be investigated == * Windows encoding changes https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8 * Lazy importer performance overhead. Our custom importer on Python 2 always returns a stub module during ``import``. Python 3's does I/O to verify the module exists then returns a lazy module that is loaded when first accessed. In addition to behavior differences, the I/O may contribute sufficient performance overhead to constitute a problem. * A mechanism for extensions to advertise that they are Python 3 compatible. Nearly every extension will break in Python 3. We may want a mechanism that requires extensions to self-declare that they are Python 3 compatible - possibly via special syntax in their source code or the presence of a well-named variable. It might have to be at the source level because Python 3 would need to evaluate code in order to obtain the value of a module-level variable. |
Note:
This page is primarily intended for developers of Mercurial.
Python 3
This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3. Our current aim is to support Python 3.5+.
1. What Works
Most of the basic and daily usage commands work with out of tree extensions being disabled. More than 98% of test suite works with Python 3. The lists of passing tests can be found at python3-whitelist
It is very likely that an alpha or beta Python 3 release will be there in first half of 2019.
Out of tree hg-extensions still don't work. There would be several unicode issues.
2. Contributing
We will be happy to review patches and speed up the work related to Python 3. Before you start there are few things related to current porting and how things work currently. Most of our efforts are to make sure have Python 2 compatibility intact while making Python 3 run.
- We have a source transformer which does following things on Python 3.
It adds b'' in front of string starting with ' or " and not having any u'', r'' or b'' in front.
Adds this line from mercurial.pycompat import delattr, getattr, hasattr, setattr, xrange, open on every python file.
Converts every occurrence of iteritems to items on Python 3.
Converts argument of *attr and encode, decode to unicodes by adding u''.
The transformer currently works on mercurial/, hgext/ and hgext3rd/.
The transformer code lies here and you can also use transformer on your .py files by adding them in the transformer.
We deal with bytes internally, we have pycompat.py which contains hacks related to various functions of os module on Python.
We also have encoding.environ which helps us using a bytes version of os.environ on both Python 2 and 3.
We are also adding r'' at some places to make it a raw string.
- Encoding issues are generally uncovered by our tests (as everything was byte string on Python 2.)
3. How to contribute
Pick a failing tests, run python3 ./run-tests.py <test-name> and try to fix the exceptions raised.
Pure-python tests are sometimes easier to port, but often need to be ported to use unittest first instead of our legacy testing system. The first step in migrating such tests to Python 3 involves porting to unittest, followed by any necessary followups to fix issues on Python 3. A list of tests that probably still need this work done can be obtained by running comm -23 <(hg files 'set:tests/test*py - grep(unittest)' | sed 's$tests/$$') contrib/python3-whitelist.
The practice we follow now is run commands which are not yet fixed and try to fix the exceptions raised. So our current approach is exception based.
4. Things need to be investigated
- Windows encoding changes
https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8
Lazy importer performance overhead. Our custom importer on Python 2 always returns a stub module during import. Python 3's does I/O to verify the module exists then returns a lazy module that is loaded when first accessed. In addition to behavior differences, the I/O may contribute sufficient performance overhead to constitute a problem.
- A mechanism for extensions to advertise that they are Python 3 compatible. Nearly every extension will break in Python 3. We may want a mechanism that requires extensions to self-declare that they are Python 3 compatible - possibly via special syntax in their source code or the presence of a well-named variable. It might have to be at the source level because Python 3 would need to evaluate code in order to obtain the value of a module-level variable.