Differences between revisions 2 and 43 (spanning 41 versions)
Revision 2 as of 2014-02-03 21:27:55
Size: 1137
Editor: AugieFackler
Comment:
Revision 43 as of 2019-07-13 07:00:41
Size: 6162
Comment: the source transformer no longer runs on hgext3rd
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#pragma section-numbers 2
Line 3: Line 5:
This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3. Nobody is actively working on this - AugieFackler works on it sporadically, and would be happy to see patches on this topic flagged with Py3 on the mailing list. The work in progress is visible at http://hg.durin42.com/hg-py3k/ - note that ChangesetEvolution is in use on that repository, and so hashes will change over time as the patchset is rebased. = Python 3 =
Line 5: Line 7:
The most significant problem at the moment is some lingering cyclic imports in the codebase: This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3.
Line 7: Line 9:
==== cmdutil -> subrepo -> cmdutil ==== <<TableOfContents>>
Line 9: Line 11:
The easiest fix for this would be to move hgsubrepo to a new file, and then fix the registration mechanism to be dependency injected somehow. == Status ==
Line 11: Line 13:
==== mercurial.repoview -> mercurial.revset -> mercurial.repoview ==== Mercurial 5.0 is the first release that officially has beta support for Python 3. Supported Python 3 versions are 3.5, 3.6, and 3.7. Python 3.8 mostly works, but there are a few known incompatibilities. Mercurial with Python 3 on Windows is not yet widely tested and there are more known issues on Windows compared to Linux, macOS, and other UNIX-like platforms.
Line 13: Line 15:
It is the project policy for Mercurial and its core extensions to be compatible with Python 3. Over 99% of tests pass with Python 3 and test regressions are treated seriously.
Line 14: Line 17:
==== mercurial.fileset -> mercurial.merge -> mercurial.subrepo -> mercurial.match -> mercurial.fileset ====
If hgsubrepo moved out of subrepo, this would also be resolve.d
Many 3rd party extensions have not yet been ported to work with Python 3.
Line 17: Line 19:
== Using ==
Line 18: Line 21:
==== mercurial.filemerge -> mercurial.match -> mercurial.fileset -> mercurial.merge -> mercurial.filemerge ==== Mercurial's `setup.py` file refuses to run with Python 3 by default. This means that `pip install Mercurial` or `python setup.py install` will not work with Python 3 by default.

Setting the `HGPYTHON3` environment variable will suppress this error and allow execution with Python 3. e.g. `HGPYTHON3=1 pip3.7 install Mercurial` or `HGPYTHON3=1 python3.7 setup.py install`.

No run-time environment variable or config option is required to use Python 3 with Mercurial: only the installation step / `setup.py` requires special action to override the Python version check.

== Things need to be investigated ==

 * Windows encoding changes
 https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8
 * Lazy importer performance overhead. Our custom importer on Python 2 always returns a stub module during ``import``. Python 3's does I/O to verify the module exists then returns a lazy module that is loaded when first accessed. In addition to behavior differences, the I/O may contribute sufficient performance overhead to constitute a problem.
 * A mechanism for extensions to advertise that they are Python 3 compatible. Nearly every extension will break in Python 3. We may want a mechanism that requires extensions to self-declare that they are Python 3 compatible - possibly via special syntax in their source code or the presence of a well-named variable. It might have to be at the source level because Python 3 would need to evaluate code in order to obtain the value of a module-level variable.

== Beta bugs ==

Following are things which don't work right now:

  * ~1% of tests fail
  * phabricator extension
  * Many 3rd party extensions
  * [[https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8|Windows filesystem encoding]]

If you find anything apart from this not working, definitely go ahead and edit this page and we will fix it.

== Porting Extensions to Python 3 ==

Nearly every extension will need to be ported to be compatible with Python 3. This is because of fundamental differences between Python 2 and Python 3.

The source code for Mercurial extensions will need to be Python 3 native and will need to be compatible with Mercurial's APIs. In many cases, existing source code will compile on Python 3 but will fail at run-time. Sources of run-time errors include:

 * Use of `str` instead of `bytes`. Mercurial uses `bytes` (`b''` strings) in almost all of its APIs and data structures. This is in contrast to much Python code, which uses `str` and `''` strings. It is common for extensions to `b''` prefix most strings in order to remain compatible with Mercurial.
 * Use of `iteritems()`, `iterkeys()`, etc. These methods from core data structures do not exist in Python 3.
 * Import of renamed modules. Python 3 refactored the locations of various modules in the Python standard library. Extensions may need to take this into account.

Do an Internet search for ''Python 3 porting'' to find well-written and comprehensive guides on generically porting code to Python 3.

Extension authors may find the [[https://www.mercurial-scm.org/repo/hg-committed/file/tip/mercurial/pycompat.py|mercurial.pycompat]] module useful. This modules contains abstractions and utilities for bridging the differences between Python 2 and 3. It is conceptually similar to the `six` Python module.

If you need help or guidance on porting extensions, you can message on IRC or the development MailingList. We will be happy to help you.

== Source Rewriting Module Importer ==

As of at least the Mercurial 5.0 release, Mercurial uses a custom module importer on Python 3 which rewrites source code dynamically as part of importing modules. This module importer is only active for the `mercurial` and `hgext` packages. This means that Mercurial's own source code and extensions are not yet native Python 3 source code. So if you look at Mercurial's source code for ideas on how to do something in an extension, behavior in the extension may differ from Mercurial itself due to the presence of this custom module importer.

In the 5.0 release, the custom module importer performs the following actions:

 * Automatically adds `b''` prefixes to strings, making all `''` literals `b''` and effectively changing `str` to `bytes` everywhere. i.e. behavior mostly matches Python 2.
 * Modules automatically have `from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode` added.
 * `getattr()`, `setattr()`, `hasattr()`, `safehasattr()`, `encode()`, and `decode()` functions and methods have string literals in arguments rewritten to the appropriate type because Python requires a `str` value instead of `bytes`. (This effectively selectively undoes the global `''` to `b''` source transformation.)
 * `iteritems()` and `itervalues()` are automatically rewritten to `items()` and `values()`, respectively.

The source rewriting module importer is intended to be a stop-gap to make porting Mercurial to Python 3 simpler and will be removed in a future release. This is why extensions do not use it.

More history about this importer is available at https://gregoryszorc.com/blog/2017/03/13/from-__past__-import-bytes_literals/

----
CategoryAudit

Note:

This page is primarily intended for developers of Mercurial.

Python 3

This is a status page for keeping track of what needs to be done to make progress on Mercurial on Python 3.

1. Status

Mercurial 5.0 is the first release that officially has beta support for Python 3. Supported Python 3 versions are 3.5, 3.6, and 3.7. Python 3.8 mostly works, but there are a few known incompatibilities. Mercurial with Python 3 on Windows is not yet widely tested and there are more known issues on Windows compared to Linux, macOS, and other UNIX-like platforms.

It is the project policy for Mercurial and its core extensions to be compatible with Python 3. Over 99% of tests pass with Python 3 and test regressions are treated seriously.

Many 3rd party extensions have not yet been ported to work with Python 3.

2. Using

Mercurial's setup.py file refuses to run with Python 3 by default. This means that pip install Mercurial or python setup.py install will not work with Python 3 by default.

Setting the HGPYTHON3 environment variable will suppress this error and allow execution with Python 3. e.g. HGPYTHON3=1 pip3.7 install Mercurial or HGPYTHON3=1 python3.7 setup.py install.

No run-time environment variable or config option is required to use Python 3 with Mercurial: only the installation step / setup.py requires special action to override the Python version check.

3. Things need to be investigated

  • Windows encoding changes

    https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8

  • Lazy importer performance overhead. Our custom importer on Python 2 always returns a stub module during import. Python 3's does I/O to verify the module exists then returns a lazy module that is loaded when first accessed. In addition to behavior differences, the I/O may contribute sufficient performance overhead to constitute a problem.

  • A mechanism for extensions to advertise that they are Python 3 compatible. Nearly every extension will break in Python 3. We may want a mechanism that requires extensions to self-declare that they are Python 3 compatible - possibly via special syntax in their source code or the presence of a well-named variable. It might have to be at the source level because Python 3 would need to evaluate code in order to obtain the value of a module-level variable.

4. Beta bugs

Following are things which don't work right now:

If you find anything apart from this not working, definitely go ahead and edit this page and we will fix it.

5. Porting Extensions to Python 3

Nearly every extension will need to be ported to be compatible with Python 3. This is because of fundamental differences between Python 2 and Python 3.

The source code for Mercurial extensions will need to be Python 3 native and will need to be compatible with Mercurial's APIs. In many cases, existing source code will compile on Python 3 but will fail at run-time. Sources of run-time errors include:

  • Use of str instead of bytes. Mercurial uses bytes (b'' strings) in almost all of its APIs and data structures. This is in contrast to much Python code, which uses str and '' strings. It is common for extensions to b'' prefix most strings in order to remain compatible with Mercurial.

  • Use of iteritems(), iterkeys(), etc. These methods from core data structures do not exist in Python 3.

  • Import of renamed modules. Python 3 refactored the locations of various modules in the Python standard library. Extensions may need to take this into account.

Do an Internet search for Python 3 porting to find well-written and comprehensive guides on generically porting code to Python 3.

Extension authors may find the mercurial.pycompat module useful. This modules contains abstractions and utilities for bridging the differences between Python 2 and 3. It is conceptually similar to the six Python module.

If you need help or guidance on porting extensions, you can message on IRC or the development MailingList. We will be happy to help you.

6. Source Rewriting Module Importer

As of at least the Mercurial 5.0 release, Mercurial uses a custom module importer on Python 3 which rewrites source code dynamically as part of importing modules. This module importer is only active for the mercurial and hgext packages. This means that Mercurial's own source code and extensions are not yet native Python 3 source code. So if you look at Mercurial's source code for ideas on how to do something in an extension, behavior in the extension may differ from Mercurial itself due to the presence of this custom module importer.

In the 5.0 release, the custom module importer performs the following actions:

  • Automatically adds b'' prefixes to strings, making all '' literals b'' and effectively changing str to bytes everywhere. i.e. behavior mostly matches Python 2.

  • Modules automatically have from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode added.

  • getattr(), setattr(), hasattr(), safehasattr(), encode(), and decode() functions and methods have string literals in arguments rewritten to the appropriate type because Python requires a str value instead of bytes. (This effectively selectively undoes the global '' to b'' source transformation.)

  • iteritems() and itervalues() are automatically rewritten to items() and values(), respectively.

The source rewriting module importer is intended to be a stop-gap to make porting Mercurial to Python 3 simpler and will be removed in a future release. This is why extensions do not use it.

More history about this importer is available at https://gregoryszorc.com/blog/2017/03/13/from-__past__-import-bytes_literals/


CategoryAudit

Python3 (last edited 2023-02-19 16:08:38 by AntonShestakov)