Integration Tests Plan

This page to host a report about Python testing experiments I've conducted, helpers for Python tests and the potential they have.

Will be completed within the 5.6Sprint

Author: GeorgesRacinet

Summary

For my own testing needs outside of the core, I've started to write integration tests, using a small testhelper Python module and the pytest framework.

The resulting tests have been fast to run, easy to write and to plug in CI.

I'm hereby proposing to upstream that helper module, so that extensions and other external code can leverage it without wondering about compatibility, and perhaps start to introduce integration tests in the core.

Why integration tests?

• only if user CLI experience or full end-to-end is not the point of the test
• they're fast
• easy to debug (pdb inline or post-mortem)
• lots of tooling out of the box

the testhelpers module

It has been developed for the testing needs of Heptapod inner components: a very specific hg extension, a custom WSGI wrapper and a gRPC server (there is a pytest-grpc package already)

Its main goal is to provide facilities for setups and to call hg commands. The main object is a wrapper class that provides a repository and high level methods to handle it: commit, update etc.

It is a good time to upstream it, because it is currently part of Heptapod, which is dropping Python 2 support

• speed and general orientation towards testing rather than a complete lib
• ease some pains (bytes) without preventing explicit assertions about them
• high-level feeling, also allowing it to be stable enough for user code.
• self-tested with itself and pytest, with 100% coverage.

Demo: revsets

(chosen because it is quite long and arguably not of much interest for CLI stability)

In preparation for the 5.6 sprint, I've translated roughly a half of test-revset.t.

On my workstation, that half of test-revset.t takes between 25 and 30 seconds. The translated version runs in 0.3 seconds.

The code can be seen here (currently alongside testhelpers.py, i.e. with the Heptapod Python code).

Here is a CI pipeline that ran it for Mercurial 5.5, and the current heads of the stable and default branches (it also ran all the tests of py-heptapod)

Discussion at 5.6 sprint

The idea was well received at the sprint. Here's an excerpt from the minutes:

Yuya: lots of tests don't care about UI; internals can be tested without invoking hg

Should we adopt pytest?

Python ecosystem is using pytest

Alphare: As long as we have a "code style" for automagic fixtures, I'm +100 for using pytest

Complexity of current tests both for performance and maintainability, revset tests used as example, consideration for the bad HTTP tests -> Jörg and Georges will look at that

Conclusion: Georges to send patch series vendoring pytest and demonstrating use/integration with run-tests.py. Fort at least 1 test to demonstrate how it works.

Upstreaming plan

I (Georges) will send patches to introduce a mercurial.testhelpers module in the core. This will provide forward compatibility.

In parallel, a separate Python distribution will be created in PyPI, for the needs of extensions, Python hooks etc that need to be tested on Mercurial versions older than the first release that will provide the testhelpers module. Compatibility down to 5.2 is already almost granted (that was the current hg version by the time development started).

Further a pytest-mercurial can help make this transparent for downstream, and can add some pytest specifics: (very) basic fixtures etc.

Some more thoughts

We have more and more native Python extensions, due to the growth of the Rust codebase. This raises some cross-platform compatibility concerns.

We could have a standard suite of integration tests that the native extensions must pass. If these are well positioned (low level enough that they point out at problems directly, but not depending on implementation-specific details), we can have separate stages of CI builds for them that would be run pre-landing.

For perspective, currently the buildbot runs on ARM64 and FreeBSD take 2 hours to complete. It would be irrealistic to put them in a pre-landing workflow.

Finally, a recurring desire is to have well specified layers of internal API. It would be a good idea to reflect that in the tests organization.