Integration Tests Plan

This page to host a report about Python testing experiments I've conducted, helpers for Python tests and the potential they have.

Will be completed within the 5.6Sprint

Author: GeorgesRacinet

Summary

For my own testing needs outside of the core, I've started to write integration tests, using a small testhelper Python module and the pytest framework.

The resulting tests have been fast to run, easy to write and to plug in CI.

I'm hereby proposing to upstream that helper module, so that extensions and other external code can leverage it without wondering about compatibility, and perhaps start to introduce integration tests in the core.

Why integration tests?

• only if user CLI experience or full end-to-end is not the point of the test
• they're fast
• easy to debug (pdb inline or post-mortem)
• lots of tooling out of the box

the testhelpers module

(to be completed)

It is a good time to upstream it, because it is currently part of Heptapod, which is dropping Python 2 support

• speed and general orientation towards testing rather than a complete lib
• ease some pains (bytes) without preventing explicit assertions about them

Demo: revsets

(chosen because it is quite long and arguably not of much interest for CLI stability)

In preparation for the 5.6 sprint, I've translated roughly a half of test-revset.t.

On my workstation, that half of test-revset.t takes between 25 and 30 seconds. The translated version runs in 0.3 seconds.

The code can be seen here (currently alongside testhelpers.py, i.e. with the Heptapod Python code).

Here is a CI pipeline that ran it for Mercurial 5.5, and the current heads of the stable and default branches (it also ran all the tests of py-heptapod)