Hypothesis

Plan to use the Hypothesis framework in Mercurial Tests suite.

1. Inline Python test (for .t) file

This made it to core.

2. Doctest testing

Having a way to use fuzzed input in doctest would be interesting. However, Hypothesis does not currently support doctests

3. Side by side comparison

One way Hypothesis could help greatly, is but generating random usage scenario. Being able to ensure that two variants of Mercurial (eg: repo format, extensions) keep the same Behavior would be very handy

For this would need:

• a list of (parameterized) action to be performed randomly.
• a way to check that result is identical
• a way to specify variant

3.1. proposal for simple initial step

Aiming at something very simple first would be ideal to have all the hard work in place and let people add complexity as needed.

• Actions:
  • generate file
  • add/remove/forget/addremove
  • commit
  • pull/push
• Checks:
  • verify (no repo corruption)
  • state of the working copy (hg status)
  • state of the repo (hg log)
• variants:
  • repository format
  • inabling intrusive extension (largefile, watchman, etc)

3.2. More advanced idea

• Having more in-depth checks, (eg, compression level),
• Selecting relevant/irrelevant checks for each variants,
• Checking difference in resources consumptions,
• Coverage based example discovery (using cov data to build better scenario)
• Static generation of .t test for high quality scenario,

CategoryDeveloper CategoryNewFeatures