Performance tracking infrastructure

Status: Project

Main proponents: Pierre-YvesDavid PhilippePepiot

Provide a continuous integration infrastructure to measuring and preventing performance regressions on mercurial

• Discussion on devel list: http://marc.info/?t=145863695000002

• Demo: https://jenkins.philpep.org/hgperf/

• Atom feed of regressions: https://jenkins.philpep.org/hgperf/regressions.xml

• Code: https://hg.logilab.org/review/hgperf

1. Goal

Mercurial code change fast and we must detect and prevent performances  regressions as soon as possible.

• Automatic execution of performance tests on a given Mercurial revision
• Store the performance results in a database
• Expose the performance results in a web application (with graphs, reports, dashboards etc.)
• Provide some regression detection notifications

2. Metrics

We already have code that produce performance metrics:

• Commands from the perf extension in contrib/perf.py
• Revset performance tests contrib/revsetbenchmarks.py

• Unit test execution time

Another idea is to produce metrics from annotated portions of unit test execution time.

These metrics will be used (after some refactoring for some of the tools that  produce them) as performance metrics, but we may need some more specifically written for the purpose of performance regression detection

3. Tool selection

After evaluating several tools we choose to use Airspeed velocity that already handle most of our needs.

• http://asv.readthedocs.org/

• used by the http://www.astropy.org/ projects (numpy)

• Code: https://github.com/spacetelescope/asv

• Presentation (2014): https://www.youtube.com/watch?v=OsxJ5O6h8s0

This tool aims at benchmarking Python packages over their lifetime.  It is mainly a command line tool, asv, that run a series of benchmarks (described  in JSON configuration file), and produces a static HTML/JS report.

When running a benchmark suite, ASV take care of clone/pulling the source repository  in a virtual env and running the configured tasks in this virtual env.

Results of each benchmark execution are stored in a "database" (consisting in JSON files). This database is used to produce evolution plots of the time required to run a test (or any metrics; out of the box, asv has support for 4 types of benchmark:  timing, memory, peak memory and tracking), and to run the regression detection algorithms.

One key feature of this tool is that it's very easy for every developer to use it on  its own development environment. For example, it provides an asv compare command allowing to compare  the results of any 2 revisions.

A demo build with a patched ASV can be seen here: https://hg.logilab.org/review/hgperf/raw-file/454c2bd71fa4/index.html#regressions?sort=3&dir=desc

4. Q & A

• Q: What revisions of the Mercurial source code should we run the performance regression tool on? (public cs on the main branch only? Which branches? ...).

  A: Let's focus on public changesets for now and on the two branch (default and stable)

• Q: How do we manage the non-linear structure of a Mercurial history ?

  A: The Mercurial repository is mostly linear as long as only one branch is concerned, however we don't (and have no reason to) enforce it. For now the plan is to follow the first parent of the merge changesets to enforce the linearity of each branches.

5. Plan

• Fix mercurial branch handling in ASV. OK https://github.com/spacetelescope/asv/pull/394

• Use revision instead of commit date as X axis in ASV. OK https://github.com/spacetelescope/asv/pull/429

• Provide ansible configuration to deploy the tool in the existing buildbot infrastructure and expose the results in a public website when new public

  changesets are pushed on the main branches. IN PROGRESS

• Parametrize benchmarks against multiple references repositories (hg, mozilla-central, ...). OK

• Parametrize revset benchmarks with variants (first, last, min, max, ...). OK

• Implement a notification system in ASV. OK https://github.com/spacetelescope/asv/issues/397

• Add unit test execution time as benchmark( We must handle when the test itself has changed )

• Write an annotation system in unit tests and get metrics execution time of annotated portions

• Write a system of scenario based benchmark. They should be written as mercurial tests (with annotation) and might be kept in a dedicated repository

• Track both improvement and regression ? A change, especially on revset, can have positive or negative

  impact on multiple benchmarks, having a global view of this information could be a good feature: OK https://github.com/spacetelescope/asv/pull/437

• Integrate the benchmark suite in main repository, allowing developpers to run asv compare locally

• Write benchmark à la .t tests (against small created repository and by calling contrib/perf commands).

6. Code

The current work in progress code can be cloned with hg clone -b default https://hg.logilab.org/review/hgperf

CategoryDeveloper CategoryNewFeatures