#pragma section-numbers 2
<<Include(A:dev)>>

= Performance tracking infrastructure =
'''Status: In production'''

'''Main proponents: [[Pierre-YvesDavid]] [[PhilippePepiot]] [[RaphaëlGomes]]'''

Provide a continuous integration infrastructure to measuring and preventing performance regressions on mercurial

  * Code: https://foss.heptapod.net/mercurial/scmperf/
  * Data: http://perf.octobus.net/
  * Atom feed of regressions: http://perf.octobus.net/regressions.xml
  * Discussion on devel list: http://marc.info/?t=145863695000002

== Goal ==

Mercurial code change fast and we must detect and prevent performances 
regressions as soon as possible.

  * Automatic execution of performance tests on a given Mercurial revision
  * Store the performance results in a database
  * Expose the performance results in a web application (with graphs, reports, dashboards etc.)
  * Provide some regression detection notifications

== Metrics ==

We already have code that produce performance metrics:

  * Commands from the perf extension in contrib/perf.py
  * Revset performance tests contrib/revsetbenchmarks.py
  * --(Unit test execution time)--

--(Another idea is to produce metrics from annotated portions of unit test execution time.

These metrics will be used (after some refactoring for some of the tools that 
produce them) as performance metrics, but we may need some more specifically
written for the purpose of performance regression detection)--


== Tool selection ==

[[https://hg.logilab.org/review/hgperf/raw-file/5aee29f2aee0/docs/tools.html|After evaluating several tools]] we
choose to use Airspeed velocity that already handle most of our needs.

  * http://asv.readthedocs.org/
  * used by the http://www.astropy.org/ projects (numpy)
  * Code: https://github.com/spacetelescope/asv
  * Presentation (2014): https://www.youtube.com/watch?v=OsxJ5O6h8s0

This tool aims at benchmarking Python packages over their lifetime. 
It is mainly a command line tool, ``asv``, that run a series of benchmarks (described 
in JSON configuration file), and produces a static HTML/JS report.

When running a benchmark suite, ``ASV`` take care of clone/pulling the source repository 
in a virtual env and running the configured tasks in this virtual env.

Results of each benchmark execution are stored in a "database" (consisting in
JSON files). This database is used to produce evolution plots of the time required
to run a test (or any metrics; out of the box, asv has support for 4 types of benchmark: 
timing, memory, peak memory and tracking), and to run the regression detection algorithms.

One key feature of this tool is that it's very easy for every developer to use it on 
its own development environment. For example, it provides an ``asv compare`` command allowing to compare 
the results of any 2 revisions.


== Q & A ==

  * '''Q''': What revisions of the Mercurial source code should we run the
  performance regression tool on? (public cs on the main branch only? Which
  branches? ...).
  
  '''A''': Let's focus on public changesets for now and on the
  two branch (default and stable)

  * '''Q''': How do we manage the non-linear structure of a Mercurial history ?
  
  '''A''': The Mercurial repository is mostly linear as long as only one branch
  is concerned, however we don't (and have no reason to) enforce it. For now
  the plan is to follow the first parent of the merge changesets to enforce the
  linearity of each branches.

== Plan ==

  * Fix mercurial branch handling in ASV. '''OK''' https://github.com/spacetelescope/asv/pull/394

  * Use revision instead of commit date as X axis in ASV. '''OK''' https://github.com/spacetelescope/asv/pull/429

  * Provide ansible configuration to deploy the tool in the existing buildbot
  infrastructure and expose the results in a public website when new public
  changesets are pushed on the main branches. '''IN PROGRESS'''

  * Parametrize benchmarks against multiple references repositories (hg, mozilla-central, ...). '''OK'''

  * Parametrize revset benchmarks with variants (first, last, min, max, ...). '''OK'''

  * Implement a notification system in ASV. '''OK''' https://github.com/spacetelescope/asv/issues/397

  * --(Add unit test execution time as benchmark( /!\ We must handle when the test itself has changed /!\ ))--

  * --(Write an annotation system in unit tests and get metrics execution time of annotated portions)--

  * --(Write a system of scenario based benchmark. They should be written as
  mercurial tests (with annotation) and might be kept in a dedicated repository)--

  * Track both improvement and regression ? A change, especially on revset, can have positive or negative
  impact on multiple benchmarks, having a global view of this information could be a good feature: '''OK''' https://github.com/spacetelescope/asv/pull/437

  * Integrate the benchmark suite in main repository, allowing developpers to run `asv compare` locally

  * Write benchmark ''à la'' `.t` tests (against small created repository and by calling contrib/perf commands).

== Code ==

The current work in progress code can be cloned with {{{hg clone -b default https://heptapod.octobus.net/octobus-public/scmperf}}}

----
CategoryDeveloper CategoryNewFeatures