Matthew Treinish, HP - subunit2sql: Tracking 1 Test Result in Millions, OpenStack Israel 2015

1. subunit2sql: Tracking 1 Test Result in Millions
Matthew Treinish
mtreinish@kortar.org
June 15, 2015

2. OpenStack CI by the Numbers
One proposed change generates:
4-25 Devstacks VMs
˜1.5k tempest tests per devstack
˜130 second level guests started per tempest run
Looking at December:
˜12K gating tempest jobs
˜15M tempest tests run in the gate
Looking at Kilo:
˜70K gating tempest jobs
˜>80M tempest tests run in the gate

3. What was missing?
An aggregate view of individual tests
Success and failure rates for tests over time
Run times for each test over time
All of this in a programatically queriable location

4. What is subunit2sql?
A python library and utilities to aggregate and store subunit streams in a SQL DB
Currently 3 utilities:
subunit2sql - Take in a subunit stream and store it in a subunit2sql DB
sql2subunit - Output a subunit stream from a subunit2sql DB
subunit2sql-db-manage - Setup and manage the DB schema
subunit2sql-graph - Tool to generate graphs for analysis with data from a
subunit2sql DB
Provides a python DB api for interacting with the DB which provides basic io
operations and some analysis tooling

5. Catching Performance Regressions
Graphing a single test’s run time:

7. Or when they’re fixed

8. Tracking an individual test’s failure rate
You can use subunit2sql-graph to figure out failure rates for tests:
which also displays percentages:
Fail Percentage: 0.2045%
Success Percentage: 99.7955%
Skip Percentage: 0.0000%

9. Where to get more information
subunit2sql Repository
subunit2sql Documentation
Infra Documentation for subunit2sql
IRC #openstack-qa or #openstack-infra channel on Freenode