2018-08-23 Apache Yetus: Precommit
•Télécharger en tant que PPTX, PDF•
0 j'aime•388 vues
This is a talk I gave at a client, covering some of the patch testing facilities of Apache Yetus.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à 2018-08-23 Apache Yetus: Precommit
Similaire à 2018-08-23 Apache Yetus: Precommit (20)
Plus de Allen Wittenauer
Dernier
Dernier (20)
2018-08-23 Apache Yetus: Precommit
- 1. Apache Yetus: Precommit Allen Wittenauer Effective Machines August 28, 2018
- 2. Apache Yetus is a collection of libraries and tools that enable contribution and release processes for software projects.
- 3. 3© 2013 Liz West https://www.flickr.com/photos/calliope/9070783026
- 5. 42,000+ JIRA Issues filed against Apache Hadoop
- 6. 2018-08-30 Initial (Primary) Goals Project consistency Actionable feedback for contributors w/out a committer Tool that also ran locally 62018-08-30
- 7. 7
- 8. 8
- 9. 2018-08-30 Rewrite (Primary) Goals Project consistency Actionable feedback for contributors w/out a committer Tool that also ran locally Branch support Reduce load on ASF Jenkins – Smarter analysis: don't run useless tests Better UI; cleaner output Modularity = easy to add new test types 92018-08-30
- 10. 10
- 11. 11
- 12. 12
- 13. 2018-08-30 Rewrite Goals, Revised Project consistency Actionable feedback for contributors w/out a committer Tool that ran locally Branch support Reduce load on ASF Jenkins – Smarter analysis: don't run useless tests Better UI; cleaner output Modularity = not only pluggable tests, but pluggable projects 132018-08-30
- 14. 14
- 15. 15
- 16. 16
- 17. 2018-08-30 Compiled Language Support – C/C++ – Java – Scala Interpreted Language Support – bash/ksh/sh: shellcheck – perl: Perl::Critic – Python: pylint – Ruby: rubocop, ruby-lint Tool Support (part 1) Build Tools – ant – autoconf – cmake – gradle – maven – make – “nobuild” Bug Systems – Bugzilla (RO) – GitHub – JIRA 172018-08-30
- 18. 2018-08-30 Unit Test Formats –ctest –Junit –TAP Runtime Support –Automation (e.g., Jenkins) –Docker-ized build environments –Multiple JVMs –Patch branches –Per-instance maven repos –Resource limit protections –Various reporting formats Tool Support (part 2) Specialized Tests – Apache RAT –@author –Dockerfile lint –Java checkstyle –Javadoc –Java findbugs –maven site –path length –Scaladoc –whitespace –XML In Development – OWASP Dependency Checks 182018-08-30
- 19. What if the entire source tree was treated as the patch?
- 21. qbt (quality build tool)
- 22. 2018-08-30 qbt Goals Stop Missing Tests Reduce ASF Jenkins Load Consistency – All sub-projects built the same way – Nightly builds using same params, environment, output as patch testing "Comprehensive Follow-up" – API expectations missing from unit tests – Inter-module findbugs issues What is the actual state of the source tree? 222018-08-30
- 23. Output
- 24. 24
- 25. 2018-08-30 test-patch Output: Header Overall Vote: – Bad: – Good: 252018-08-30
- 26. 2018-08-30 test-patch output: Vote Table 262018-08-30 Pre-Check Compile Cycle (Branch) Compile Cycle (Patch) Unit and other Tests
- 27. 2018-08-30 test-patch Output: Test Failure Table 272018-08-30
- 28. 2018-08-30 test-patch Output: Footer 282018-08-30 Configuration Information More info on Failures Configuration Information
- 29. Misc. Hints
- 30. 2018-08-30 Git Hub Support Multiple ways to get the PR: – GH:<PR #> – GHSHA:<PR SHA> – https://github.com/your/repo/pull/## – https://github.com/your/repo/pulls/##.patch – https://github.com/your/repo/pulls/##.diff – JIRA comment (if enabled) Branch awareness = merge branch of the PR Squashed commits work better – Yetus does not honor the commit order vs. parent branch i.e., git apply not git merge – Per-line commenting goes a bit wonky (if enabled) 302018-08-30
- 31. Patch File Naming Conventions JIRA Project and Issue Number Branch Revision Patch Filename HBASE-55 Master 0 HBASE-55.000.patch HBASE-55 Master 3 HBASE-55.003.patch HBASE-55 branch-1 0 HBASE-55-branch-1.000.patch HBASE-55 branch-1.2 2 HBASE-55.branch-1.2.002.patch HBASE-55 branch-1 2 HBASE-55.branch-1.2.patch ??
- 32. Log Files are the Key Located in --patch-dir/Jenkins artifact dir/linked in footer Three kinds – Full logs of build tool runs – Individual test logs – Diffs of pre- and post- runs Tell them apart: – "Before": branch-* – "After"/qbt: patch-* – diffs of the two: diff-* – Ant/Maven/etc module – JDK version (MultiJDK mode only) Generics lack -fixes
- 33. 2018-08-30 Administrivia Website: https://yetus.apache.org Mailing list: dev@yetus.apache.org Twitter: @ApacheYetus Current Release: 0.7.0 (2018-01-29) (0.8.0 undergoing vote right now) Yetus is an archaic synonym of the Cymbium genus of gastropods First ASF TLP with a ‘Y’ 332018-08-30
Notes de l'éditeur
- Like a toolbox, one doesn't need to use all the tools available to complete a job. This talk will concentrate on only one of them.
- This talk is going to focus on precommit. test-patch was a single executable for a very long time so many people know only that part of precommit. I tend to use the terms interchangeably.
- This doesn't count the projects that have moved out from underneath the Apache Hadoop umbrella. But that's still a lot of tickets! The only way to handle that many is to automate the process.
- Nigel Daley, the first person who was sort of working on QA and release facilities, decided to build such a tool. 1) A tool makes contributors get the some of the same feedback regardless of the person doing the review. 2) It was important that contributors get feedback without the need of a committer to get involved. This greatly reduces both the amount of time needed to get a contribution committed. 3) Running a tool locally means that contributors can get mini-reviews without even uploading the patch!
- The first run of test-patch that I could find in JIRA!
- Years later, a group of folks associated with ODPi decided to host a bug bash to run through issues. By this time, test-patch was showing its age (it hadn't really been updated in a while.) If a ton of patches got submitted, it would bring the ASF Jenkins infrastructure to its knees...
- Enter a major rewrite. Additional goals: Ability to control which branch a patched was being tested on. If a patch is only modifying shell code, it doesn't make sense to run the entire java unit test. If a patch only modifies documentation, don't run unit tests. etc, etc. it also liimits testing to only be where code was modified (As a result, it trades execution time for accuracy during patch testing. More on this later) The old output was hard to read. The old script was one big monolithic script. Very hard to add stuff.
- An early run of the new version. Note that this patch took 19s to test vs. the hours it would have otherwise taken.
- Good ideas tend to propagate. Nigel's original code got spread out everywhere within the ASF.
- Others noticed the work that was happening in Hadoop and wanted to update their version too.
- Rather than end up in the exact same pattern, change the code to be smarter: if 95% of the code between projects is the same, why not just make the 5% configurable?
- Fast forward... Apache Yetus is born as a top level project feeding a lot of other projects.
- A rough outline of test-patch architecture. Almost everything outside of the core driver code is replaceable and expandable by providing more snippets.
- Josh Elser was pretty typical of the reactions of the re-write. Everyone was used to seeing everything run all the time and it was shocking that just modifying the build instructions didn't trigger a full build and test of the entire source tree!
- Lots of things are supported out of the box!
- Runtime support is special: There are special things triggered if you tell test-patch that it is running in an automated environment! Docker-ized build support includes cleaning the local docker image cache as well as killing broken containers that might still be running. Unit tests and other things can be configured to run with multiple JVMs to get better version coverage. 4) Maven doesn't lock the repo, so a very common situation on a lot of build hosts is having two maven runs updating the same maven repo. This is bad because the two runs might stomp all over each other. test-patch can be configured to set the maven repos so this doesn't happen. 5) Most of the work in the past year or so has been spent on adding resource limits to prevent broken build systems from crashing boxes. 6) Reports can be sent to various bug systems so that end users don't have to look at the CI logs to see what happened.
- --empty-patch == "no patch was provided, so just look at the source tree as is"
- aka daily/nightly builds with precommit as the driver. This provides consistency for both types of builds.
- precommit is smart enough to run each maven module (or whatever) separately. This means all unit tests than can run, will run, regardless of what module their in and regardless of whether a previous module failed. Hadoop is a large project. Breaking up the nightly jobs into chunks left a lot of the source basically not getting tested. (I estimate that hadoop-tools wasn't part of a nightly build for *several years*) Additionally, running everything in one job meant all of the startup and shutdown costs were quadrupled 4) in order to combat patch testing accuracy, the nighty build acts as a way to run through the days commits and make sure all is ok.
- All the output of a patch run as seen in JIRA!
- header is a quick summary
- pre-check is mostly "is the patch safe?" "is configuration sane?" etc types stuff. Patches can be failed here and will short-cut to the end of the testing. Next is a precompile->compile->postcompile cycle aka the Compile Cycle. It gets executed for both the source tree prior to the patch and after the patch has been applied. This allows test-patch to pinpoint with really great accuracy exactly what got broken by the patch. In some cases, this also means some stats show how much change. Then unit tests and various other one-offs run.
- Some plugins are able to isolate exactly what broke and provide specific guidance. Others are typically to verbose and don't that.
- That's ok because access to logs are also provided. Some times these are specifically what's different and sometimes not. It just depends upon the nature of the failure.
- When using github, test-patch will test against the branch that the PR was attempting to merge into. It's also important to know that test-patch applies the patch on top of that branch. It is not merge a la a deck of cards.
- When using files or JIRA, be careful of how things are named. The bottom one confuses test-patch because a branch-1.2 exists.
- When using Jenkins or other CI systems, the patch-dir should be set as the artifacts dir to save off. This allows for people to get to the logs and for test-patch to reference them in the job output.