Attend this talk and learn how Puppet Labs handles community contributions and the FOSS tools we’ve published to automate much of the process. Puppet Labs handles thousands of contributions from hundreds of contributors and we’ve integrated Github, TravisCI and Trello to manage all of it. Come see how we do it and what we’ve built! Jeff McCune Software Developer, Puppet Labs Recovering systems engineer turned software engineer, responsible for helping the Puppet community contribute their code to our open source projects.

1. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 1/29
Automating Community Contributions
Jeff McCune
Twitter: @0xEFF
Email: jeff@puppetlabs.com
http://puppetlabs.com

2. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 2/29
We'll Cover:
Process
Challenging 2012 process
Improved 2013 process
Automation & Tooling
Github (patches)
Travis (tests)
Trello (work items)
Puppet Webhooks (automation)
Questions
Please ask questions during or after the talk, there will be time.

3. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 3/29
Merge more patches with automation
Dedicated people work well
Context switching is a challenge
Automation
Reduce context switching
Single pane of glass

4. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 4/29
2012 Process
Platform Team
Bug Fixes
New Features
Security Incidents
Community Pull Requests
Two types of work
Planned - Bugs and Features
Unplanned - Pull Requests and Security
Pull requests often came last in the priority list

5. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 5/29
2012 Process Outcome
Large and growing backlog
No response for months
Sprints to "catch up"
Public complaints from key community members
Stressed out team

6. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 6/29
2012 Results

7. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 7/29
2013 Process
Community Team
Community Pull Requests
Goal: 2 PM next business day response
One type of work, not two
Reduced context switching
Platform Team
Bug Fixes
New Features
Security Incidents
Reduced context switching

8. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 8/29
2013 Process
Community Team responsibilities:
Community Pull Requests
Platform Team responsibilities:
Bug Fixes
New Features
Security Incidents
One type of work:
Unplanned - Pull Requests and Security
Pull requests are consistently prioritized.

9. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 9/29
2013 Results
We're merging ~ 15 community patches per week

10. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 10/29
Process Conclusion
Dedicated People
Minimize context switching
Decreased backlog, faster response time.
Backlog down to 25 from 80
New pull requests responded to within two days instead of months.
Automating the process
With a solid process in place, we focused on automating the tedious aspects.

11. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 11/29
Work in Flight
Trello

12. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 12/29
Automation and Tooling
Github (patches)
Travis (tests)
Trello (work items)
Three separate tools, lots of switching back and forth.
Puppet Webhooks puts it all in once place.

13. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 13/29
Why more Tooling?
Too many sources of information are tedious and error prone to review.
Issue Tracker (Should I work on this now?)
Jenkins and Travis (Does this patch break stuff?)
Github (What are people saying about this patch?)
CLA (Has the contributor signed the license agreement?)
Trello (What should we be working on?)
All of this information is important. It's also spread out on four or more different sites. It
was often overlooked.
A solution is to move all of the information into one place.

14. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 14/29
All information in Trello
Our goal is to get all of the information we need to review a pull request into Trello.
New pull request creates a card.
Code comments
Contributor bio
CLA Information
select(contributor)
Completed this week
Puppet, Facter, Hiera, stdlib on one board

15. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 15/29
Gepetto Bot
Gepetto Bot is a service account with a profile on Github, Trello, and Heroku. The
puppet-webhooks Heroku app logs into these services as Gepetto Bot.

16. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 16/29
Consolidated Information
Gepetto Bot gathers contributor bio using the Github API:

17. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 17/29
Board Columns
Cards move left to right. Gepetto Bot creates them in the Response Needed column.
Response Needed
Accepted
Doing
Waiting on Contributor
Waiting > 1 week
Going through CI
Finished this Iteration

18. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 18/29
How it works
Webhooks
Github executes a callback when events happen. The callback is a very simple HTTP
POST with a JSON body containing data about the event.
Think push notifications for the web.

19. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 19/29
How it works
The Puppet Webhooks code running in Heroku receives the HTTP POST and:
1. Github posts JSON to our Heroku app
2. Persist the JSON to a Delayed Job queue
3. Start a Delayed Job worker process using workless gem
4. Data is posted to Trello in a matter of seconds

20. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 20/29
Why use Delayed Job?
Heroku provides:
Small PostgreSQL databases for free
750 hours of process time for free
31 x 24 = 744 hours for the web process
6 "extra" hours for free
Automatic retries with back-off
The workless gem allows us to only run DJ worker processes when there's actually
work to be done.
Note: While this is all free of charge, a credit card is required to use the Heroku API
with workless.

21. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 21/29
Automatic Process Management
The web process runs persistently.
bundle exec thin start ­p $PORT ­e $RACK_ENV
The worker process runs only when there's work to be done.
bundle exec rake jobs:worksilent
Workless automatically adjusts the worker dyno:

22. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 22/29
What is automated?
New Pull Request => New Trello Card
Code Comment => Trello Comment
Collect Contributor Bio
Card due Date
Weekly Summary of completed cards
Unimplemented event handlers
Pull Request is closed
New code is pushed
Travis CI tests complete

23. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 23/29
Weekly summary email
Sent to puppet-dev mailing list weekly
Automated summary of completed Trello cards
Scans card comments and extracts summary:matches

24. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 24/29
How the weekly summary works
Github Gist as a persistent data store!
1. App reads a Liquid template from a raw github URL
2. Collect all completed Trello cards
3. Fill in the template with the card data
4. Post the resulting Markdown back to the Gist
No persistent state in the app or the database. Liquid template is easily updated.

25. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 25/29
Summary Template
Header:
## Puppet Dev Community Summary
* Finished Card Total: **{{ cards.size }}**
{% for section in sections %} * {{ section[0] }}: {{ section[1].size }}
{% endfor %}
Sections:
{% for section in sections %}## {{ section[0] }}
Cards:
{% for card in section[1] %}#### [{{card.title}}]({{card.url}})
{{card.message }}

26. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 26/29
Post back to Github
heroku run bundle exec rake jobs:summary

27. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 27/29
Tooling Wrap Up
Automate the Tedious Stuff
Try the Puppet Webhooks App
github.com/puppetlabs/puppet-webhooks

28. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 28/29
What we covered
Process
Context Switches = (ಥ_ಥ)
Dedicated People = (◕‿◕)
Automation & Tooling
Github, Travis, Trello
Puppet Webhooks automates all three
Key Message
Merge more patches with automation

29. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 29/29
Thank You! Questions?
Jeff McCune
0xEFF
jeff@puppetlabs.com