Keynote at Geode Summit 2016 by Dr. Justin Erenkrantz, Bloolmberg LP. Creating the Future of Big Data Through "The Apache Way" and why this matters to the community
#GeodeSummit Keynote: Creating the Future of Big Data Through 'The Apache Way"
1. >>>>>>>>>>>>>>>>>>>>>
CREATING THE FUTURE
OF BIG DATA THROUGH
"THE APACHE WAY”
WHY THIS MATTERS TO THE
COMMUNITY
Dr. Justin R. Erenkrantz, Bloomberg LP
justin@erenkrantz.com / @jerenkrantz
2. WHY SHOULD I PAY ATTENTION?
» Mentor to Apache Geode and HAWQ
» Commi5er to Apache HTTP Server, APR, Subversion, Serf
» Former President and Director of The Apache SoBware
FoundaDon
» Ph.D. from University of California, Irvine
» DissertaDon: "ComputaDonal REST: A New Model for
Decentralized, Internet-Scale ApplicaDons”
» Head of Compute Architecture at Bloomberg LP
» ~50 billion Dcks DAILY flow through our systems
2
3. TECH @ BLOOMBERG: OPEN SOURCE
3
» The core of our Bloomberg Professional plaorm has evolved away
from proprietary code
» FoundaDons of our next-generaDon infrastructure - OpenStack, Ceph,
Hadoop, Spark, Solr, Chromium, Chef - are all open-source
» No longer can vendors tell us that they won’t fix a criDcal bug
» Places a lot of pressure on our partners to collaborate openly
» Giving back to the community - h"ps://github.com/bloomberg/
» Allows us to innovate at the higher levels – helping our customers
make sense of the firehose of informaDon that is available to them
6. PHILOSOPHY OF THE APACHE SOFTWARE FOUNDATION
6
» Let the contributors do what they do best: contribute.
FoundaDon exists to do the rest.
» Does not pay for contribuDons
» Many are sponsored by a third-party
» Staff ASF has are focused on infrastructure/PR/etc
» Does not pick “winners” or “losers”
» “CompeDDon” between ASF projects perfectly
acceptable as long as there are healthy
communiDes…think Geode and Ignite (!)
8. ROLE OF APACHE INCUBATOR
8
» Each project (TLP) is run relaDvely autonomously
» Project karma does not automaDcally carry over
» If I can commit to Geode, it doesn’t mean I can
commit to Ignite! (But, I could likely earn it easily!)
» Incubator was formed in 2003 as we were struggling to
scale the foundaDon and repeat the model. It worked.
» If a podling does not have a healthy community, it’ll
never graduate. That’s OK. If the podling does
become a TLP, but later loses its community, it’ll end
up in the Arc. That’s OK, too.
9. TRANSPARENCY & MERITOCRACY
9
» Roy’s Mantra: "If it's not on the list, it didn't happen.”
» Apache in the age of GitHub, JIRA, ReviewBoard, etc.
» Is the mailing list doomed?
» Generation gap may mean email isn’t preferred
» Tools are always secondary to process
» Transparency is the aim: allows others to have a voice
» The tools and process are never about prohibiting face-to-
face contact - but, ensuring that there is equal access for
participation and permitting asynchronous decision making
» Making decisions in a synchronous echo chamber (Slack,
IRC, etc.) is not conducive to transparency
10. MAKING DECISIONS
10
» Voting is the way contributors are (and feel) empowered
» “Binding” votes from recognized contributors (PMC)
» Vote on code, ideas, and, most importantly, releases
» Minimum acceptable quorum: 3 voters
» Minimum acceptable time frame: 72 hours
» The power of the dreaded “-1” (veto)
» Code can be vetoed, but not releases
» Veto should be cast as a last resort; used to foster
discussion
14. NORMS OF THE COMMUNITY
14
» Over the years, most disputes I have seen come down
to norms that were not agreed upon or documented
» Forming an explicit consensus on release versioning
and compaDbility rules up-front is so incredibly helpful.
» Projects always have a tension between “new
features” and compaDbility. Decide where the
community wants to be early on.
» The Geode wiki secDon is great. Keep it up!