Many of us data science and business analytics practitioners perform research and analysis for decision makers on a regular basis. The deliverable of such analysis often results in a Power Point presentation, and/or a model that needs to be productionalized. The code used to produce the analysis also needs to be considered a deliverable. Many of us perform analysis without reproducibility in mind. With the increasing democratization of data, it is becoming more and more important for people that may not have scientific training to be able to create analysis that can be picked up by somebody else who can then reproduce your results. That, and creating reproducible research is just solid science. We are going to spend an evening walking though the various tools available to create reproducible research on Big Data. You will get introduced to the Tidyverse of R packages and how to use them. We will discuss the ins and outs of various notebook technologies like Jupyter, and Zeppelin. You will have an opportunity to learn how to get up and running with R and Spark and the various options you have to learn on real clusters instead of just your local environment. There also be a quick introduction to source control and the various options you have around using Git. The theme of the evening will be “getting started”. We will go over various training resources and show you the optimal path to go from zero to master. Some commentary will be provided around the current state of the job market and intel from the front lines of the data science language wars. This is a large topic and the evening will be fairly dynamic and responsive to the needs of the audience. Bob Wakefield has spent the better part of 16 years building data systems for many organizations across various industries. He has been running Hadoop in a lab environment for 3 years. He is the principal of Mass Street Analytics, LLC a boutique data consultancy. Mass Street is a Hortonworks Consultant Partner and Confluent Partner. In his spare time, he likes to work on an equity investment application that combines various sources of information to automatically arrive at investing decisions. When he is not doing that, you’ll find him flying his A-10 simulator. Full CV can be found here: https://www.linkedin.com/in/bobwakefieldmba/

1. Reproducible Research with R,The
Tidyverse, Notebooks, and Spark
Bob Wakefield
Principal
bob@MassStreet.net
Twitter:
@BobLovesData

2. Who is Mass Street?
• Boutique data consultancy
• We work on data problems big or “small”
• We have a focus on helping organizations
move from a batch to a real time paradigm.
• Free Big Data training

3. Mass Street Partnerships and
Capability
•Hortonworks Partner
•Confluent Partner
•ARG Back Office

4. Bob’s Background
• IT professional 16 years
• Currently working as a Data Engineer
• Education
• BS Business Admin (MIS) from KState
• MBA (finance concentration) from KU
• Coursework in Mathematics at Washburn
• Graduate certificate Data Science from Rockhurst
•Addicted to everything data

5. Follow Me!
•Personal Twitter: @BobLovesData
•Company Twitter: @MassStreet
•Blog: DataDrivenPerspectives.com
•Website: www.MassStreet.net
•Facebook: @MassStreetAnalyticsLLC

6. KC Learn Big Data Objectives
•Educate people about what you
can do with all the new technology
surrounding data.
•Grow the big data career field.
•Teach skills not products

7. ACM Kansas City
We’re looking for a speaker
willing to talk in deep detail
about data engineering
challenges their organization is
experiencing.

8. This Evening’s Learning
Objectives

9. All Material Can Be Downloaded
from GitHub

10. People Associated With
Johns Hopkins
People NOT Associated
With Johns Hopkins
Tidyverse
Reproducible Research
Literate Statistical
Programming
Opinionated Analysis
Data Science as a Science
Brian Caffo
Not So Standard
Deviations Podcast
Roger Peng
Hillary Parker
Hadley Wickham
It’s All Six Degrees of Johns Hopkins’ Biostatistics Department
Jeff Leek
Simply Statistics
Website

11. Motivations For This Evenings Discussion

12. Changes to the Career Field in the Past 15
Months
• Rise of Python for Data Science/Engineering
• Rise of notebooks (Jupyter, Zeppelin, R Notebook)
• Data Science SaaS (cloud, cloud, and more cloud)
• R got a nice NLP package
• Deep Learn all the things!
• Rise of Spark.

13. Reproducible Research
• Introduction to the topic came from the Not So
Standard Deviations Podcast.
• Researches and software engineers approach data
science wildly differently.
• Both sides can learn from the other.

14. Reproducible Research
• Someone should be able to run your exact analysis and get your
result.
• Goal is to reproduce NOT replicate.
• Reproduce = validate your work
• Replicate = validate the conclusions of the study
• This is a lot harder than it sounds.
• Reproducibility hasn’t been totally figured out.
• I still struggle with dependencies
• Build tools for R?

15. Reproducible Research
• Elements of reproducibility
1. Analytic data (the Tidy data)
2. Analytic code
3. Documentation
4. Distribution
• Of these, distribution is the trickiest

16. Reproducible Research
• Literate Statistical Programming
• Combine your analysis and your code into a single
document
• There are several tools for this
• Markdown
• RMarkdown/knitr
• R Studio
• Notebooks

17. Reproducible Research
• A proposed structure of analysis*
• Defining the question
• Defining the ideal dataset
• Determining what data you can access
• Obtaining the data
• Cleaning the data
• Exploratory data analysis
• Statistical prediction/modeling
• Interpretation/Challenging of results
• Synthesis and write up
• Creating reproducible code
*From “Report Writing for Data Science in R”

18. Reproducible Research
• Reproducibility Checklist*
• Start with good science
• Don’t do things by hand
• Don’t point and click
• Teach a computer
• Use version control
• Keep track of your software environment
• Don’t save output
• Set your seed
• Think about the entire pipeline
*From “Report Writing for Data Science in R”

19. Opinionated Analysis Development
• Read Opinionated Analysis Development
• Link in references
• Opinionated analysis = analysis that follows certain
practices
• Follows on to the principals of reproducible research
• Lays out a framework for how an analysis should be
completed

20. Tidy Data
• Three rules that make data tidy:
• Each variable must have its own column.
• Each observation must have its own row.
• Each value must have its own cell.
• No you’re not crazy. Yes that’s third normal form.
• I don’t have to deal with this issue often if ever.

21. Tidyverse
• Used to be called the Hadleyverse
• An ecosystem of packages designed with common
APIs and a shared philosophy
• Helps you get your data tidy
• Also assumes that your data is tidy

22. Part II: Tools
• Git
• Modern Source Control
• Code repositories: GitHub and Bitbucket
• GUI: Sourcetree
• Rmarkdown/knitr
• Appears to be strictly an R Studio thing
• Pandoc Markdown/Jupyter
• Julia, Python, R
• Rebranded Ipython

23. Part II: Tools
• SparklyR/Databricks
• SparklyR provides an R API for Spark with dplyr
• AWS/S3
• Helps solve the problem of accessibility to data
• Can be annoying to manage
• Tidyverse
• A set of packages that makes working with data easier