In recent years, Data Science evolved into its own profession as a response to the proliferation of data that needed to be analyzed and made actionable — a job that could not be adequately addressed by any single one of its predecessors, largely Computer Science, Quantitative Sciences,and Consulting. To practice Data Science is to perform tasks along a wide spectrum ranging from research to application to productization, often simultaneously within the same project. The project management methodologies of its predecessors — such as Agile methodology from software engineering or Gantt charts from consulting — are insufficient for the multifaceted nature of a data science project. However, a hybrid approach, combining and adapting existing techniques, is much more successful.
In this talk, we describe the approaches in use in our seasoned and growing Data Science team at Civis Analytics. Specific strategies can minimize wasted time and technical debt for different types of projects. We offer practical tips regarding how development of scripts and production-level modules should be organized. We explain how to conduct an exploratory
investigation while advancing general algorithmic development and implementation at the same time.
Finally, we discuss the open source project management and collaboration tools we have found useful and how to incorporate combinations seamlessly to manage practical
data science workflows.
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
A Hybrid Approach to Data Science Project Management
1. A HYBRID
APPROACH TO DATA
SCIENCE PROJECT
MANAGEMENT
Elaine Lee
O P E N
D A T A
S C I E N C E
C O N F E R E N C E_
BOSTON 2015
@opendatasci
2. Building a Data-Driven WorldTM
Open Data Science Conference
A Hybrid Approach to Data
Science Project Management
Elaine Lee
elee@civisanalytics.com
@elaineklee
3. 3Open Data Science Conference#ODSC
Organizations want to be data-driven but many obstacles stand in their way:
• Communication not trickling up to executives and key decision makers
• Silos between departments, making it difficult to share and collaborate on
analysis
• Data ingestion (ETL or Extract-Transform-Load) is difficult and time-consuming
• Lack of meaningful, yet customizable visual reporting
• Inability to flexibly scale up or down technological needs at a reasonable cost
• Inadequate or overwhelming learning resources about data science
A Common Problem With Many Faces
4. 4Open Data Science Conference#ODSC
Where should Enroll America direct its insurance signup efforts?
Mapping the Uninsured in America
5. 5Civis Analytics | Proprietary and Confidential
As a company, Civis traces its
origins to the 2012 Obama for
America analytics team.
We built a scientific
understanding of each voter.
Our data science influenced
every strategy and tactic: voter
targeting, messaging, media
buys, and fundraising.
This meant the campaign could
allocate resources where impact
would be greatest.
We ran the first
individualized
presidential
campaign
Civis Analytics | Proprietary and Confidential Open Data Science Conference#ODSC
6. 6Civis Analytics | Proprietary and Confidential
Today, we
leverage data
science to help
our clients in
politics, non-
profits, and the
corporate world.
Civis Analytics | Proprietary and Confidential Open Data Science Conference#ODSC
7. Open Data Science Conference#ODSC Open Data Science Conference#ODSC
An easy-to-use,
end-to-end, incredibly
extendable, data science
platform in the cloud for
teams who want to make
great data-driven decisions
to drive their organizations
forward.
Introducing
Civis
8. 8Open Data Science Conference#ODSC
The Civis Approach
ProductConsulting R&D
Applied Data Science
• Tackles the toughest data
science problems we can
find
Data Science R&D
• Generalizes and
automates the solution for
many scenarios
Software Engineering
• Integrates solutions into
user-empowering software
• Highly collaborative departments
• All departments contribute to both our services arm and product development
9. 9Open Data Science Conference#ODSC
The Civis Approach
Our unique team structure allows
us to solve your biggest problems
with custom solutions and the
technology to scale them.
10. 10Open Data Science Conference#ODSC
Strategies and philosophies
• Teams based on Civis’s product and consulting needs:
• “Built around code”
• Semi-annual departmental day-long off-sites to plan upcoming R&D initiatives
• Academia-influenced: evidence-based approaches to finding and reporting best
solutions
• Software development-influenced: standups, code review
• Favorite tools:
Data Science R&D
R&D
Modeling
Methodology
Unstructured
Data
Engineering
11. 11Open Data Science Conference#ODSC
Tools
• Share and discuss data science news
• Receive feedback from colleagues
using our tools
• Discuss implementation
• Lower communication costs compared
to email
Data Science R&D
12. 12Open Data Science Conference#ODSC
Tools
• Prototype new workflows
• Used like a log book to record and
present results
• Share preliminary results with
members of other departments
Data Science R&D
13. 13Open Data Science Conference#ODSC
Tools
• Department heads set milestones,
check progress, and make project
staffing decisions
• Collaboratively plan development on
new functionality or organizational
processes (e.g. recruiting)
Data Science R&D
14. 14Open Data Science Conference#ODSC
Tools
Strategies
• Designate “tag team” on R&D as
default R&D resources for client
engagements
• This is the Modeling Methodology
team
• Other R&D teams’ members may be
staffed on engagements depending on
expertise required
• R&D team member always serves as the
Consulted in the RACI model
• Transparency about challenges is
paramount
R&D <-> ADS
15. 15Open Data Science Conference#ODSC
1. Assemble a project team of R&D data
scientists and Applied Data Scientists
2. Work with Enroll America to refine
requirements and come up with a plan
of analysis, ultimately resulting in the
design and execution of a phone
survey on a sample of individuals,
followed by building a predictive
model for the rest of the country.
3. The Applied Data Science Manager
has weekly calls with Enroll America
and status meetings with the project
team.
4. The project team delivers the
predictions and analysis to Enroll
America.
R&D <-> ADS: A Case Study
Mapping the Uninsured in America
The project team completes a postmortem
and determines these activities could be
automated: model building
16. 16Open Data Science Conference#ODSC
Tools
Strategies
• Designate teams at the interface to
triage issues and plan new
development:
• R&D: “Engineering” team
• Tech: “Modeling” team
• Use module or project-specific chatrooms
to get answers to ad-hoc questions
quickly
• Identify opportunities to form cross-
functional teams, e.g.:
• Developing apps using the Platform’s
API
• Knowledge sharing on best practices
R&D <-> Tech
17. 17Open Data Science Conference#ODSC
1. After the postmortem for the Enroll
America engagement, R&D begins
prototyping automated modeling
functionality and discussing its
implementation with the Tech
department.
2. R&D’s Engineering team finishes the
prototype and works with Tech’s
Modeling team to integrate it as a new
feature in the Platform.
3. During integration, ad hoc
discussions occur on GitHub and
Hipchat to address usability
questions, e.g. resource usage and
input/output specifications.
R&D <-> Tech: A Case Study
Mapping the Uninsured in America
The integration team successfully builds
and integrates the Build Model module in
the Platform.
18. Open Data Science Conference#ODSC
Our approach to data science consulting and product development
is enriched by valuable perspectives of our employees, who come
from a wide array of backgrounds, making our project management
strategies a hybrid of more conventional techniques.
Conclusion
Notes de l'éditeur
Hi everyone, it’s great to be here. My name is Elaine Lee.
I am a Data Scientist in the R&D department at Civis Analytics.
Civis is a Chicago-based data science consulting and software startup, and I’m excited to tell you a little bit about our company and the work that we do.
In particular, I’ll be talking about how the R&D department juggles concurrent development of both our consulting services and our cloud-based data science platform.
I’ll be emphasizing approaches borrowed from other more established industries as it pertains to department projects as well as interdepartmental collaborations.
Many of you are already familiar with data science and the potential it has to change the way things are done.
However, data science has a high barrier of entry for some teams, from a technical standpoint and organizational standpoint.
It can be difficult to wrap your head around the technical needs and quantitative concepts that go into data science.
In addition, it can be hard to assemble the right team to do data science and to keep the work organized.
Picture a team of data scientists working on the same project. Some of them have written R or Python scripts to process the data, do feature engineering, and build models on it.
Some of them have taken the results of the models and produced charts and visualizations in Excel, Tableau, or D3.
All the work is being kept in a few different places – Dropbox, Google Drive, Github, MySQL, …
It is difficult for this hypothetical team to figure out what exactly has been done, and even worse, what efforts have been duplicated.
It is also incredibly difficult to validate the analysis.
Does this sound familiar to anyone?
Fortunately, many of us at Civis Analytics have faced these challenges in our previous work, but we’ve made those challenges a thing of the past!
It didn’t happen overnight, but we were constantly coming up with new ideas to improve the data science workflow by, well, working on a variety of consulting projects and researching new methods.
Today I will talk about what some of these ideas are.
In addition, I will tell the story of how one client engagement provided us a valuable exercise in collaboration and data science best practices we’ve internalized.
Throughout my talk today, I will be using our project with Enroll America to illustrate a lot my concepts.
Enroll America was one of our first clients in 2013.
They wanted our help identifying Americans without health insurance so they knew where to direct their outreach.
This was a challenging problem because of its large scope – they want to do outreach throughout the country! – and it wasn’t obvious what’s predictive of being uninsured.
Why did Enroll America specifically seek us out to solve this problem?
Let’s talk a little about what expertise Civis has for tackling problems like Enroll America’s.
The founding members of Civis Analytics were part of Obama For America’s analytics team in his 2012 re-election campaign.
There, we developed the beginnings of a framework for doing person-level analytics (which is highly relevant for Enroll America).
With scientific levels of rigor, we built models to understand all sorts of relevant vote-related behaviors in order to better identify and persuade supporters, which translated to optimizing how the campaign’s resources were used.
The campaign spanned many months and during that time, lots of models were being built and refined; their results were constantly being sent to those in the field to take action upon.
Developing an organized and repeatable workflow was especially crucial in order to minimize costs, time spent – especially since the staff was small, and any inadvertent human error, especially when models are built at such a large scale.
After the campaign ended in 2012, we re-examined the strategies we employed and the problems they solved.
We realized that if we generalized them, we could solve similar problems for clients in the political, non-profit, and corporate worlds.
Which is exactly what Civis did.
What you see here is a sample of clients, in addition to Enroll America, that we have helped better target their advertising dollars, identify potential customers for greener sources of electricity, and determine public awareness and sentiment on their brand or cause.
In the past year, we took it a step further and we formed a partnership with Discovery Communications to inform more sophisticated audience targeting approaches, ratings forecasting, and marketing spend.
We anticipate making more partnerships like this in the future.
The examples I gave are all problems with a similar flavor to what Civis successfully solved in 2012 – identifying and reaching the people you care about most.
Our diverse client portfolio, innovative approaches, and proven track record have made Civis Analytics’ consulting services highly sought after in the predictive analytics space.
However, we’re equally passionate about removing obstacles to doing data science.
Our steady client pipeline enables us to formalize our approach in the form of a cloud-based data science application.
Our software, Civis, or “the Platform”, supports the entire workflow of a typical data science project, from data warehousing to data processing to predictive modeling to reporting.
This enables organizations to easily take control of their own data and unlock their insights.
This is how we turn our client work experiences into software.
We select novel problems brought forth by our clients and work with them to deliver a solution.
This is primarily addressed by our Applied Data Science department.
Simultaneous to this, we’ve been conducting research and experimenting with different methods to solve the problem, with one eye towards determining how to generalize the solution.
This is primarily done by the Data Science R&D department.
Finally, solutions are integrated into our software platform by the Software Engineering, or Tech, department.
Users of our software platform – clients and our Applied Data Scientists – provide us valuable feedback which are continuously incorporated.
This unique, synergistic cycle enables us to deliver high quality results to our customers.
In our day-to-day work, all departments pitch in on both lines of business, ensuring fluency on all the company’s offerings and thus better decision making.
We also collaborate across departments on all projects, big or small.
Today I will be focusing on how my department, the DS R&D department, manages its workload and how it works with the Applied Data Science and Tech departments.
The R&D department is the only department that is intimately aligned with both lines of business.
We’re split into 3 different teams.
Modeling Methodology focuses on developing new modeling workflows.
Unstructured data specializes in data that can’t neatly be summarized by a flat file, like text data.
Engineering is responsible for managing our production codebases of new features for our software product.
Our department is “built around code”:
“We're trying to build up knowledge and best practices, and being built around code lowers our communication costs, errors, redundancy, and facilitates us making software.”
To roadmap what we build, based on what we’ve learned from recent client engagements, we have day-long semi-annual department off-sites.
When developing new methodologies, we use an academic-influenced approach – empirical and thorough such that our recommended solution covers all the edge cases.
When building out workflows, we follow guidelines common to most software development projects, including some ideas from the Agile methodology – we have daily standups to make sure everyone’s on the same page about the status of the codebases and we do code reviews before any changes are shipped. Our standups are on a per-repository basis, so it doesn’t waste anyone’s time.
To do our work, these are our favorite tools. Let’s take a look at how we use them.
Hipchat and Github form the backbone of our communications.
To those not familiar with these tools, Hipchat is an instant messaging tool for organizations.
Github is a web interface, built on top of the version control system, git, for teams to collaborate on a codebase.
These tools are crucial to our philosophy on being built around code
They enable members across the company to participate by asking questions and generally weighing in
Departmental members use it to discuss implementation
These tools are much faster than email since it makes it easier to ask questions and get answers, since anyone who knows the answer can see the request and thus respond.
When developing new methods, we like to use Jupyter and Google Drive.
We use Jupyter for its Ipython Notebook capabilities.
It allows us to run Python code, especially modules from our codebase, interactively – it allows us to chain components together to make new workflows.
Jupyter also has presentation functionality, so we also use it as a log book to record and present results in internal meetings.
Sometimes we also use Google Drive to record and share results with members of other departments, such as Applied Data Scientists, who have a vested interest in the project but don’t require all the details.
Finally, to take the “pulse” on the R&D department as a whole, department heads use Google Drive and Asana for big picture planning.
Asana is a project management tool which gives department heads a birds eye view of what each team member is working on and how each project is progressing.
Google Drive tools are used to collaborate on planning documents, be it plans for new functionality to build or revising organizational processes, such as rewriting our hiring exam.
That was how we, the R&D department, work together.
How do we work with the Applied Data Scientists, the data scientists in our consulting arm?
To make project staffing seamless, we designate a tag team to serve as the first point of contact for client engagements.
This is the Modeling Methodology team.
However, other R&D data scientists may be staffed on a project depending on expertise required.
The R&D data scientist always serves as the Consulted in the RACI model.
The RACI model is a popular project management model used in consulting.
It emphasizes explicit roles for each team member to ensure accountability.
R is for Responsible, a role held by the applied data scientists.
A is for Accountable; this is the Applied Data Science Manager or project manager
C is for consulted.
And I is for Informed (the client)
Lastly, we are open with Applied Data Scientists about R&D challenges in order to avoid schedule slips on the client engagement.
The project plan is often tracked in Trello, a popular bulletin board app, with bulletin boards for each milestone’s requirements.
Let’s revisit our client story – Mapping the Uninsured in America – to illustrate concretely how we work together.
After Enroll America shared their problem to us, we assembled a project team of R&D data scientists and Applied data scientists to solve it.
We worked with Enroll to refine the problem statement into a set of requirements, ultimately resulting in the design and execution of a phone survey on a sample of individuals, followed by building a model to capture the rest of the country.
The project gets under way.
Throughout the project, the Applied Data Science Manager has weekly status calls with Enroll and with the project team to make sure we’re on schedule.
Occasionally we staffed a couple extra data scientists to the project to make sure we delivered results on time when there was risk of a schedule slip.
For example, we brought in an extra data scientist towards the end of the project to help produce graphs and visualizations of the results.
Finally, we finished our analysis and presented our predictions to Enroll America.
Afterwards, we did a post mortem and realized that automated model building would’ve made us more efficient.
This is because we conducted our experiment in waves and built similar models as the results came in, with the only difference being the input data.
Also, the analysts were each working on individual components of the analysis, writing their own R scripts which had a lot of overlap (such as the data processing steps), which meant a lot of time was wasted.
So that’s how we work with the Applied Data Scientists on consulting projects.
How do we work with the Tech department?
Much like how we work with the Applied Data Science department, we’ve designated a team to interface with the Tech department and they have as well.
That would be the Engineering team on our side and the Modeling team on their side.
The Engineering team in Data Science are data scientists who speak software development and the Modeling team in the Tech department are software engineers who speak data science.
Most of our communications are done using module or project-specific chatrooms and github issue tickets, which gets answers quickly.
To promote really inspired product development, we identify opportunities to form cross-functional teams,
Such as using the Platform’s API to develop new apps
And teaching each other best practices for software development via brownbag sessions.
Let’s revisit the Enroll America project for an example of how the R&D data scientists work with the software engineers.
After the post mortem for the Enroll engagement, we began prototyping automated modeling functionality, communicating to the Tech department the motivation for it and including them in discussions about implementation and feasibility.
Once we finish the prototype, ensuring that it passes all the tests and code review, the Engineering team in R&D work with the Modeling team in Tech to integrate it as a new feature in the Platform.
We use Github and Hipchat to discuss questions that come up, such as resource usage, input/output specifications, and data visualizations we wanted to provide to the end user.
Together, the R&D department and the Tech department successfully built and integrated the Build Model module that exists today in Platform.
In summary, a lot of our approaches have a common theme, which is minimizing communication costs within the R&D department and with other departments.
This is evidenced by our embrace of some free or open-source tools for collaboration and our general belief in transparency about challenges.
We also emphasize collaborative opportunities between departments to strengthen our cohesiveness as a team, be it working on a client engagements together or learning best practices in a seminar format.
A lot of our ideas come from the valuable perspectives of our employees, who come from a wide array of backgrounds.
Thus, our project management strategies are a hybrid of techniques seen in more established industries such as software engineering, consulting, and academia.
I hope the tips presented in my talk today has made doing data science more manageable for your team.
Thank you for your time.
