Creating great products powered by big data can be challenging.
Data science work is often ambiguous, which can make results unpredictable and scheduling almost impossible.
Many of the popular software engineering processes just won’t work for these innovative and ambitious projects. Waterfall falls apart; it just doesn’t make sense to define the product before understanding the limitations of the data and technology. And shoehorning data experiments into tight agile sprints is both difficult and doesn’t necessarily lend itself to discoveries that involve a lot inspiration and perspiration before a light bulb moment.
Even with a working process, few teams collaborate truly effectively. Projects that involve machine learning, algorithm development, or other deeply technical endeavors, are filled with advanced math and complicated terminology, which leaves plenty of teams with communication gaps that prevent the synergy realized when working cohesively.
Thankfully there are solutions to these problems! Based on personal experience, and interviews with many other leaders spearheading big data initiatives, this session aims to distill these lessons into actionable strategies you can use to improve process and communication for your own team.
3. What are they
doing all day?
Tuesday, February 26, 13
4. Data science is different
http://data.whicdn.com/images/29273643/funny-science-news-experiments-memes-super-science_large.jpg
Tuesday, February 26, 13
5. Research doesn’t fit
the traditional SDLC
image src:http://laurajul.dk/wp-content/uploads/2011/10/Screen-shot-2011-10-06-at-00.03.57.png
Tuesday, February 26, 13
6. Good help
is hard to find
(and keep)
image src: legoexpress.tumblr.com
Tuesday, February 26, 13
7. What are they
doing all day?
Tuesday, February 26, 13
8. Bring transparency
image src: www.ideachampions.com
Tuesday, February 26, 13
9. Communication
Logistics
Trust
Tuesday, February 26, 13
12. Do you speak the same language?
image src: abclang.livejournal.com
Tuesday, February 26, 13
13. What does it mean to be finished?
image source: ladywhodoesntlunch.blogspot.com
Tuesday, February 26, 13
14. Define
“quality”
image source: http://www.sodahead.com/fun/
Tuesday, February 26, 13
15. R
Give them
a lesson in
P
semantics
What do precision
and recall mean?
Tuesday, February 26, 13
16. Before
Precision: 80%
Recall 25%
Tuesday, February 26, 13
17. After
For the top search terms* accessories appear
25% of the time on the first page.
For top search terms the head products are
present 90% of the time in the top 3 results,
98% of the time in the top5.
✴ Top search terms are the 1000 most popular queries on our website
over the last 30 days.
Tuesday, February 26, 13
18. Measure with data that matters
image source: http://www.freepatentsonline.com/6971185-0-large.jpg
Tuesday, February 26, 13
19. This is a really hard
problem.
Tuesday, February 26, 13
20. We know it is hard...
Tuesday, February 26, 13
21. We know it is hard...
but we don’t know
why it is hard.
Tuesday, February 26, 13
34. They don’t
call it
research
for nothin’
image source: http://sausagetails.com/2012/07/
Tuesday, February 26, 13
35. You can’t predict the future
image source: maxseesmovies.blogspot.com
Tuesday, February 26, 13
36. Applying “agile” to R&D
image source:http://mousebreath.com/wp-content/uploads/2011/08/funny-farmers.jpg
Tuesday, February 26, 13
37. Applying “agile” to R&D
Backlog of
Trips to Hawaii experiments
That’s doesn’t
sound agile....
image source:http://mousebreath.com/wp-content/uploads/2011/08/funny-farmers.jpg
Tuesday, February 26, 13
38. Applying “agile” to R&D
Regular Demos
Backlog of
Trips to Hawaii experiments
That’s doesn’t
sound agile....
image source:http://mousebreath.com/wp-content/uploads/2011/08/funny-farmers.jpg
Tuesday, February 26, 13
39. Applying “agile” to R&D
Regular Demos Defined workflow
with iterations
Backlog of
Trips to Hawaii experiments
That’s doesn’t
sound agile....
image source:http://mousebreath.com/wp-content/uploads/2011/08/funny-farmers.jpg
Tuesday, February 26, 13
40. No Start Here
Collect
Data Ye
Do we have the s
right data?
Data
No
Do we have the
Are we
infrastructure to
finished or do Build it...
analyze the
we need
Re data?
more.....? s ea
rch
Y es
Get results Run experiments
Tuesday, February 26, 13
41. Experiments can take a while...
image source: www.wallpapermay.com
Tuesday, February 26, 13
42. Take tools out of the equation
image source: http://www.holesinyoursocks.com/2011/02/14/funny-monday-tool-love-happy-valentines-day/
Tuesday, February 26, 13
43. Focus on feeds and files
Image source: http://carcat.files.wordpress.com/2009/03/funny-pictures-cat-searches-for-a-file.jpg
Tuesday, February 26, 13
44. Format & storage standards
/prices
/full
/date=2012-07-01
price-obs.2012-07-01.csv.gz
/date=2012-07-02
/inc
/date=2012-07-01
2012-07-01T00-10-00.csv.gz
2012-07-01T00-20-00.csv.gz
Tuesday, February 26, 13
45. Where is your golden set?
image source: ads/2011/09/4bc2dd714ecb9eebb3a66d074638.jpeg
Tuesday, February 26, 13
48. What are they
doing all day?
Tuesday, February 26, 13
49. Building trust
image source: http://writealoud.com/funny-dinosaur-pictures/
Tuesday, February 26, 13
50. Their motivations
image src: http://www.fredhoogervorst.com/oni.app/local/upload/03897400db.jpg
Tuesday, February 26, 13
51. Their motivations
Hard problems
to solve
image src: http://www.fredhoogervorst.com/oni.app/local/upload/03897400db.jpg
Tuesday, February 26, 13
52. Their motivations
Hard problems
to solve
My work in the wild
serving customers
image src: http://www.fredhoogervorst.com/oni.app/local/upload/03897400db.jpg
Tuesday, February 26, 13
53. Their motivations
Hard problems
to solve
Recognition for a
job well done
Being GOLD!
My work in the wild
serving customers
image src: http://www.fredhoogervorst.com/oni.app/local/upload/03897400db.jpg
Tuesday, February 26, 13
54. Their motivations
Hard problems
to solve
Recognition for a
job well done
Open the door to
higher-ups
Being GOLD!
My work in the wild
serving customers
image src: http://www.fredhoogervorst.com/oni.app/local/upload/03897400db.jpg
Tuesday, February 26, 13
69. Think with your business hat
image source: memegenerator.net
Tuesday, February 26, 13
70. How do you surface
ideas & insights?
image source: http://static.someecards.com/someecards/usercards/1327680406361_4248272.png
Tuesday, February 26, 13
71. Show ‘em
image source: neslihandurmusoglu.edublogs.org
Tuesday, February 26, 13
72. And
show ‘em
often
image source: http://www.social-science.co.uk/research/
Tuesday, February 26, 13