Slides from Mike Keating's, Big Data Project lead at News International, talk on Big Data at News. 18th Big Data London meetup.

1. Big Data at News International!
Welcome !
Big Data at News International!
!
Big Data Event - 29/05/2013!
Mike Keating: Product Owner!
@mikerkeating!
Slide 01/14!

2. Big Data at News International!
Introduction!
• Where to start!
• Data and decisions!
• Our technology choices!
• Lessons!
!
Big Data Event - 29/05/2013!Slide 02/14!

3. Big Data at News International!
Where to start…!
Big Data Event - 29/05/2013!Slide 03/14!
Lots of data, suppliers, technology and teams!
!
Take control, bring digital data into one place!
Link data sets and look for new approaches!
!
Make data/ information/ knowledge/ insight available!
Build awareness and decision making capability!

4. Big Data at News International!
Making the basics available via dashboards!
Big Data Event - 29/05/2013!Slide 04/14!

5. Big Data at News International!
Understanding our content and consumption!
Big Data Event - 29/05/2013!Slide 07/14!
Visits per Visitor
ViewsperVisit
Behaviours Across Website Sections

6. Big Data at News International!
Understanding our content in a social world!
Big Data Event - 29/05/2013!Slide 08/14!

7. Big Data at News International!
Tracking
subscrip.on
growth
across
products!
Big Data Event - 29/05/2013!Slide 05/14!
Days Following First Subscription
Subscriptions
Product Growth Following Launch

8. Big Data at News International!
Analysing attributes that indicate churn!
Big Data Event - 29/05/2013!Slide 06/14!

9. Big Data at News International!
Designing products based on patterns in navigation !
The
iPhone
–
Octopus
Naviga4on
The
Website
–
Flower
Petal
Naviga4on
The
iPad
Edi4on
-­‐
Linear
Naviga4on
Big Data Event - 29/05/2013!Slide 10/14!

10. Big Data at News International!
Our technology choices – what?!
Big Data Event - 29/05/2013!Slide 11/14!
Infrastructure: AWS EC2, S3, RDS, EMR, Cloudformation, Vagrant!
Ops: Jenkins, Anthill Pro, Maven, Nexus, Zabbix, CloudWatch!
Code & Config: Puppet, Github!
Data Retrieval: Java & Python!
Data Pipeline: Java Map Reduce, Apache Crunch, !
Spark, MRUnit, Celery, RabbitMQ, Python, Flume!
Data Storage: HDFS, HBase, AWS S3, MySQL, Redis!
Data Schema: Avro!
Data Access: Python APIs, Tornado, S3, Hive & AWS EMR!
Analysis: R, Pandas, Excel -> analyst’s choice!
Visualisation: R, D3, Highcharts, Google Charts!
Products: APIs, JS+HTML+CSS!

11. Big Data at News International!
Our technology choices – why?!
Big Data Event - 29/05/2013!Slide 11/14!
• Team: Build on team’s skillsets and knowledge. !
• Recruitment: Be conscious of “hire-ability”!
• Open Source: Big wins from usage; great
communities; contribute back!
• Versions: use what works; work with alpha releases
but not as production code!
• Consistency: Try and use what you do today, e.g.
AWS!
• Flexibility: Use a “better” product where it makes
sense!

12. Big Data at News International!
Our technology choices – who?!
Big Data Event - 29/05/2013!Slide 11/14!
• Tech Lead: Architecture; Design; Hands-On!
• Delivery Manager: Project Management w Agile!
• Hadoop: build from Java Map Reduce !
• Python: Tornado, Native Python, real-time processing!
• Data Science: Hive, R, Modeling!
• DevOps: AWS, Vagrant, Puppet!
• Experience: Practical experience of Hadoop in production!
• Capability: Ability to learn new tech, design and build!
• Demo: contributions to projects, working examples!

13. Big Data at News International!
Lessons!
Big Data Event - 29/05/2013!Slide 12/14!
• Building awareness & common knowledge!
• Building on existing teams, systems and their work!
• Looking for extra capability and output!
• Focus on visuals – it needs to be sharable/ visible!
• Working with range of teams to share outputs!
• Making good tech choices!

14. Big Data at News International!
Thanks!
Big Data Event - 29/05/2013!Slide 13/14!
Big Data Team – DevOps, Hadoop, Python, UI,
Analysts, Test!
!
Technology Teams – Design, Production Ops, Perf
Testing, Security, Products, Platforms, Service Desk!
!
Editorial, Marketing, Commercial, Finance Teams!

15. Big Data at News International!
!
!
Thanks!
Big Data at News International
Contact
Us
News
Interna.onal
Technology
@techatni
Mike
Kea.ng;
Product
Owner
@mikerkea.ng
Jobs
via
hBp://joinnitech.co.uk/
Big Data Event - 29/05/2013!Slide 14/14!