This is a talk on a fundamental approach to thinking about scalability, and how Hadoop, HBase, and Lucene are enabling companies to process amazing amounts of data. It's also about how Social Media is making the traditional RDBMS irrelevant.

1. Building a Business on Open Source
Distributed Computing
company: www.visibletechnologies.com
blog: www.roadtofailure.com
twitter: @lusciouspear

2. Social Media and Scaling

3. Social Media and Scaling
•Scalability Matters Now.

4. Social Media and Scaling
•Scalability Matters Now.
•SM produces large, complex data

5. Social Media and Scaling
•Scalability Matters Now.
•SM produces large, complex data
•Anyone can collect the web

6. Social Media and Scaling
•Scalability Matters Now.
•SM produces large, complex data
•Anyone can collect the web
•Make a Twitter in a few days

7. Social Media and Scaling
•Scalability Matters Now.
•SM produces large, complex data
•Anyone can collect the web
•Make a Twitter in a few days
•Easy to get TBs of data

8. Social Media and Scaling
•Scalability Matters Now.
•SM produces large, complex data
•Anyone can collect the web
•Make a Twitter in a few days
•Easy to get TBs of data
•Big Data enabling new fields for
companies

9. What Visible Does

10. What Visible Does
•BI and Brand Management on Social
Media

11. What Visible Does
•BI and Brand Management on Social
Media
•Listen, Monitor, Engage

15. Old Product: RDBMS

16. Old Product: RDBMS
•A few MSSQL servers on boxes

17. Old Product: RDBMS
•A few MSSQL servers on boxes
•Lots of ETL

18. Old Product: RDBMS
•A few MSSQL servers on boxes
•Lots of ETL
•Several TB, inserts slow, deletes
impossible, random fail

19. Why RDBMS Bad

20. Why RDBMS Bad
•Nonlinear scale cost

21. Why RDBMS Bad
•Nonlinear scale cost
•Used as a storage abstraction

22. Why RDBMS Bad
•Nonlinear scale cost
•Used as a storage abstraction
•Mainly Select, Join, Group, Count

23. Why RDBMS Bad
•Nonlinear scale cost
•Used as a storage abstraction
•Mainly Select, Join, Group, Count
•Specialized Scale-Out ones ‘meh’

24. Why RDBMS Bad
•Nonlinear scale cost
•Used as a storage abstraction
•Mainly Select, Join, Group, Count
•Specialized Scale-Out ones ‘meh’
•Impedance Mismatch - Try to be High-
Throughput, Low-Latency

25. Why RDBMS Bad
•Nonlinear scale cost
•Used as a storage abstraction
•Mainly Select, Join, Group, Count
•Specialized Scale-Out ones ‘meh’
•Impedance Mismatch - Try to be High-
Throughput, Low-Latency
•Swiss-army knife, unstable,
transactions, advanced SQL, tuning

26. Why OSS?

27. Why OSS?
•Previously all MS

28. Why OSS?
•Previously all MS
•It exists!

29. Why OSS?
•Previously all MS
•It exists!
•Scaling + Licensing = No

30. Why OSS?
•Previously all MS
•It exists!
•Scaling + Licensing = No
•Can’t build a platform without source

31. Why OSS?
•Previously all MS
•It exists!
•Scaling + Licensing = No
•Can’t build a platform without source
•It’s Enterprise Now!

32. Goals for New Platform

33. Goals for New Platform
•“Golden Timeline”

34. Goals for New Platform
•“Golden Timeline”
•Search/Analyze *any* data

35. Goals for New Platform
•“Golden Timeline”
•Search/Analyze *any* data
•Linear Cost

36. Goals for New Platform
•“Golden Timeline”
•Search/Analyze *any* data
•Linear Cost
•Not Hacked Together

37. Goals for New Platform
•“Golden Timeline”
•Search/Analyze *any* data
•Linear Cost
•Not Hacked Together
•“Collect the Social Internet”

38. HOW TO SCALE

39. HOW TO SCALE
•What makes you special?

40. HOW TO SCALE
•What makes you special?
•What are you willing to sacrifice?

41. HOW TO SCALE
•What makes you special?
•What are you willing to sacrifice?
•How will you structure the data?

42. Avoiding Impedance Mismatch

43. Avoiding Impedance Mismatch
•Most problems can be divided into
High or Low latency

44. Avoiding Impedance Mismatch
•Most problems can be divided into
High or Low latency
•Get a lot of data eventually, or a little
now

45. Avoiding Impedance Mismatch
•Most problems can be divided into
High or Low latency
•Get a lot of data eventually, or a little
now
•MapReduce vs. Sharding/Indexing

46. Ecosystem
Compiled
Pig Cascading Hive
Processing
Katta / Applications
Raw
Zookeeper
MapReduce
Processing
Structured
HBase
Storage
Unstructured
Hadoop DFS
Storage

47. Simple Workflow
Semantic Unstructured
Hadoop Collect
Analysis Analysis
Structured
Analysis
Hadoop + Store in
HBase HBase
Store in
Indexing
Hadoop
Lucene+ Load/
Pull
Solr+ Replicate
Indexes
Katta Shards Search

48. Unstructured Processing Cluster
Semantic Unstructured Structured
Internet Collect Store
Analysis Analysis
HBase
HTML XML
Records

49. Hadoop + MR

50. Hadoop + MR
•Special: Crunch web-scale data fast

51. Hadoop + MR
•Special: Crunch web-scale data fast
•Sacrifice: Low-Latency, Transactions,
Random Access, Updates

52. Hadoop + MR
•Special: Crunch web-scale data fast
•Sacrifice: Low-Latency, Transactions,
Random Access, Updates
•Structure: Chunked flat files

53. Structured Processing Cluster
Enriched Data
Structured
Analysis
Unstructured Store in
Cluster HBase
Store in Search
Indexing
Hadoop Cluster
HBase
Records
Sharded
Lucene Index
Lucene Index

54. Document Structure
ContentID: 00BAC189
Title: Iron Maiden Rules
Body: I think Janick Gers is an amazing guitarist blah blah
PostDT: 20090718
ParentID: 0FDEADBEEF
Permalink: www.roadtofailure.com/post?=20

55. HBase

56. HBase
•Special: Scalable random/sequential
access almost as fast as RDBMS

57. HBase
•Special: Scalable random/sequential
access almost as fast as RDBMS
•Sacrifice: Joins, Secondary Indexes,
Transactions (kind of)

58. HBase
•Special: Scalable random/sequential
access almost as fast as RDBMS
•Sacrifice: Joins, Secondary Indexes,
Transactions (kind of)
•Structure: BigTable - column oriented

59. Search Cluster
Lucene Load/
Pull
Indexes from Replicate
Indexes
HDFS Shards Search
Lucene Lucene
Indexes Indexes

60. Search

61. Katta + Solr

62. Katta + Solr
•Special: Sharded search

63. Katta + Solr
•Special: Sharded search
•Sacrifice: Consistency, high-throughput

64. Katta + Solr
•Special: Sharded search
•Sacrifice: Consistency, high-throughput
•Structure: Reverse index

65. BI

66. BI
•Group, Sort, Filter, Count, Sum

67. BI
•Group, Sort, Filter, Count, Sum
•Semi-additive (Avg) rare but not hard

68. BI
•Group, Sort, Filter, Count, Sum
•Semi-additive (Avg) rare but not hard
•MapReduce Jobs

69. BI
•Group, Sort, Filter, Count, Sum
•Semi-additive (Avg) rare but not hard
•MapReduce Jobs
•Faceted Search

70. Examples

73. Challenges

74. Challenges
•Scaling Search

75. Challenges
•Scaling Search
•Understanding Latency

76. Challenges
•Scaling Search
•Understanding Latency
•What do we need ‘now’? Can
customers wait for big data?

77. Challenges
•Scaling Search
•Understanding Latency
•What do we need ‘now’? Can
customers wait for big data?
•Monitoring

78. Recap: Rules for Scaling

79. Recap: Rules for Scaling
•RDBMS is not a Swiss-Army Knife

80. Recap: Rules for Scaling
•RDBMS is not a Swiss-Army Knife
•Know your sacrifices

81. Recap: Rules for Scaling
•RDBMS is not a Swiss-Army Knife
•Know your sacrifices
•Know your specialness

82. Recap: Rules for Scaling
•RDBMS is not a Swiss-Army Knife
•Know your sacrifices
•Know your specialness
•Know your data structure

83. Recap: Rules for Scaling
•RDBMS is not a Swiss-Army Knife
•Know your sacrifices
•Know your specialness
•Know your data structure
•Ponder Latency

84. What Next?

85. What Next?
•HBase Analytics?

86. What Next?
•HBase Analytics?
•“What would make a bank trust it”

87. What Next?
•HBase Analytics?
•“What would make a bank trust it”
•Teach people to think about data

88. ...

89. The End
company: www.visibletechnologies.com
blog: www.roadtofailure.com
twitter: @lusciouspear
bradfordstephens@gmail.com