Given at http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/events/62000472/

1. Mozscape: NoSQL at Terabyte Scale
Phil Smith
Software Engineer

2. What We Do
SEO & Inbound Marketing Metrics
www.opensiteexplorer.org

3. What We Do
www.opensiteexplorer.org
Collect back links across the web

4. What We Do
www.opensiteexplorer.org
Collect back links across the web
Compute metrics estimating value

5. What We Do
www.opensiteexplorer.org
Collect back links across the web
Compute metrics estimating value
Serve links and metrics with API and OSE

6. How We Do
Crawl the Web
~25-30 billion pages per month

7. How We Do
Crawl the Web
~25-30 billion pages per month
20 Crawler machines

8. How We Do
Crawl the Web
~25-30 billion pages per month
20 Crawler machines
~256 MB/sec aggregate download rate

9. How We Do
Compute Aggregates and Metrics
1:5 to 1:50 Compression Ratios

10. How We Do
Compute Aggregates and Metrics
1:5 to 1:50 Compression Ratios
Aggregates are Parallelized Linear Scans

11. How We Do
Compute Aggregates and Metrics
1:5 to 1:50 Compression Ratios
Aggregates are Parallelized Linear Scans
Communication Avoided where Possible

12. How We Do
Surface with a Read-Only API
~12 TB per Release in Amazon S3

13. How We Do
Surface with a Read-Only API
~12 TB per Release in Amazon S3
6 m2.4xlarge Instances for Cache

14. How We Do
Surface with a Read-Only API
~12 TB per Release in Amazon S3
6 m2.4xlarge Instances for Cache
~28k Requests per Minute

15. Observations and Strategy
Billions of Small, Similar Records
De-normalization Avoids Complex Joins
Batch-style Emphasizes Spatial Locality

16. Data Layout
Column-Orientation exploits Locality
Broken into 5GB chunks for S3
~64KB Compression Runs within

17. Compression
Tuned to Overcome Disk Read Bound
By-Column, Run & Gap Encoding on LZO
Customized Pipelines per Column

18. Job Control
Each Stage has Parallel, Idempotent Tasks
Tasks are Procs with easy Command Line
stdout, exit code are logged to track state

19. Checkpoints
S3
Barrier Barrier
Table Scan Checkpoint Table Scan
Time

20. Indexing
Columns have BDBs indexing by ID
Subset of IDs map to Compression Runs
Decompress Run and Scan to find Record

21. Physical Deployment
Crawlers run in Colo for white-listed IPs
Batch Process and API layer in EC2
The API might be in a colo too, but
ELB + Autoscaling are nice

22. Questions?
We’re Hiring!