Scaling hadoopapplications

Scaling Hadoop Applications Milind Bhandarkar Distributed Data Systems Engineer [ [email_address] ]

About Me ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/tmartin/32010732/

Outline ,[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/86798786@N00/2202278303/

Importance of Scaling ,[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/palmdiscipline/2784992768/

Explosion of Data (Data-Rich Computing theme proposal. J. Campbell, et al, 2007) http://www.flickr.com/photos/28634332@N05/4128229979/

Hadoop ,[object Object],[object Object],[object Object],[object Object]

Scalability of Parallel Programs ,[object Object],[object Object],[object Object],http://www-03.ibm.com/innovation/us/watson/

Reduce Latency ,[object Object],[object Object],http://www.flickr.com/photos/adhe55/2560649325/

Increase Throughput ,[object Object],http://www.flickr.com/photos/mikebaird/3898801499/

Three Equations http://www.flickr.com/photos/ucumari/321343385/

Amdahl’s Law http://www.computer.org/portal/web/awards/amdahl

Multi-Phase Computations ,[object Object],[object Object],http://www.computer.org/portal/web/awards/amdahl

Amdahl’s Law, Restated http://www.computer.org/portal/web/awards/amdahl

MapReduce Workflow http://www.computer.org/portal/web/awards/amdahl

Little’s Law http://sloancf.mit.edu/photo/facstaff/little.gif

Message Cost Model http://www.flickr.com/photos/mattimattila/4485665105/

Message Granularity ,[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/philmciver/982442098/

Alpha-Beta ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Causes of Sublinear Scalability http://www.flickr.com/photos/sabertasche2/2609405532/

Sequential Bottlenecks ,[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/bogenfreund/556656621/

Load Imbalance ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/kelpatolog/176424797/

Over-Partitioning ,[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/infidelic/3238637031/

Critical Paths ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/cdevers/4619306252/

Algorithmic Overheads ,[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/sbprzd/140903299/

Synchronization ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/susan_d_p/2098849477/

Task Granularity ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/philmciver/982442098/

Hadoop Best Practices ,[object Object],[object Object],[object Object],[object Object],[object Object],http://www.flickr.com/photos/yaffamedia/1388280476/

Questions ? http://www.flickr.com/photos/horiavarlan/4273168957/

Scaling hadoopapplications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Scaling hadoopapplications

Similar to Scaling hadoopapplications (20)

Scaling hadoopapplications