This document summarizes a meetup on big data technologies. It defines big data as large volumes of diverse data that are created and analyzed rapidly to extract value. The three V's of big data are described as volume, variety, and velocity. Examples of structured, semi-structured, and unstructured data sources are provided. Hadoop and MapReduce are introduced as key technologies for processing large datasets in a distributed manner across clusters of commodity servers. HDFS provides a fault-tolerant storage system and MapReduce allows for parallel processing of data.

1. ICTA Technology Meetup 11
By Crishantha Nanayakkara

2. Meetup Recap
●
1 – Enterprise Application Integration
●
2 – Enterprise Level High Availability Options
●
3 – SOA Security
●
4 – Towards Hybrid Mobile App Development
●
5 – The Semantic Web and Linked Data
●
6 – Enterprise Application Design Patterns
●
7 – GIS – An Introduction
●
8 – The Future of the Database World
●
9 – The Enterprise Storage Management
●
10 – An Introduction to Content Management with Joomla

3. The Scope
●
Big Data – The Definition
●
The Sources of Data
●
Structured, Semi­Structured vs Unstructured Data
●
Relational Data vs Big Data
●
Towards Big Data
●
Big Data Adoption in other countries
●
Big Data Technologies and the ecosystem
●
Big Data Open Source and Commercial Options

4. Big Data Definition

5. A new generation of technologies and
architectures, designed to economically
extract VALUE from very large VOLUMES of a
wide variety of data by enabling high­
VELOCITY capture, discovery, and/or analysis.

6. The Three Vs of Big Data

7. The Three Vs of Big Data
●
Volume – Big
●
Variety – From different sources and types
●
Velocity – Frequency of its generation: how
quickly the data arrives and is stored, and how
quickly it can be retrieved

8. The Sources of Data

9. The Sources of Data
●
Documents
●
Sensor Data
●
Emails
●
Click Streams
●
Images
●
Relational Databases
●
Logs
●
Social Media feeds
●
Videos

10. Structured, Semi Structured and
Unstructured Data

11. Structured Data
●
Structured:
–
The information with a high degree of
organization
–
Seamless and readily search­able by
straightforward search algorithms or
operations
–
e.g: relational databases, spreadsheets, XML

12. Semi­Structured Data
●
Semi­Structured:
–
This is a form of structured data that does not
conform to an explicit and fixed schema
–
The data is inherently self­describing and
contains tags or other markers to enforce
hierarchies of records and fields within the
data
–
e.g: web logs, social media feeds

13. Unstructured Data
●
Unstructured:
–
This type of data consists of formats which
cannot easily be indexed into relational tables
for analysis or querying
–
e.g.: images, videos

14. Relational Vs Big Data

15. Relational Data vs Big Data
●
●
●
Thinking of Big Data as “just lots more enterprise
data” is tempting, but it’s a serious mistake.
Big Data is commonly generated outside of traditional
enterprise applications
Big Data is often composed of unstructured or semi­
structured information types that continually arrive in
enormous amounts

16. Relational Data vs Big Data
●
To get maximum value from Big Data, it needs to be
associated with traditional enterprise data,
automatically or via purpose built applications,
reports, queries, and other approaches

17. Towards Big Data

18. The Digital Universe
●
From 2005 to 2020, the digital universe will grow from 130
exabytes to 40,000 exabytes, or 40 trillion gigabytes.

19. According to IDC, the Big Data technology and service market was about
US$4.8 billion in 2011. The market is projected to grow at a compound annual
growth rate (CAGR) of 37.2% between 2011 and 2015.
By 2015, the market size is expected to be US$16.9 billion.
[Source: IDC. Worldwide Big Data Technology and Services 2012­2015 Forecast.]

20. Gartner reported that more than 65 billion devices were connected
to the internet by 2010. By 2020, this number will go up to 230 billion
[Source: https://www.gartner.com/doc/1799626]

21. The Opportunity for Big Data
●
●
●
Only a tiny fraction of the digital universe has been
explored for analytic value so far.
By 2020, as much as 33% of the digital universe will
contain information that might be valuable if
analyzed.
But only if it is tagged and analyzed. That is the
opportunity for Big Data.
Source: IDC's Digital Universe Study, 2012

23. The Candidates for Big Data
●
Not all data is necessarily useful for Big Data
analytics. However, some data types are particularly
good for analysis
–
Surveillance Footage
–
Embedded medical devices
–
Entertainment and Social Media
–
Images and Voice Data
–
Data Processing
Source: IDC's Digital Universe Study, 2012

25. The Statistics
●
●
●
Over a history that spans more than 30 years, SQL
database servers have traditionally held gigabytes of
information — and reaching that milestone took a
long time.
In the past 15 years, data warehouses and enterprise
analytics expanded these volumes to terabytes.
And in the last 5 years, the distributed file systems
that store Big Data now routinely house petabytes of
information.

26. The Big Data Adoption in the
World

27. Source: http://www.informationweek.com/government/information-management/white-house-shares-200-million-big-data/232700522

28. http://www.informationweek.com/regulations/federal-standards-body-focuses-on-big-data-cloud/d/d-id/1102703?

29. Singapore Transport System
(Land Transport Authority ­ LTA)
Source: How Cities using Big Data in Asia? - FutureGov Report

30. Singapore Transport System
(Land Transport Authority ­ LTA)
●
Data Collection:
–
Junction Electronic Eyes
–
Green Link Determining System
–
Web cams
–
Parking Guidance Systems
–
Expressway monitoring Systems
–
Traffic Scan
Source: How Cities using Big Data in Asia? - FutureGov Report

31. Singapore Transport System
(Land Transport Authority ­ LTA)
●
Data Processing:
–
–
●
All the data is fed into this integrated i­
Transport Processing System
The data is aggregated, integrated and
analyzed
Data Dissemination:
–
Via web portals, radio broadcasting, navigation
devices, smart phones, etc
–
Certain data elements are given as “open data”
Source: How Cities using Big Data in Asia? - FutureGov Report

32. Singapore National Environment
Agency
●
Dengue related data:
–
The data is pulled from dengue cases, public
feedback, mosquito inspections and other
sources for analysis.
–
Making use of GIS to identify high­risk
areas,they are also able to prioritize places
for checks
Source: How Cities using Big Data in Asia? - FutureGov Report

33. iPlan Project
(Urban Redevelopment Authority ­ URA)
●
iPLAN is among the world’s first nationwide
enterprise GIS systems for urban planning and
it contains comprehensive land, building,
planning and approval information which is
readily available to URA’s planners
Source: How Cities using Big Data in Asia? - FutureGov Report

34. Kuala Lampur Government
●
The government has created a Big Data
Analytics fund to support four government­
initiated projects by 2015 focusing on,
–
Transport,
–
Planning, Environment and
–
Security
Source: How Cities using Big Data in Asia? - FutureGov Report

35. Technologies behind
Big Data

36. Reference: http://www.bdisys.com/27/1/17/BIG%20DATA/HADOOP

37. Hadoop

38. Hadoop – An Introduction
●
●
●
●
Hadoop is a framework that provides open source
libraries for distributed computing using MapReduce
software and its own distributed file system Hadoop
Distributed File System (HDFS)
Open Source, written in Java
Maintained by Apache Software Foundation as a top
level project
Original deployments
–
Yahoo, Facebook, LinkedIn

39. Hadoop – The Core Components
●
The kernal(core) of Hadoop provides:
–
–
●
A reliable shared storage (HDFS)
An Analysis system (MapReduce)
There are other components in Hadoop, which
makes a complete Hadoop ecosystem

40. Hadoop Architecture
●
●
●
Designed to scale out from a few computing nodes to
thousands of machines, each offering local computation
and storage
Leverages the power of massive parallel processing to
take advantage of Big Data, generally by using lots of
inexpensive commodity servers, which has a high
tolerance of hardware failure. In Hadoop, hardware
failure is taken as rule rather than an exception
Designed to abstract away much of the complexity of
distributed processing. This lets developers focus on the
task at hand

41. Hadoop Architecture
Reference: Hadoop In Action

42. Hadoop Architecture
p
Reference: Hadoop In Action

43. Hadoop Distributed File System
(HDFS)

44. Scale Up Vs Scale Out
Reference: http://quickfileaccounting.wordpress.com/2013/07/02/scale­out­vs­scale­up/

45. Scale Up Vs Scale Out

46. HDFS
●
●
●
A fault­tolerant storage system that can store huge
amounts of information
Scale up incrementally and survive storage failure
without losing data
Hadoop clusters are built with inexpensive computers.
If one computer (or node) fails, the cluster can
continue to operate without losing data or
interrupting work by simply re­distributing the work
to the remaining machines in the cluster

47. HDFS
●
●
●
HDFS manages storage on the cluster by breaking
files into small blocks and storing duplicated copies of
them across the pool of nodes
In the common case, HDFS stores three complete
copies of each file by copying each piece to three
different servers
If any two servers can fail, and the entire file will still
be available HDFS notices when a block or a node is
lost, and creates a new copy of missing data from the
replicas it manages.

48. HDFS
●
HDFS offers two key advantages over RAID:
–
It requires no special hardware, since it can be
built from commodity servers, and can
survive more kinds of failure – a disk, a node
on the network or a network interface

49. HDFS

50. Reference: Hadoop In Action

51. MapReduce

52. MapReduce
●
●
Hadoop takes advantage of HDFS’ data
distribution strategy to push work out to many
nodes in a cluster. This allows analyses to run
in parallel and eliminates the bottlenecks
imposed by monolithic storage systems.
Hadoop uses MapReduce for this task.

53. MapReduce

54. MapReduce
●
●
A new programming framework — created and
successfully deployed by Google — that uses the
divide­and­conquer method (and lots of commodity
servers) to break down complex Big Data problems
into small units of work, and then process them in
parallel
MapReduce is built on the proven concept of divide
and conquer: it’s much faster to break a massive task
into smaller chunks and process them in parallel.

55. MapReduce
●
MapReduce is a data processing algorithm that uses a
parallel programming implementation. In simple
terms, MapReduce is a programming paradigm that
involves distributing a task across multiple nodes
running a "map" function. The map function takes the
problem, splits it into sub parts and sends them to
different machines so that all the sub­parts can run
concurrently. The results from the parallel map
functions are collected and distributed to a set of
servers running "reduce" functions, which then takes
the results from the sub­parts and re­combines them
to get the single answer.

57. Source: http://www.youtube.com/watch?v=HFplUBeBhcM (MapR Demo)

58. Hadoop Eco­System

59. The Hadoop Eco­System
●
In addition to MapReduce and HDFS, Hadoop
also refers to a collection of other software
projects that uses the MapReduce and HDFS
framework
–
HBase
–
Hive
–
Pig
–
Mahout
–
Zookeeper
–
Sqoop

60. The Hadoop Eco­System
Reference: http://www.bdisys.com/27/1/17/BIG%20DATA/HADOOP

61. Apache Pig
●
●
●
This is a platform for analyzing large data sets
that consists of a high­level language for
expressing data analysis programs
Those who want to have a simple job tracking
with MapReduce, can use Apache Pig.
This can reduce the overhead of learning and
writing complex MapReduce jobs mainly in
Java Language

62. Apache Pig
Pig
Pig
MapReduce
MapReduce
HDFS
HDFS

63. Apache Hive
●
●
Those who like to use SQL like query
languages for job tracking with MapReduce
and whom does not like Apache Pig style of
coding can use Apache Hive
Hive manages data stored in HDFS and
provides a query language based on SQL (and
which is translated by the runtime engine to
MapReduce jobs) for querying the data

64. Apache Hive
Pig
Pig
Hive
Hive
MapReduce
MapReduce
HDFS
HDFS

65. Apache HBase
●
●
A distributed, column­oriented database.
HBase uses HDFS for its underlying storage,
and supports not only batch­style computations
real time queries (random reads) as well.
Facebook messages are using Apache Hbase as
the real time processing

66. Apache HBase
Pig
Pig
Hive
Hive
HBase
HBase
MapReduce
MapReduce
HDFS
HDFS

67. Apache ZooKeeper
●
●
●
A distributed, highly available coordination
service most of the components in the Hadoop
ecosystem.
It stores some of the metadata of the Apache
Hbase as well
ZooKeeper provides primitives such as
distributed locks that can be used for building
distributed applications.

68. Apache Zookeeper
Pig
Pig
Hive
Hive
ZooKeeper
ZooKeeper
HBase
HBase
MapReduce
MapReduce
HDFS
HDFS

69. Apache Sqoop
●
A tool for efficiently moving data between
relational databases and HDFS

70. Hadoop support for GIS
http://esri.github.io/gis-tools-for-hadoop/

71. Hadoop Distributions
●
Open Source:
–
●
Apache Hadoop
Commercial:
–
Cloudera
–
Hortonworks
–
MapR
–
AWS MapReduce
–
Microsoft HDInsight

72. NoSQL

73. References
●
●
●
●
Big Data Right Now: Five Trendy Open Source Technologies:
http://techcrunch.com/2012/10/27/big­data­right­now­five­trendy­
open­source­technologies/
An Introduction to NOSQL Data Management for Big Data:
http://data­informed.com/introduction­nosql­data­management­big­
data/
Overview of Big Data and NOSQL Technologies as of January 2013:
http://www.syoncloud.com/big_data_technology_overview
The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and
Biggest Growth in Far East, December 2012, EMC Corporation, [IDC
Report 2012]

74. References
●
●
●
●
●
White House Shares $200 Million Big Data Plan:
http://www.informationweek.com/government/information­
management/white­house­shares­200­million­big­data/232700522
Federal Standards Body Focuses On Big Data, Cloud:
http://www.informationweek.com/regulations/federal­standards­
body­focuses­on­big­data­cloud/d/d­id/1102703?
The Internet of Things Is Coming:
https://www.gartner.com/doc/1799626
What is Data Science?
http://radar.oreilly.com/2010/06/what­is­data­science.html
Google Flu Trends :
http://www.google.org/flutrends/about/how.html

75. References
●
●
●
●
Hadoop: The Definitive Guide, Second Edition, by Tom White.
Copyright 2011 Tom White, 978­1­449­38973­4
Hadoop In Action: by Chuck Lam, 2011, 978­1­935­18219­1
MapR Demo on Introduction to MapReduce:
http://www.youtube.com/watch?v=HFplUBeBhcM
Basic Introduction to Apache Hadoop by HortonWorks:
http://www.youtube.com/watch?v=OoEpfb6yga8