Big Data Airline Project at UAEU

1. Big Data
Airlines Project
ZIYAD SALEH

2. What is Big Data ?
Big data is a broad term for very large or complex data sets that are
difficult to process using traditional data processing applications.
Big Data is Terra bytes (1024 GB) of data to be processed and
analyzed, terra bytes of new data is being generated daily, which
means the speed of analyzing this huge flow of data is a challenge.
Big data can be described by the 4 Vs which are: Volume, Velocity,
Variety and Veracity.

6. Small Data Vs. Big Data

9. Map Reduce

10. Map Reduce model

12. Project Scope

13. The Scope is limited to :
1. Installing and configuring Hadoop Map/Reduce
platform.
2. Analyzing a big data sample belonging to U.S
domestic flights performance and delay for 5 years
to try to figure out
1. Top carriers experiencing delays.
2. Top airports and states with departure delays.
3. Plotting state delay in a thematic map of USA

14. Source of Data for the project
Datasets will be collected
from :
U.S. Department of
Transportation's (DOT) –
Statistical Computing

15. Dataset size will be between 500 GB and 1 TB and
covering 5 years of flight statistics.
Size of Data

16. Field Name Description
Year Year of the scheduled flight
Month Month of the scheduled flight (1–12).
Day Day of the month (1–31).
DepTime Actual departure time of the flight
CRSDepTime Scheduled departure time
ArrTime Actual arrival time in HH/MM format
CRSArrTime Scheduled arrival time
FlightNum Flight number.
ArrDelay Arrival delay
DepDelay departure delay, in minutes
CarrierDelay Delay (in minutes) caused by factors within control of the carrier.
WeatherDelay Delay (in minutes) caused by extreme weather conditions
NASDelay Delay (in minutes) within the control of the National Airspace System (NAS)
SecurityDelay Security delay (in minutes) caused by security reasons
LateAircraftDelay Delay (in minutes) due to the same aircraft arriving late at a previous airport.
Table 1 : Airline Dataset Dictionary.

17. Data Pre-Processing , Processing and Analytics
Data pre-processing:
Data will be cleansed and some artifacts will be filtered out as
necessary. Many fields in the airline data set need to be discarded as
they are irrelevant to the subject of delay that we are concerned on.
Data Processing and Analytics :
Data will be processed using java programming on Map/Reduce to
reduce the size of the data and produce an organized smaller
datasets.
Next, the resulting datasets will be analyzed using additional tools
like R.

18. Data Storage
Data will be stored in the HDFS multiple
storage nodes with total size between
500 GB and 1 TB.
Airlines
Big Data
HDFS

19. Target Analysis:
During the 5 years of all US domestic airlines flight
information
1. Which carriers have the most aggregated
delay in their flights ?
2. What are the states with most delays. ) ?

20. Design

21. Airlines Project Workflow and Design
Master Node Node 1
Node 2
Node 3
Node 4
Name
Node
Job
Tracker
Airlines
Big Data
Task
Java Code
Reducer Node
HDFS
Mapper
Reducer
Top Airlines

22. Implementation

23. Software and Tools
1. CentOS Linux Operating System.
2. Apache Hadoop
3. Cloudera CDH 5.3 virtual machine
4. Oracle VM Virtual Box Manager
5. Eclipse IDE
6. Java (Oracle JDK )
7. Maven
8. Microsoft Excel and Access 2010.
9. The R statistical tool

24. Mapper :

25. Reducer:

26. R:

27. Findings

28. US Airlines Delay (Per Carrier)
0
0.2
0.4
0.6
0.8
1
1.2
WN AA OO MQ US DL UA XE NW CO EV 9E FL YV OH B6 AS F9 HA AQ PI HP EA PS TW
ArrivalOnTime
ArrivalDelays
DepartureOnTime
DepartureDelays
Cancellations
Diversions

29. Thematic Map of US Airlines Delay (Per State)

30. Conclusion

31. Conclusion:
 Big Data is the large amount of continuously generated data that cannot be processed and
analyzed using traditional data management tools .
 Big data is a new topic that is rising dramatically , reshaping the future , and a large demand
for big data scientist is taking place and will continue to happen during the coming period of
time.
 Hadoop is an open source framework for storing and processing large datasets using clusters
of commodity hardware.
 Big Data analytics is attracting both business and policy makers to leverage from this new
phenomenon towards more informed decisions and planning for the future.
 Big Data now , Normal Data tomorrow.

32. Big Data Tutorials

33. Online Big Data Tutorials:
1. Udemy : https://www.udemy.com/course/subscribe/?courseId=336982&dtcode=lGCe31035ujY
2. Udacity : https://www.udacity.com/courses#!/data-science
3. EMC : https://education.emc.com/guest/campaign/data_science.aspx
4. Coursera : https://www.coursera.org/course/datasci
5. CalTech’s : Learning from Data http://work.caltech.edu/telecourse.html
6. MIT : Open Courseware http://ocw.mit.edu/courses/sloan-school-of-management/15-062-
data-mining-spring-2003/index.htm
7. Stanford’s OpenClassroom
http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=MachineLearning
8. Big Data University : https://bigdatauniversity.com/curriculum-map/

34. Thank You
Ziyad Saleh
34
‫ينفعنا‬ ‫ما‬ ‫علمنا‬ ‫اللهم‬..‫علمتنا‬ ‫بما‬ ‫وانفعنا‬
‫علما‬ ‫وزدنا‬