1. Analyzing Historical Data of
Applications on HadoopYARN:
for Fun and Profit
Mayank Bansal，Zhijie Shen

2. Agenda
• Who we are ?
• Why we need New History Server?
• Application History Server
• Timeline Server
• Future Work

3. Who we are
• Hadoop Architect @ ebay
• Apache Hadoop Committer
• Apache Oozie PMC and Committer
• Current
• Leading Hadoop Core Development for
YARN and MapReduce @ ebay
• Past
• Working on Scheduler / Resource
Managers
• Working on Distributed Systems
• Data Pipeline frameworks
Mayank Bansal

4. Who we are
• Software Engineer @ Hortonworks
• Apache Hadoop Committer
• Apache SAMZA PPMC and Committer
Zhijie Shen

5. Agenda
• Who we are ?
• Why we need New History Server?
• Application History Server
• Application Timeline Server
• Future Work

6. MR JobHistory Server
• We already have Job History Server
• It is only for Map Reduce Customized
• Storage is HDFS only
• Storage is very MR specific
• Counters
• Mappers and Reducers
• If you have only Map Reduce you are good.

7. Hadoop-2
Single Use System
Batch Apps
Multi Purpose Platform
Batch, Interactive, streaming

8. YARN

9. Issues with current Job History
• What if I have other Applications
• RM crashes
• Hard Limit on # Apps
• Upgrades / Updates

10. Agenda
• Who we are ?
• Why we need New History Server?
• Application History Server
• Timeline Server
• Future Work

11. Application History Server
• Separate Process
• Pluggable Storage
• HDFS
• In-Memory
• Resource Manager directly writes to Storage
• Aggregated Logs
• Separate UI, CLI and Rest End Point

12. Application History Server
Storage:
• It stores generic Data
• Application level data (queue, user etc…)
• List of ApplicationAttempts
• Information about each ApplicationAttempt
• List of containers for ApplicationAttempt
• Generic information about each container.

13. Application History Server

14. Application History Server
• CLI Interface
$ yarn application -status <Application ID>
$ yarn applicationattempt -list <Application ID>
• REST APIs
• http://localhost:8188/ws/v1/applicationhistory/app
s/appid

15. Application History Server
• Scalability for storage
• One file per application
• File format is protobuff
• Size of HDFS files
• Multiple RM threads writing to History Storage
# of
Containers
100 1K 10 K 100K
Size of the File 19 KB 184 KB 1.8 MB 19 MB

16. Agenda
• Who we are ?
• Why we need New History Server?
• Application History Server
• Timeline Server
• Future Work

17. Timeline Service - Motivation
• YARN takes care of it
– Relieving the application from monitoring service
• Application diversity
– Framework specific metadata/metrics

18. Timeline Service – Data Model
• Entity Type
– An abstract concept of anything
• Entity
– One specific instance of a entity type
– Defining the relationship between entities
• Event
– Something happens to an entity

19. Timeline Service – Architecture
• LevelDB Store
• Client Library
• REST Interfaces

20. Timeline Service – Store
• LevelDB based store
– Key-value store
– Lightweight
– License compatible
• Implementing reader/writer interfaces
• Support data retention

21. Timeline Service – Client
• TimelineClient
– Wrap over REST POST method
– POJO objects
• TimelineEntity
• TimelineEvent
– In Client/AM/Container

22. Timeline Service – APIs
• Rest APIs, JSON as the media
• Get timeline entities
– http://localhost:8188/ws/v1/timeline/{entityType}
• Get timeline entity
– http://localhost:8188/ws/v1/timeline/{entityType}/{entityId}
• Get timeline events
– http://localhost:8188/ws/v1/timeline/{entityType}/events

23. Timeline Service – Security
• HTTP SPNEGO
• Kerberos Authentication
• Delegation Token
– Performance
– AM/Container no Kerberos
• Access Control
– Admin/owner
– Timeline entity-level
Browser or
REST clientTimeline Client Timeline Client
Client AM
2. Pass DT via AppSubmissionContext
1. Get DT
Put Data via
Kerberos or
DTs
3. Put Data w/ DT
Timeline Authen ca on Filter
Timeline REST APIs
Timeline
ACLs
Manager
Access Verifica on
Pass
authen cated
user informa on
Get Data via
Kerberos
HadoopsupportforKerberos,SPNEGO,ACLs
Timeline Service security
Timeline Server

24. Timeline Service – Use Case (1)

25. Timeline Service – EarlyAdopter (2)

26. Timeline Service – Early Adopter (3)

27. Agenda
• Who we are ?
• Why we need New History Server?
• Application History Server
• Timeline Server
• Future Work

28. To Be Continue…
• Integrating the generic history and timeline data
• Rebasing MR Job history server on the timeline
server
• Making the timeline server rendering the timeline
data

29. To Be Continue…
Scale
• Leveldb does not handle ebay scale
• We need something which can horizontally scale
• HBASE

30. Questions
30
Mayank Bansal
mabansal@ebay.com
mayank@apache.org
Zhijie Shen
zshen@hortonworks.com
zjshen@apache.org