NAVTEQ uses Cloudera Distribution including Apache Hadoop (CDH) and HBase with Cloudera Enterprise support to process and store location content data. With HBase and its distributed and column-oriented architecture, NAVTEQ is able to process large amounts of data in a scalable and cost-effective way.

1. Geo-based Content Processing using HBase Ravi Veeramachaneni NAVTEQ 1

3. Solution

4. Why HBase?

5. Challenges

6. Why Cloudera?2

8. Cost:Expensive Oracle license cost

9. Technology: Inherent limitations of RDBMS

10. Need for real-time data Availability

11. Content needs to be updated and delivered much faster than before

12. Unable to deliver better, richer, faster contextual content more efficiently

13. Need to support both customers with connected and disconnected devices

14. Need to support flexible functionality

15. Need to support both structured and unstructured data

16. Need to decouple content from the map

17. Need flexibility to quickly add new content providers

18. Need to support community input3

21. As it runs on top of Hadoop

22. HBase stores null values for free

23. Saves both disk space and disk IO time

24. HBase supports unstructured data through column families

25. Individual records are called cells

26. Cells are addressed with row key/column family/cell qualifier/timestamp tuple

27. HBase has built-in version management

28. HBase provides fast table scans for time ranges and fast key based lookups

29. Map Reduce data input

32. Providing for horizontal scalability (Hadoop/HBase)

33. Provide business rules management system to adapt to changing processing needs

34. Flexible business rules based on the supplier and content information

35. Flexible business flows and processing to meet SLA

36. Provide high value and high quality content to customers fast

37. Corroborate multiple sources to produce the best quality information

38. Utilize open source software whereeverapplicable

39. Use commercial support for open source solutions to have a piece of mind6 Our Approach for Content Processing Challenges

40. 7 Content Processing Overview Batch and Transactional Ingestion Supplier n Supplier 1 Supplier 2 Location ID Permanent ID Source & Blended Record Management PUBLISHING Permanent ID Location ID real-time, on-demand

41. 8 System Decomposition

43. 8-node VMWare Sandbox Cluster

44. Flaky, unstable, RS Failures

45. Switched to CDH

46. No Cloudera support

47. Early 2010, hbase 0.20.x (CDH2)

48. 10-node Physical Sandbox Cluster

49. Still had lot of challenges, RS Failures, META corruption

50. Cluster expanded significantly with multiple environments

51. Signed Cloudera support

52. No official Cloudera support for HBase

53. Late 2010, hbase 0.89 (CDH3b3)

54. Most stable than any other version in the past

55. Multiple teams at NAVTEQ exploring to use Hadoop/HBase

56. Official Cloudera supported version

57. Current

58. Waiting on moving to CDH3 official release

59. Looking into Hive, Oozie, Lucene/Solr integration, Cloudera Enterprise and few other

60. Several initiatives to use HBase9

62. Dual-quad core with HT

63. 64GB RAM (ECC)

64. 4x2TB (JBOD)

65. RHEL 5.4

66. 500+ CPU cores

67. 200+ TB configured Disk Capacity

68. Multiple environmentsDEVELOPMENT INTEGRATION STAGING PRODUCTION 10

70. Transition to Column-oriented or flat schema

71. Row-key design/implementation

73. Performance was horrible

74. Initially we had about 100 or so, now downed to 8

75. Compression

76. LZO didn’t work out well with CDH2, using default Block compression

77. Need to revisit with CDH311 Challenges/Lessons Learned

79. Everything in HBase stored as Byte Array with the exception of timestamp

80. AVRO didn’t work well – deserialization issue

81. Developed configurable serialization mechanism that uses JSON except Date type

82. Indexing

83. Were using ITHBase and IHBase from contrib – doesn’t work well

84. Redesigned schema without need for index

85. We still need it though

86. Cluster configuration

87. Configuration is the key

91. Number of open files (ulimit) to 32K or even higher

92. vm.swapiness to lower or 013 Challenges/Lessons Learned

94. Several tunable parameters (Hadoop, HBase, OS, JVM, Networking, Hardware)

95. Scalability

96. Interacting with transactional-based (interactive) systems from batch oriented system

97. Security

98. Introduced in CDH3b3 but in flux, need robust RBAC

99. Reliability

100. Name Node is SPOF

101. HBase is sensitiveRegion Server Failures 14 Challenges/Lessons Learned

103. Cluster configurations were reviewed and made some recommendations

104. Tickets resolved within the SLAs

105. Knowledgeable support team

106. Ability to have access to technology experts, if needed

107. With Cloudera support for almost a year

108. Our needs and demand increasing and looking towards enterprise support15 Why Cloudera?

110. Job management, backup, restore, user provisioning, general administrative tasks, etc.

111. Support for Secondary Indexes

112. Indexing and Searching (Lucene/Solr integration?)

113. HA support for Name Node

114. Need Data Replication for HA & DR

115. Best practices and supported material16 Features Needed

116. 17 Thank You Q & A