Hive 3 a new horizon

5. © Hortonworks Inc. 2011- 2018. All rights reserved | 5 Self-service question #1: Why is my query slow? Noisy neighbors Poor schema Inefficient queries Unstable demand Smart query log search Storage Optimizations Query Optimizations Demand Shifting Hortonworks Data Analytics Studio

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved DAS 1.0 Smart query log search ⬢ Query log reports: Most expensive queries, Long running queries, Hot files/tables, Space usage by table etc. ⬢ Query log filter/search: Tables not using statistics, queries not optimized by CBO etc. Storage optimizations ⬢ Storage heatmap and data layout optimization suggestions ⬢ Storage level optimization recommendations ⬢ Batch operations Query optimizations ⬢ Query level optimization recommendations ⬢ Detailed query level report ⬢ Admin alerts on expensive queries Quality of life changes ⬢ Query editor auto complete ⬢ Data browser (Top 20 rows sample data) ⬢ Specify output destination (S3, CVS etc.) ⬢ Query kill

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved One of the Extensible DataPlane Services ⬢ DAS 1.0 available now for HDP 3.0! ⬢ Monthly release cadence ⬢ Replaces Hive & Tez Views ⬢ Separate install from stack Hortonworks Data Analytics Studio HORTONWORKS DATAPLANE SERVICE DATA SOURCE INTEGRATION DATA SERVICES CATALOG …DATA LIFECYCLE MANAGER DATA STEWARD STUDIO +OTHER (partner) SECURITY CONTROLS CORE CAPABILITIES MULTIPLE CLUSTERS AND SOURCES MULTIHYBRID *not yet available, coming soon EXTENSIBLE SERVICES IBM DSX* DATA ANALYTICS STUDIO

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved Hive3: EDW analyst pipeline Tableau BI systems Materialized view Surrogate key Constraints Query Result Cache Workload management • Results return from HDFS/cache directly • Reduce load from repetitive queries • Allows more queries to be run in parallel • Reduce resource starvation in large clusters • Active/Passive HA • More “tools” for optimizer to use • More ”tools” for DBAs to tune/optimize • Invisible tuning of DB from users’ perspective • ACID v2 is as fast as regular tables • Hive 3 is optimized for S3/WASB/GCP • Support for JDBC/Kafka/Druid out of the box ACID v2 Cloud Storage Connectors

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Hive3: EDW analyst pipeline Tableau BI systems Materialized view Surrogate key Constraints Query Result Cache Workload management • Results return from HDFS/cache directly • Reduce load from repetitive queries • Allows more queries to be run in parallel • Reduce resource starvation in large clusters • Active/Passive HA • More “tools” for optimizer to use • More ”tools” for DBAs to tune/optimize • Invisible tuning of DB from users’ perspective • ACID v2 is as fast as regular tables • Hive 3 is optimized for S3/WASB/GCP • Support for JDBC/Kafka/Druid out of the box ACID v2 Cloud Storage Connectors

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Hive-1010: Information schema & sysdb Question: Find which tables have a column with ‘ssn’ as part of the column name? use information_schema; SELECT table_schema, table_name FROM information_schema.columns WHERE column_name LIKE '%ssn%'; Question: Find the biggest tables in the system. use sys; SELECT tbl_name, total_size FROM table_stats_view v, tbls t WHERE t.tbl_id = v.tbl_id ORDER BY cast(v.total_size as int) DESC LIMIT 3;

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved HIVE-1555: JDBC connector • How did we build the information_schema? • We mapped the metastore into Hive’s table space! • Uses Hive-JDBC connector • Read-only for now • Supports automatic pushdown of full subqueries • Cost-based optimizer decides part of query runs in RDBMS versus Hive • Joins, aggregates, filters, projections, etc

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved JDBC Table mapping example CREATE TABLE postgres_table ( id INT, name varchar ); CREATE EXTERNAL TABLE hive_table ( id INT, name STRING ) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES ( "hive.sql.database.type" = "POSTGRES", "hive.sql.jdbc.driver"="org.postgresql.Driver", "hive.sql.jdbc.url"="jdbc:postgresql://...", "hive.sql.dbcp.username"="jdbctest", "hive.sql.dbcp.password"="", "hive.sql.query"="select * from postgres_table", "hive.sql.column.mapping" = "id=ID, name=NAME", "hive.jdbc.update.on.duplicate" = "true" ); In Postgres In Hive

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved Druid Connector - Joins between Hive and realtime data in Druid Bloom filter pushdown greatly reduces data transfer Send promotional email to all customers from CA who purchased more than 1000$ worth of merchandise today. create external table sales(`__time` timestamp, quantity int, sales_price double,customer_id bigint, item_id int, store_id int) stored by 'org.apache.hadoop.hive.druid.DruidStorageHandler' tblproperties ( "kafka.bootstrap.servers" = "localhost:9092", "kafka.topic" = "sales-topic", "druid.kafka.ingestion.maxRowsInMemory" = "5"); create table customers (customer_id bigint, first_name string, last_name string, email string, state string); select email from customers join sales using customer_id where to_date(sales.__time) = date ‘2018-09-06’ and quantity * sales_price > 1000 and customers.state = ‘CA’;

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved Kafka connector Transformation over stream in real time I want to have moving average over sliding window in kafka from stock ticker kafka stream. create external table tickers (`__time` timestamp , stock_id bigint, stock_sym varchar(4), price decimal (10,2), exhange_id int) stored by 'org.apache.hadoop.hive.kafka.KafkaStorageHandler’ tblproperties ("kafka.topic" = "stock-topic", "kafka.bootstrap.servers"="localhost:9092", "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe"); create external table moving_avg (`__time` timestamp , stock_id bigint, avg_price decimal (10,2) stored by 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' tblproperties ("kafka.topic" = "averages-topic", "kafka.bootstrap.servers"="localhost:9092", "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe"); Insert into table moving_avg select CURRENT_TIMESTAMP, stock_id, avg(price) group by stock_id, from tickers where __timestamp > to_unix_timestamp(CURRENT_TIMESTAMP - 5 minutes) * 1000

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved Managed and External Tables • Hive 3 cleans up semantics of managed and external tables • External: Outside control and management of data • Managed: Fully under Hive control, ACID only • Non-native tables are external • ACID: Full IUD on ORC, Insert-only on other formats • Defaults have changed • Managed: ORC + ACID • External: TextFile • Two tablespaces with different permissions & ownership

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved Differences between external and managed tables • Storage based auth (doAs=true) is supported for external tables • Ranger and SBA can co-exist in HDP 3 (Ranger is default) • Script to convert from file permissions to Ranger policies on tables Note: SBA in HDP 3 requires ACL in HDFS. ACL is turned on by default in HDP3 Hive managed table ACID on by default No SBA, Ranger auth only Statistics and other optimizations apply Spark access via HiveWarehouseConnector External tables No ACID, Text by default SBA possible Some optimizations unavailable Spark direct file access

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved ACID v2 V1: CREATE TABLE hello_acid (load_date date, key int, value int) CLUSTERED BY(key) INTO 3 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional'='true'); V2: CREATE TABLE hello_acid_v2 (load_date date, key int, value int); • Performance just as good as non-ACID tables • No bucketing required • Fully compatible with native cloud storage

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved Materialized view Optimizing workloads and queries without changing the SQL SELECT distinct dest,origin FROM flights; SELECT origin, count(*) FROM flights GROUP BY origin HAVING origin = ‘OAK’; CREATE MATERIALIZED VIEW flight_agg AS SELECT dest,origin,count(*) FROM flights GROUP BY dest,origin;

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved Materialized view - Maintenance • Partial table rewrites are supported • Typical: Denormalize last month of data only • Rewrite engine will produce union of latest and historical data • Updates to base tables • Invalidates views, but • Can choose to allow stale views (max staleness) for performance • Can partial match views and compute delta after updates • Incremental updates • Common classes of views allow for incremental updates • Others need full refresh

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved Constraints & defaults • Helps optimizer to produce better plans • BI tool integrations • Data Integrity • hive.constraint.notnull.enforce = true • SQL compatibility & offload scenarios Example: CREATE TABLE Persons ( ID Int NOT NULL, Name String NOT NULL, Age Int, Creator String DEFAULT CURRENT_USER(), CreateDate Date DEFAULT CURRENT_DATE(), PRIMARY KEY (ID) DISABLE NOVALIDATE ); CREATE TABLE BusinessUnit ( ID Int NOT NULL, Head Int NOT NULL, Creator String DEFAULT CURRENT_USER(), CreateDate Date DEFAULT CURRENT_DATE(), PRIMARY KEY (ID) DISABLE NOVALIDATE, CONSTRAINT fk FOREIGN KEY (Head) REFERENCES Persons(ID) DISABLE NOVALIDATE );

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved Default clause & Surrogate keys • Multiple approaches • Sequence number is dense & increasing, but: Bottleneck in distributed DBMS • UUID is easy & distributable, but large and slow • Surrogate key UDF is easy & distributable & fast, but: No sequence and has gaps CREATE TABLE AIRLINES_V2 (ID BIGINT DEFAULT SURROGATE_KEY(24,16,24), CODE STRING, DESCRIPTION STRING, PRIMARY KEY (ID) DISABLE NOVALIDATE); INSERT INTO AIRLINES_V2 (CODE, DESCRIPTION) SELECT * FROM AIRLINES; ALTER TABLE FLIGHTS ADD COLUMNS (carrier_sk BIGINT); MERGE INTO FLIGHTS f USING AIRLINES_V2 a ON f.uniquecarrier = a.code WHEN MATCHED THEN UPDATE SET carrier_sk = a.id;

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved ⬢ Solution ● Query fails because of stats estimation error ● Runtime sends observed statistics back to coordinator ● Statistics overrides are created at session, server or global level ● Query is replanned and resubmitted Hive-17626: Optimizer is learning from planning mistakes ⬢ Symptoms ● Memory exhaustion due to under provisioning ● Excessive runtime (future) ● Excessive spilling (future)

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved HIVE-17481: LLAP workload management ⬢ Effectively share LLAP cluster resources – Resource allocation per user policy; separate ETL and BI, etc. ⬢ Resources based guardrails – Protect against long running queries, high memory usage ⬢ Improved, query-aware scheduling – Scheduler is aware of query characteristics, types, etc. – Fragments easy to pre-empt compared to containers – Queries get guaranteed fractions of the cluster, but can use empty space

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved Common Triggers ● ELAPSED_TIME ● EXECUTION_TIME ● TOTAL_TASKS ● HDFS_BYTES_READ, HDFS_BYTES_WRITTEN ● CREATED FILES ● CREATED_DYNAMIC_PARTITIONS Example CREATE RESOURCE PLAN guardrail; CREATE TRIGGER guardrail.long_running WHEN EXECUTION_TIME > 2000 DO KILL; ALTER TRIGGER guardrail.long_running ADD TO UNMANAGED; ALTER RESOURCE PLAN guardrail ENABLE ACTIVATE; Guardrail Example

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved Resource plans example CREATE RESOURCE PLAN daytime; CREATE POOL daytime.bi WITH ALLOC_FRACTION=0.8, QUERY_PARALLELISM=5; CREATE POOL daytime.etl WITH ALLOC_FRACTION=0.2, QUERY_PARALLELISM=20; CREATE RULE downgrade IN daytime WHEN total_runtime > 3000 THEN MOVE etl; ADD RULE downgrade TO bi; CREATE APPLICATION MAPPING tableau in daytime TO bi; ALTER PLAN daytime SET default pool= etl; APPLY PLAN daytime; daytime bi: 80% etl: 20% Downgrade when total_runtime>3000

34. 34 © Hortonworks Inc. 2011–2018. All rights reserved HIVE-18513: Query result cache Returns results directly from storage (e.g. HDFS) without actually executing the query If the same query had ran before Important for dashboards, reports etc. where repetitive queries is common Without cache With cache

35. 35 © Hortonworks Inc. 2011–2018. All rights reserved HIVE-18513: Query result cache details • hive.query.results.cache.enabled=true (on by default) • Works only on hive managed tables • If you JOIN an external table with Hive managed table, Hive will fall back to executing the full query. Because Hive can’t know if external table data has changed • Works with ACID • That means if Hive table has been updated, the query will be rerun automatically • Is different from LLAP cache • LLAP cache is a data cache. That means multiple queries can benefit by avoiding reading from disk. Speeds up the read path. • Result cache effectively bypasses execution of query • Stored at /tmp/hive/__resultcache__/, default space is 2GB, LRU eviction • Tunable setting hive.query.results.cache.max.size (bytes)

36. 36 © Hortonworks Inc. 2011–2018. All rights reserved Metastore Cache • With query execution time being < 1 sec, compilation time starts to dominate • Metadata retrieval is often significant part of compilation time. Most of it is in RDBMS queries. • Cloud RDBMS As a Service is often slower, and frequent queries leads to throttling. • Metadata cache speeds compilation time by around 50% with onprem mysql. Significantly more improvement with cloud RDBMS. • Cache is consistent in single metastore setup, eventually consistent with HA setup. Consistent HA setup support is in the works.

38. 38 © Hortonworks Inc. 2011–2018. All rights reserved Hive 3 feature summary ⬢ EDW offload – Surrogate key and constraints – Information schema – Materialized views ⬢ Perf – Query result & metastore caches – LLAP workload management ⬢ Real-time capabilities with Kafka – Ingest in ACID tables – Instantly query using Druid ⬢ Unified SQL – JDBC connector – Druid connector – Spark-hive connector ⬢ Table types – ACID v2 and on by default – External v Managed ⬢ Cloud – AWS/GCP/Azure cloud storage natively supported now

40. 40 © Hortonworks Inc. 2011–2018. All rights reserved Hive is broadening its reach SQL over Hadoop • External tables only • No ACID • No LLAP • SBA OK (doAs=True) • Some perf penalty Hive as EDW • ACID • LLAP • doAs=False • Managed tables • Column-level security • Stats, better perf etc. External table/ Direct Hive Warehouse Connector Spark

41. 41 © Hortonworks Inc. 2011–2018. All rights reserved Why change and why change now? • Advancements: Today’s workloads require higher performance, better security, better ingest and update tooling. All of which requires Hive to take more control over things • Predictability: What is perceived to work today doesn’t actually work • Visible breakages are better than hidden, implied breakages • Innovation: HDP 3 is a major release where we have the opportunity to innovate on the architecture, next point is at least a few years out

42. 42 © Hortonworks Inc. 2011–2018. All rights reserved Hive’s features require more control • Interactive query at scale • Data caching and query result caching • LLAP Workload management • Materialized view • ACID everywhere, single system that supports data change, query, etc. • No need for bucketing • Now faster with ACID v2, on par performance with non ACID tables • Information schema, Kafka integration and many other goodies

43. 43 © Hortonworks Inc. 2011–2018. All rights reserved Features: Spark access to ACID & column security ⬢ Hive supports traditional ACID semantics – ORC with delta files to support low-latency writes – Compaction to prevent storage fragmentation – Custom readers to reconcile deltas on read ⬢ ACID tables use extended Metastore format ⬢ Spark doesn’t read/write ACID tables ⬢ Spark doesn’t use ACID Metastore format Support Spark reads/writes for ACID tables

44. 44 © Hortonworks Inc. 2011–2018. All rights reserved Features: Spark access to Ranger tables – Column-level access control – Column masking • “show only first four chars of string column” – Row-level access control • “show only rows WHERE …” Support Spark reads/writes for Ranger tables Ranger UI

45. 45 © Hortonworks Inc. 2011–2018. All rights reserved Driver MetaStore HiveServer+Tez LLAP DaemonsExecutors Spark Meta Hive Meta Executors LLAP Daemons ACID TablesX X Spark can’t read/write ACID tables Spark doesn’t use ACID Metastore format

46. 46 © Hortonworks Inc. 2011–2018. All rights reserved Driver MetaStore HiveServer2 LLAP DaemonsExecutors Spark Meta Hive Meta Executors LLAP Daemons Isolate Spark and Hive Catalogs/Tables Leverage connector for Spark <-> Hive Uses Apache Arrow for fast data transfer HWC HWC

47. 47 © Hortonworks Inc. 2011–2018. All rights reserved Connector WRITE API a) hive.executeUpdate(sql : String) : Bool • Create, Update, Alter, Insert, Merge, Delete, etc… b) df.write.format(HIVE_WAREHOUSE_CONNECTOR) • Write DataFrame using LOAD DATA INTO TABLE c) df.write.format(STREAM_TO_STREAM) • Write Streaming DataFrame using Hive-Streaming

48. 48 © Hortonworks Inc. 2011–2018. All rights reserved Driver MetaStore HiveServer2 LLAP DaemonsExecutors Spark Meta Hive Meta HWC (Thrift JDBC) Executors LLAP Daemons a) hive.executeUpdate(“INSERT INTO s SELECT * FROM t”) 1. Driver submits update op to HiveServer2 2. Process update through Tez and/or LLAP 3. HWC returns true on success 1 2 3

49. 49 © Hortonworks Inc. 2011–2018. All rights reserved Example: LOAD to Hive df.select("ws_sold_time_sk", "ws_ship_date_sk") .filter("ws_sold_time_sk > 80000") .write.format(HIVE_WAREHOUSE_CONNECTOR) .option("table", “my_acid_table”) .save()

50. 50 © Hortonworks Inc. 2011–2018. All rights reserved Driver MetaStore HiveServer2 LLAP DaemonsExecutors Spark Meta Hive Meta Executors LLAP Daemons b) df.write.format(HIVE_WAREHOUSE_CONNECTOR).save() 1.Driver launches DataWriter tasks 2.Tasks write ORC files 3.On commit, Driver executes LOAD DATA INTO TABLE HDFS /tmp 1 2 3 ACID Tables

51. 51 © Hortonworks Inc. 2011–2018. All rights reserved Example: Stream to Hive val df = spark.readStream.format("socket") ... .load() df.writeStream.format(STREAM_TO_STREAM) .option(“table”, “my_acid_table”) .start()

52. 52 © Hortonworks Inc. 2011–2018. All rights reserved Driver MetaStore HiveServer+Tez Executors Spark Meta Hive Meta Executors c) df.write.format(STREAM_TO_STREAM).start() 1.Driver launches DataWriter tasks 2.Tasks open Txns 3.Write rows to ACID tables in Tx ACID Tables 1 2 3

54. 54 © Hortonworks Inc. 2011–2018. All rights reserved • Ran all 99 TPCDS queries • Total query runtime have improved multifold in each release! Benchmark journey TPCDS 10TB scale on 10 node cluster HDP 2.5 Hive1 HDP 2.5 LLAP HDP 2.6 LLAP 25x 3x 2x HDP 3.0 LLAP 2016 20182017 ACID tables

55. 55 © Hortonworks Inc. 2011–2018. All rights reserved • Faster analytical queries with improved vectorization in HDP 3.0 • Vectorized execution of PTF, rollup and grouping sets. • Perf gain compared to HDP 2.6 • TPCDS query67 ~ 10x! • TPCDS query36 ~ 30x! • TPCDS query27 ~ 20x! OLAP Vectorization

56. 56 © Hortonworks Inc. 2011–2018. All rights reserved SELECT * FROM ( SELECT AVG(ss_list_price) B1_LP, COUNT(ss_list_price) B1_CNT , COUNT(DISTINCT ss_list_price) B1_CNTD FROM store_sales WHERE ss_quantity BETWEEN 0 AND 5 AND (ss_list_price BETWEEN 11 and 11+10 OR ss_coupon_amt BETWEEN 460 and 460+1000 OR ss_wholesale_cost BETWEEN 14 and 14+20)) B1, ( SELECT AVG(ss_list_price) B2_LP, COUNT(ss_list_price) B2_CNT , COUNT(DISTINCT ss_list_price) B2_CNTD FROM store_sales WHERE ss_quantity BETWEEN 6 AND 10 AND (ss_list_price BETWEEN 91 and 91+10 OR ss_coupon_amt BETWEEN 1430 and 1430+1000 OR ss_wholesale_cost BETWEEN 32 and 32+20)) B2, . . . LIMIT 100; TPCDS SQL query 28 joins 6 instances of store_sales table Shared scan - 4x improvement! RS RS RS RS RS Scan store_sales Combined OR’ed B1-B6 Filters B1 Filter B2 Filter B3 Filter B4 Filter B5 Filter Join

57. 57 © Hortonworks Inc. 2011–2018. All rights reserved • Dramatically improves performance of very selective joins • Builds a bloom filter from one side of join and filters rows from other side • Skips scan and further evaluation of rows that would not qualify the join Dynamic Semijoin Reduction - 7x improvement for q72 SELECT … FROM sales JOIN time ON sales.time_id = time.time_id WHERE time.year = 2014 AND time.quarter IN ('Q1', 'Q2’) Reduced scan on sales

60. 60 © Hortonworks Inc. 2011–2018. All rights reserved ⬢ Hive on Kubernetes solves: – Hive/LLAP side install (to main cluster) – Multiple versions of Hive – Multiple warehouse & compute instances – Dynamic configuration and secrets management – Stateful and work preserving restarts (cache) – Rolling restart for upgrades. Fast rollback to previous good state. Hive on Kubernetes (WIP) Kubernetes Hosting Environments AWS GCP Data OS CPU / MEMORY / STORAGE OPENSHIFTAZURE CLOUD PROVIDERS ON-PREM/ HYBRID DATA PLANE SERVICES Cluster Lifecycle Manager Data Analytics Studio (DAS) Organizational Services COMPUTE CLUSTER SHARED SERVICES Ranger Atlas Metastore Tiller API Server DAS Web Service Query Coordinators Query Executors Registry Blobstore Indexe r RDBMS Hive Server Long-running kubernetes cluster Inter-cluster communication Intra-cluster communication Ingress Controller or Load Balancer Internal Service Endpoint for ReplicaSet or StatefulSet Ephemeral kubernetes cluster

69. 69 © Hortonworks Inc. 2011–2018. All rights reserved Other changes • Hive-CLI is gone, beeline now connects to HS2 • Make sure Beeline has network connectivity to HS2 • Hive on MapReduce is gone • By default, MapReduce jobs on Hive will fail • There is a global option to let jobs automatically fallback to Tez. Good for moving a lot of jobs instead of manually one by one. • WebHCat is gone • Because it depends on Hive-CLI • We’ll bring it back in HDP 3.1 • Oozie hive action does not work, will be re-implemented in HDP 3.x to use HS2

70. 70 © Hortonworks Inc. 2011–2018. All rights reserved Workload management • Cluster is divided into query pools (optionally nested) • Resource plan can be switched without stopping queries, e.g. based on time of day • Queries are automatically routed to pools based on user name or app • Rules to kill, move, or deprioritized queries based on DFS usage, runtime, etc. • Docs here

Hive 3 a new horizon

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

Hive 3 a new horizon

Plus De Contenu Connexe

Diaporamas pour vous (20)

Similaire à Hive 3 a new horizon (20)

Plus par Artem Ervits (6)

Plus récents (20)