Sempre più Clienti stanno migrando i propri processi di analisi e manipolazione dei dati da Apache Hadoop e da strutture on-premise al cloud AWS con l’obiettivo di ridurre i costi e migliorare la disponibilità e le performance. AWS offre un’ampia gamma di servizi di analytics, incluse soluzioni per i processi batch, processi stream, Machine Learning e orchestrazione dei flussi di dati. In questa sessione approfondiremo le componenti chiave del tuo ambiente attuale e forniremo le best practice per migrare i carichi di lavoro ai prodotti analytics di AWS più opportuni. Presenteremo diversi servizi AWS come Amazon EMR, Amazon Athena, Amazon Redshift, Amazon Kinesis e altri.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data freedom, migrating
BigData workloads to AWS
Giorgio Nobile - Solutions Architect

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Deconstructing current big data environments
• Identifying challenges with on-premises or unmanaged architectures
• Migrating components to Amazon EMR and AWS analytics services
- Choosing the right engine for the job
- Building out an architecture
- Architecting for cost and scalability
- Security
• Customer migration stories

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deconstructing on premises
BigData environments

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Server rack 1
(20 nodes)
Server rack 2
(20 nodes)
Server rack N
(20 nodes)
Core
On premises Hadoop clusters
• A cluster of 1U machines
• Typically 12 Cores, 32/64 GB RAM,
and 6 - 8 TB of HDD ($3-4K)
• Networking switches and racks
• Open-source distribution of Hadoop
or a fixed licensing term by
commercial distributions
• Different node roles
• HDFS uses local disk and is sized for
3x data replication

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workload types running on the same cluster
• Large Scale ETL: Apache Spark, Apache Hive with Apache Tez, or Apache
Hadoop MapReduce
• Interactive Queries: Apache Impala, Spark SQL, Presto, Apache Phoenix
• Machine Learning and Data Science: Spark ML, Apache Mahout
• NoSQL: Apache HBase
• Stream Processing: Apache Kafka, Spark Streaming, Apache Flink, Apache
NiFi, Apache Storm
• Search: Elasticsearch, Apache Solr
• Job Submission: Client Edge Node, Apache Oozie
• Data warehouses like Pivotal Greenplum or Teradata

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security
• Authentication: Kerberos with local KDC or Active
Directory, LDAP integration, local user management,
Apache Knox
• Authorization: Open-source native authZ (i.e.,
HiveServer2 authZ or HDFS ACLs), Apache Ranger,
Apache Sentry
• Encryption: local disk encryption with LUKS, HDFS
transparent-data encryption, in-flight encryption for
each framework (i.e., Hadoop MapReduce encrypted
shuffle)
• Configuration: Different tools for management based
on vendor

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Role of a Hadoop administrator
• Management of the cluster (failures, hardware
replacement, restarting services, expanding cluster)
• Configuration management
• Tuning of specific jobs or hardware
• Managing development and test environments
• Backing up data and disaster recovery

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Swim lane of jobs
Over-utilized Under-utilized

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Identifying on premises challenges

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On premises: Over-utilization and idle capacity
• Tightly coupled compute and storage requires buying excess capacity
• Can be over-utilized during peak hours and under-utilized at other
times
• Results in high costs and low efficiency

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On premises: System management difficulties
• Managing distributed applications and availability
• Durable storage and disaster recovery
• Adding new frameworks and doing upgrades
• Multiple environments
• Need team to manage cluster and procure hardware

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migrating workloads to Amazon EMR

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Low Cost
Pay an hourly rate
Open-Source Variety
Latest versions of software
Managed
Spend less time monitoring
Secure
Easy-to-manage options
Flexible
Customize the cluster
Easy to Use
Launch a cluster in minutes
Why Amazon EMR?

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key migration and TCO considerations
• Decouple storage and compute with Amazon S3
• Transient clusters and auto scaling
• Deconstruct workloads and map to open-source tools
• Choosing instance types and EC2 Spot Instances

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Translate use cases to the right tools
- Low-latency SQL -> Athena or Presto or Amazon Redshift
- Data warehouse/Reporting -> Spark or Hive or Glue or Amazon Redshift
- Management and monitoring -> EMR console or Ganglia metrics
- HDFS -> Amazon S3
- Notebooks -> Zeppelin Notebook or Jupyter (via bootstrap action)
- Query console -> Athena or Hue
- Security -> Ranger (CF template) or HiveServer2 or IAM roles
Storage
S3 (EMRFS), HDFS
YARN
Cluster Resource Management
Batch
MapReduce
Interactive
Tez
In Memory
Spark
Applications
Hive, Pig, Spark SQL/Streaming/ML, Flink, Mahout, Sqoop
HBase/Phoenix
Presto
Athena
Streaming
Flink
Glue
Amazon Redshift

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Many storage layers to choose from
Amazon DynamoDB
Amazon RDS Amazon
Kinesis
Amazon Redshift
Amazon S3
Amazon EMR
Amazon Elasticsearch

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Decouple compute and storage by using Amazon S3 as your
data layer
HDFS
S3 is designed for 11
9’s of durability and is
massively scalable
EC2 Instance
Memory
Amazon S3
Amazon EMR
Amazon EMR
Intermediates
stored on local
disk or HDFS
Local

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HBase on Amazon S3 for scalable NoSQL

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 tips: partitions, compression, and file
formats
• Avoid key names in lexicographical order
• Improve throughput and S3 list performance
• Use hashing/random prefixes or reverse the date-time
• Compress data set to minimize bandwidth from S3 to EC2
• Make sure you use splittable compression or have each file be the
optimal size for parallelization on your cluster
• Columnar file formats like Parquet can give increased performance
on reads

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TCO – Transient or long-running clusters

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Options to submit jobs
Amazon EMR
Step API
Submit a Spark
application
Amazon EMR
AWS Data Pipeline
Airflow, Luigi, or other
schedulers on EC2
Create a pipeline
to schedule job
submission or create
complex workflows
AWS Lambda
Use AWS Lambda to
submit applications to
EMR Step API or directly
to Spark on your cluster
Use Oozie on your
cluster to build DAGs
of jobs

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance and hardware
• Transient or long running
• Instance types
• Cluster size
• Application settings
• File formats and Amazon S3
tuning
Master Node
r5.2xlarge
Slave Group - Core
c5.4xlarge
Slave Group – Task
m5.2xlarge (EC2 Spot)
Considerations

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On-cluster UIs to quickly tune workloads
Manage applications
SQL editor, Workflow designer,
Metastore browser
Notebooks
Design and execute
queries and workloads

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Spot for
task nodes
Up to 80%
off Amazon
EC2
On-Demand
pricing
On-Demand for
core nodes
Standard
Amazon EC2
pricing for
On-Demand
capacity
Meet SLA at predictable cost Exceed SLA at lower cost
Use Spot and Reserved Instances to lower costs

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Instance fleets for advanced Spot provisioning
Master Node Core Instance Fleet Task Instance Fleet
• Provision from a list of instance types with Spot and On-Demand
• Launch in the most optimal Availability Zone based on capacity/price
• Spot Block support

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lower costs with Auto Scaling
EMR Job Amazon Cloudwatch metric Amazon Cloudwatch Alarm
AWS Autoscaling PolicyAWS Autoscaling Activity

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security – Encryption

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security – Authentication and authorization
Tag: user = MyUserIAM user: MyUser
EMR role
EC2 role
SSH key

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security – Authentication and authorization
• LDAP for HiveServer2, Hue, Presto,
Zeppelin
• Kerberos for Spark, HBase, YARN,
Hive, and authenticated UIs
• EMRFS storage-based permissions
• SQL standards-based and storage-
based authorization
AWS Directory Service
Self-Managed Directory

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security – Authentication and authorization
• Plug-ins for Hive, HBase, YARN,
and HDFS
• Row-level authorization for
Hive (with data-masking)
• Full auditing capabilities with
embedded search
• Run Ranger on an edge node –
visit the AWS Big Data Blog
Apache Ranger

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security – Governance and auditing
• AWS CloudTrail for EMR APIs
• Custom AMIs
• S3 access logs for cluster S3 access
• YARN and application logs
• Ranger for UI for application level auditing

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer Examples

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FINRA: Migrating from on-prem to AWS
Petabytes of data generated
on-premises, brought to AWS,
and stored in Amazon S3
Thousands of analytical
queries performed on EMR and
Amazon Redshift.
Stringent security
requirements met by
leveraging VPC, VPN,
encryption at-rest and in-
transit, CloudTrail, and
database auditing
Flexible
Interactive
Queries
Predefined
Queries
Surveillance
Analytics
Data Management
Data Movement
Data Registration
Version Management
Amazon S3
Web Applications
Analysts; Regulators

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lower cost and higher scale than on-premises

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FINRA saved 60% by moving to HBase on EMR

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn
Models
ModelsImpressions
Clicks
Activities
Calibrate
Evaluate
Real
Time
Bidding
Amazon S3
ETL Attribution
Machine
Learning
Amazon S3Amazon Kinesis
• 2 petabytes processed daily
• 2 million bid decisions per second
• Runs 24 X 7 on 5 continents
• Thousands of ML models
trained per day

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
aws.amazon.com/emr
blogs.aws.amazon.com/bigdata