"雲上打造資料湖 (Data Lake)：智能化駕馭商機 (Level 300) 講師： Ivan Cheng, Solutions Architect, AWS 客戶分享：Stanley Huang, Trend Micro 趨勢科技"

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecting Data Lake in Cloud
Driving Insights and Adding Intelligence
Ivan Cheng (鄭志帆)
Solutions Architect | AWS
Stanley Huang (黃士展)
Sr. Manager | Trend Micro

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Answer new questions and move faster
Raymond, Head of ProductLim, Head of Finance
Which regions are the new product selling well?
What is the demand forecast by product category?
What is the social sentiment about our products?
How do quality issues impact cost of production?
Can I look at supplier performance over time?
How can we reduce our inventory holding costs?

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
To answer new questions quickly, we look to a
modern data architecture design
Massive upfront costs
Overprovisioned capacity
Long implementation times
Pay as you go, for what you use
Decoupled pipelines and engines
Experimentation platform
Ingest/
Collect
Consume/
visualize
Store Process/
analyze
1 4
0 9
5

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
ERP
Data analysts
DATA PIPELINES
Ingest/
Collect
Consume /
visualize
Store Process /
analyze
1 4
0 9
5

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
ERP
DATA PIPELINES
Data
Lake
expdp
Data Data analysts
Data Warehouse
Amazon Redshift
Direct Query
Amazon Athena
She asks for the SMALLEST amount of data to answer her questions.
If it isn’t good enough, she asks for another small slice to be loaded to the DATA LAKE

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Characteristics of a Data Lake
Future
Proof
Flexible
Access
Dive in
Anywhere
Collect
Anything

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
DATA PIPELINES
Data
Lake
He first looks to the DATA LAKE, and imports only the category data he needs
He imports JUST ENOUGH data to see if the market is responding to products.
Business users
Transactions
ERP
Social media
Data
Stream
Capture
Amazon
Kinesis
Events
Amazon
QuickSight
Data Warehouse
Amazon Redshift
Stream Data
Amazon
ElasticSearch

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common data pipeline configuration
Raw Data
Amazon S3
Highly decoupled configurations scale better, are more fault tolerant, and cost optimized
ETL (Hadoop)
Amazon EMR
Triggered Code
AWS Lambda
Staged Data
(Data Lake)
Amazon S3
ETL & Catalog Management
AWS Glue
Data Warehouse
Amazon Redshift
Triggered Code
AWS Lambda

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
ERP
Data analysts
Business users
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
Social media

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Direct relationship with consumers
Krzysztof, Data ScientistBala, Head of Marketing
What are our customer segments, based on usage?
Can predict user preference?
How are people using the product?
What kinds of people are in/decreasing usage?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
Data scientists
Business users
Connected
devices
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
Sandbox
ML / Analytics / DLWeb logs /
clickstream

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Innovate for new revenues - personalization and forecasting
Transactions
ERP
Data analysts
Data scientists
Business users
Connected
devices
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
ML / Analytics
Social media
Web logs /
clickstream

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest ServingData
sources
Modern data architecture
Real-time engagement and interactive customer experiences
Transactions
ERP
Data analysts
Data scientists
Business users
Engagement platformsConnected
devices
Automation / events
DATA PIPELINES
EVENT PIPELINES
Data
Event Action
Insights
Data
Lake
ML / Analytics
Predict /
Recommend
AI Services
Social media
Web logs /
clickstream

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stanley Huang 黃士展
Sr. Manager, Web Service Engineering (WSE), Consumer, R&D, Trend Micro
Data Lake Journey in
Trend Micro Consumer Products

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Founded
RD HQ
Revenue
Customers
1988, United States
Taipei, Taiwan
$1.2B USD
500,000+ businesses,
Millions of consumers
5800+ Employees in 50+ Countries
100% of the
top 10 automotive
companies.
45 of the top 50
global
corporations.
100% of the
top 10 telecom
companies.
100% of the top
10 banks.
100% of the top
10 oil companies.
A world safe for exchanging digital information

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep Security

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Lake for Consumer Products

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Home Network Security

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business Requirement
Real-time
Dashboard
Business Metrics Feature
Comparison

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture 1.0
Analyze & ConsumeCatalog & TransformIngest & StoreData
Source
Amazon
Redshift
AmazonKinesis
Streams
Lambda
function
Lambda
function
Lambda
Amazon EMR
Amazon S3
Transformed Data
Amazon S3
Raw Data
application
backend
Tableau
(Data Visualization)
Lambda
Amazon
Pinpoint
mobile app
AmazonKinesis
Streams

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Timely Updated Business Metrics
Dashboard Business Metrics
Original image source: https://www.interaction-design.org/literature/article/google-s-heart-framework-for-measuring-ux

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Feature Comparison

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Frequent Release Cycles of the Products
High Availability Data Quality Auto-recovery

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build-Measure-Learn Feedback Loop
Now2017, Sep.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture 2.0
Analyze & ConsumeCatalog & TransformIngest & StoreData
Source
Amazon
Redshift
AmazonKinesis
Streams
Lambda
function
Lambda
function
Lambda
Amazon EMR
Amazon S3
Transformed Data
Amazon S3
Raw Data
application
backend
Amazon SQS
Task queue Lambda
function
Tableau
(Data Visualization)
Amazon SNS
Notification
Lambda
Amazon
Pinpoint
mobile app
AmazonKinesis
Streams

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Culture of Data Driven
Exploratory
Data Analysis
Self-service
ETL
Machine Learning
Platform

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture 3.0
Analyze & ConsumeCatalog & TransformIngest & StoreData
Source
Amazon
Redshift
Amazon
Pinpoint
AmazonKinesis
Streams
Lambda
function
Lambda
function
Lambda
Amazon EMR
Amazon S3
Transformed Data
Amazon S3
Raw Data
mobile app
Amazon EMR
application
backend
Amazon S3
Enriched
Data
Redshift
Spectrum
Amazon
Athena
Glue Catalog Glue Catalog
Amazon SQS
Task queue Lambda
function
Tableau
(Data Visualization)
Downstream
Service
Amazon SNS
Notification
Lambda
AmazonKinesis
Streams
Amazon SageMaker

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Feature Engineering for Machine Learning

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Focus on Customer’s Value
• Secure and anonymize data
• Real-time Dashboard
• Work with PM and product team at early stage
• Design data pipelines to fit the dashboard
• Frequent Release Cycles
• Store data in raw format
• Ensure data set could be rebuilt by event replaying
• Data-Driven Culture
• Open data to stakeholder who has domain knowledge
• Prepare technical documents for data and tools
Takeaway

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Central Storage
Secure, Cost Effective
Storage in S3
S3
Kinesis Direct Connect Snowball DMS
Data Ingestion
Get your data into S3
quickly and securely
Athena Quicksight EMR Redshift
Processing & Analytics
Use predictive and prescriptive
analytics to gain better understanding
Glue ETL
Protect & Secure
Use entitlements to ensure data is secure and users identities are verified
Security Token
Service
Cloudwatch Cloudtrail KMS
Catalog & Search
Access & Search Metadata
DynamoDB Amazon ESGlue Catalog
Access & User Interface
Give your users easy & secure access
API Gateway IAM Cognito

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ready to build better business from your ideas?
Short list projects that
directly impact
customer engagement
and adoption
Build simple data
pipelines that allow you
to test new ideas, and
fill your data lake
Ask our solution architects
and professional services
teams to help you build

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please complete the session survey
Thank you!