by John Pignata, Startup Solutions Architect, AWS Calling all serverless developers! Wild Rydes (www.wildrydes.com), the world’s leading unicorn transportation startup, needs your help! The company's rydesharing network of unicorns has grown to thousands, fulfilling hundreds of thousands of passenger rydes each day. Your mission is to collect, store, process, and analyze data to track the real-time location and health of our unicorns. In this workshop, learn how to build infrastructure to process data streams in real time using Amazon Kinesis. Build serverless applications using Amazon Kinesis Analytics to aggregate and summarize data and use AWS Lambda to store aggregated data in Amazon DynamoDB. Finally, use Amazon Kinesis Firehose to build a data lake in Amazon S3, and use Amazon Athena to run ad-hoc queries against it. Requirements: laptop, text editor, AWS account, and AWS Command Line Interface (CLI) installed and configured.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Serverless Data Processing on AWS Workshop
John Pignata
Startup Solutions Architect

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What to Expect
This workshop is an interactive exercise that builds infrastructure to
collect, process, and persist data without using servers
Agenda
• Overview of workshop scenario, modules, and the services we’ll
utilize
• Review of workshop prerequisites and tools
• Execution of four modules—each about thirty minutes in length

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Workshop Modules

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Wild Rydes is an innovative
transportation service that
helps people get to their
destination faster and hassle-
free via unicorns
www.wildrydes.com

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Challenge
Your operations team needs a way to monitor the status
and health of thousands of unicorns in real-time
Where are they?
How many magic points do they have left?
How fast are they traveling?

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Module 1: Real-time Streaming Data
Shadowfax Kinesis
stream
Consumer
Build and demonstrate a stream for
real-time data from the unicorn
fleet
• Create an Amazon Kinesis stream
• Use the Kinesis command-line producer
to write simulated sensor data to the
stream every second
• Use the Kinesis command-line consumer
to read the sensor data from the stream
• Use the Unicorn Dashboard to view
unicorns in real-time on a map

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Module 2: Streaming Aggregation
Shadowfax Kinesis
stream
Consumer
Create a serverless application to
summarize sensor data every minute
• Create an Amazon Kinesis Data Analytics
application to output one row per minute
per unicorn to a new stream
• Use the Kinesis command-line producer to
write simulated sensor data to the stream
• Use the Kinesis command-line consumer
to read the summarized sensor data from
the stream
Kinesis Data
Analytics
application
Kinesis
stream

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Module 3: Stream Processing
Shadowfax Kinesis
stream
Build an AWS Lambda function to
persist sensor data to an Amazon
DynamoDB table
• Create a new DynamoDB table
• Build a Lambda function and configure it
to trigger when data is in the stream
• Use the Kinesis command-line producer
to write simulated sensor data to the
stream
• Use the DynamoDB console to verify that
data are being persisted
Lambda
DynamoDB

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Module 4: Data Lake
Shadowfax Kinesis
stream
Write raw sensor data to Amazon S3
via Amazon Kinesis Firehose and run
queries using Amazon Athena
• Create an Amazon S3 bucket to store raw
sensor data from our unicorns in JSON
format
• Create a Kinesis Firehose delivery stream
to deliver batches from the Kinesis
stream to Amazon S3
• Use Amazon Athena to run queries
against the raw data stored in Amazon S3
Kinesis
Data
Firehose
S3 Bucket
Athena

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Tools & Pre-requisites

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
You’ll need an AWS account and access to
administer resources for:
• AWS Identity and Access Management
(IAM)
• AWS Cloud9
• Amazon Simple Storage Service (Amazon
S3)
• Amazon DynamoDB
• AWS Lambda
• Amazon Kinesis Data Streams
• Amazon Kinesis Data Firehose
• Amazon Kinesis Data Analytics
• Amazon Athena
The workshop instructions
assume only one participant
is using a given AWS account
at a time
Use a personal account or
register a new account rather
than using your
organization’s AWS account
You will receive $10 of AWS
credit to cover any expenses
incurred in this workshop
AWS Account

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
AWS Region
• Build within a single region that supports all necessary services
• The Region Table shows what services are supported within each region
• https://aws.amazon.com/about-aws/global-infrastructure/regional-
product-services/ (search “aws region table”)
• Regions for this workshop:
• US East (N. Virginia)
• US West (Oregon)
• EU (Ireland)

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Web Browser
This workshop has been tested
in the latest versions of Mozilla
Firefox and Google Chrome
Both browsers feature a
Developer Console which you
may need to use to see log
information from the Unicorn
Dashboard

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Lab Guide
Includes setup
information, instructions
for each module, and
step-by-step instructions
if you get stuck

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
AWS Cloud9 IDE

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Unicorn Dashboard

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Kinesis Command-line Clients
• Producer
• Simulates unicorn sensor data
• Unicorn name
• Timestamp
• Current position (latitude, longitude)
• Distance traveled in the last second in meters
• Magic points
• Health points
• Data encoded as a JSON object
• Consumer
• Reads from the end of the stream and outputs data

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
https://dataprocessing.wildrydes.com

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS