Knowledge Sharing Session about the Amazon Web Service

1. Amazon Web Services
(AWS)
Knowledge Sharing Session 27-03-2020

2. What is AWS?
● “Amazon Web Services”
○ For Business People
■ “The cloud”
■ “Web-scale”
■ “Elastic”
○ For Tech people:
■ Cool platform to build stuff on
● Main idea:
○ Rent computers
○ Not monthly basis, Not daily basis, not hourly basis
○ Paid by the MILLISECOND

3. Short AWS History
● Launched in 2002 for internal Amazon usage
○ Standardize internal Amazon structure
○ “Maybe provide public access at some point”
● Public launch in 2004 with “EC2”-Service
○ Allow others to rent virtual computers
● 2006 AWS S3 is launched
○ Store data in the cloud
○ “Developers should worry about what to do with the data, not where to store it”
● 2016 AWS is the most profitable business at Amazon
● 2018 AWS announces “Ground control stations” to communicate with
customer satellites

4. Short AWS History
● 25,000 employees
● 35 billion dollar revenue

5. AWS Availability
● 18 Regions
● Each split into 3-4 “Availability zones (a,b,c,d)
⇒ You can deploy your software everywhere in the world!
● Even if the Data centers go down, you can stay online in
a different availability zone!

6. Services we use and love(or hate, sometimes)
The following services are sorted by importance to
us!

7. Services we use and love (or hate, sometimes)
- EC2 - Virtual Computers
- Where do you want it?
- Which region?
- What operating system?
- Windows or which type of linux?
- How big do you want your computer to be?
- T3.nano
- 2 CPUs, 500MB Ram (Like a computer from 2002)
- $0.0052 per hour (12 cents/day)
- R5.24xlarge
- 96 CPUs, 768GB Ram (Like really really big)
- $7.296 per hour (175$/day)

8. Services we use and love (or hate, sometimes)
- S3 - Data storage
- Mostly like an infinite Dropbox
- Paid by the GB
- 2.3 cent/GB/month
- 1TB = 23€/month
- Thats cheap!
- Structure:
- Top level “bucket”
- Contains something like directories
- Contain files
- Usage:
- A bit annoying
- Use AWS libraries
- Or hope that other libraries implement it (rasterio does!)
- Cool features:
- Can do infinite versioning of the data inside!

9. Services we use and love (or hate, sometimes)
- EFS - Shared Data storage
- Concept:
- Imagine like C:/ and D:/ on your computer, you also have EFS:/
- Cool thing:
- Everybody can “mount” EFS onto his local machine
- Or onto machines in AWS!
- ⇒ Shared directory that everyone can access!
- Cost:
- 36 cent/GB (15 times the cost of S3!!)
- Throughput is limited!
- You can some for free depending on the size of the data you have on EFS
- More is super expensive! (During DeBa we paid 200€/day for speed!)
- Why do we use it?
- It’s like super convenient

10. Services we use and love (or hate, sometimes)
- Batch - Run large jobs and let AWS handle the rest!
- Concept:
- Define “jobs”
- What program do you want to run? (Docker container)
- What hardware is required?
- Run 1000 jobs with different parameters
- AWS automatically starts the right type and the right amount of computers for you
- Shuts them down when the jobs are done
- Cost:
- No overhead. Pay exactly the instances that are used
- Why do we use it?
- In combination with Airflow, we can schedule complicated pipelines of different modules
- Download Imagery >> Classify Imagery >> Calculate Critical Grids

11. Services we use and love (or hate, sometimes)
- Lambda (often called “serverless”)
- “I want to just run a small thing, I don’t care about these servers and stuff”
- Run a script that runs less than 15 minutes
- Often milliseconds
- Amazon starts a computer, runs your thing, and shuts it down again.
- You pay only the milliseconds that your script was running!
- Can scale infinitely
- What we use it for:
- Our webapp!
- Scheduled Backups
- DynamoDB (NoSQL database)
- Store for key-value pairs
- We use it to store all our config files!
- Accessible from everywhere

12. Services we use and love (or hate, sometimes)
- Secrets Manager
- “Soo my program needs to login here. Should I just hard code the credentials?”
- Stores credentials
- Accessible through a simple key
- e.g. get_secret(“liveeo-db-read-only-user”)
- Gives our database credentials.
- Cool thing: We can change our database credentials in ONE place at they are changed
everywhere!
- Elastic Block Storage (EBS)
- EC2 machines by default don’t have an own hard disk!
- Instead they use EBS
- It’s a network file system (similar to EFS!)
- But not shared

13. Services we use and love (or hate, sometimes)
- Relational Database Service (RDS)
- “I want to run a database, but setting it up is so hard”
- Cloudwatch
- “Is my machine still doing something?”
- Gathers logs and hardware metrics of all machines!
- Elastic Container Registry (ECR)
- “Okay, so my program is working now, where do I put it?”
- Store for our Docker Containers
- Identity and Access Management (IAM)
- “Hey Lorenzo, this says I don’t have the rights to do it. Can you fix it?”
- Allows very fine grained permission management for users, machines and roles

14. Less known services we use
Cost Explorer Allows fine grained analysis of all costs
Virtual Private Cloud (VPC) Which computers can see and communicate with which
other computers
S3 Glacier +: Ridiculously cheap data storage
-: Takes a couple of minutes/hours to get the data back!
Autoscaling Groups Let AWS figure out how many computers to start!
Cloudformation Define a super complex cloud architecture as code
API Gateway Expose an API to the outside world!

15. Even less known services we use
Cloud9 (WebIDE) I want to program, but I don’t want to install
anything on my computer
Simple Notification Service
(SNS)
“Send me a mail when it’s done!”
Trigger notifications or events, based on other events!
Route53 Handles Domains (like our eu.liveeo.io)

16. Other really cool shit that is available on AWS
SageMaker Do super fast and efficient Machine Learning on AWS!
Managed Blockchain Run a Blockchain on AWS
Ground Station Manage your satellites via AWS!
Lex Handle Chatbot Questions and Answers!
Comprehend Find out what a text is talking about (Including sentiment
analysis!)
IoT Core Manage all your IoT devices centrally

17. Final words
AWS offers a huge range of tools to tackle thousands of possible use-cases