Google App Engine - Devfest India 2010

Google App Engine

Feb – March, 2010
Patrick Chanezon
Developer Advocate
Google Developer Relations
chanezon@google.com

Google Confidential and Proprietary 1

Monday, March 8, 2010

Agenda

- App Engine introduction
- Why build it?
- App Engine tour
- What's different?
- Wrap up
- Questions
2


4. The spirit is unmoving, one, swifter than the mind. The
senses do not reach It as It is ever ahead of them. Though
Itself standing still It outstrips those who run.
In It the all-pervading air supports the activities of beings

Isha Upanishad
3


Vayu computing?

Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg
4


Vayu Computing?

7−10
Then they said to Vayu (Air): "O Vayu! Find out who this great
Spirit is." "Yes," he said, and hastened to It. Brahman asked him:
"Who are you?" He replied "I am known as Vayu; I am
also called Matarisva." Brahman said: "What power is in you,
who are so well known?"
Vayu replied: "I can carry off all−whatever there is on earth."
Brahman put a straw before him and said: "Carry this." He
rushed toward it with all his ardour but could not move it.
Then he returned from the Spirit and said to the gods: "I could
not find out who this Spirit is,"
Kena Upanishad
5


What is cloud computing?

SaaS
PaaS
IaaS

Postage
Place
Here
6

IaaS value proposition…

7


Google App Engine
“We wear pagers so you don’t have to”

pager

8

Google App Engine

- Easy to build
- Easy to maintain
- Easy to scale

9


By the numbers

- Built 100K apps
- Maintained by 250K developers
- Scaled to 250M pageviews daily

semi-transparent collage of apps

10


gigya Socialize

11


gigya Socialize - traffic

12


Gigya Socialize - traffic

13


App Engine

14


App Engine

15


Time.com

16


Socialwok

17


Social networking at scale

18


Social networking at scale

>40M Users

18


PubSubHubbub
http://code.google.com/p/pubsubhubbub/
The future is distributed: There will be big hubs, many
small hubs, and tons of publishers and subscribers.
Publishers, subscribers, and hubs may play multiple roles.

19


Hell's Kitchen on facebook

20


Handling peak traffic: mental_floss

21


Elastic / redundant capacity

22


Short term events: earthhour.org

23


http://www.allforgood.org/

24


http://www.snapabug.com/

25


Japan: Q&A for our Future

27


http://www.google.com/tipjar

28


Chillingo Crystal
Gaming meets Social

Zombie Dash Angry Birds LITE underground Meltdown Cogs

Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds
Extreme The Fallen King


Why build it?

Wrench

30

It's just too difficult

31


Hosting means hidden costs
• Idle capacity
• Software patches & upgrades
• License fees
• Lots of maintenance
• Traffic & utilization forecasting
• Upgrades

32


Cloud development in a box
• SDK & “The Cloud”
• Hardware
• Networking
• Operating system
• Application runtime
Java, Python
o

• Static file serving
• Services
• Fault tolerance
• Load balancing

33


Distributed Cloud Computing
Reliability and scalability


Know your Distributed Memes

- Divide & conquer
- Sharding
- Tolerate / expect failure
- Idempotency
- Just scale it up

35


Rocket science
Monolithic Computing

36


Rocket science
Monolithic Computing
Hot spare
“Just in case”

36


Distributed Meme:
Expect, tolerate and design for failure

MTFB: O(Years)

37


Distributed Meme:

MTFB: O(Years)

MTFB: O(Months)

37


Distributed Meme:

MTFB: O(Years)

MTFB: O(Months)

MTFB: O(Minutes/Seconds)

37


Monolithic computing

stop watch
small

stop watch
large

38


Distributed web hosting platform

39



X
39



Retry
X
39


Vertical / horizontal scalability

40


Vertical scalability

41


Vertical scalability

Horizontal scalability

41


Tolerate, plan for and expect small failures


Easy to deploy & scale

1

2

http://www.yourdomain.com/

43


Thinking about scalability
Just a few users....

....the tools, platform and design don't matter too
much

44


Thinking about scalability
Lots and lots of users...

....you must design for scalability

45


But the problem is...

When?

http://digg.com/
http://slashdot.org/

46


The real problem is...

When it happens.

Can you afford to put
everything else on hold?

47


Google App Engine

Leveraging Google's
platform to better serve
your customers

48



49



- Great for web apps
• Request based, data backed
- Parallel processing
- Scales automatically
- Available globally
- Configuration free
- Built-in DoS protections 50


Distributed datastore

51



52



- Arbitrary horizontal scaling

52




52



- Scales to 'Internet scale'

52



- Predictable query performance

52



- No deadlocks

52



- No deadlocks
- No global schema
52


Distributed memcache

Memcache

53


Distributed Meme: Divide & Conquer
Specialized services

Memcache Datastore URL Fetch

Mail XMPP Task Queue

Images Blobstore User Service

54


Distributed Meme: Sharding

Single Sharded

Counter Counter Counter Counter

55


Language runtimes

Duke, the Java mascot
Copyright © Sun Microsystems Inc., all rights reserved.
56


JVM languages

- Scala
- JRuby (Ruby)
- Groovy
- Quercus (PHP)
- Rhino (JavaScript)
- Jython (Python)
57


Ensuring portability

58


Complete Java development stack

59


Google Plugin for Eclipse

60


Google Plugin for Eclipse

61


Google Web Toolkit (GWT)

62


Google Web Toolkit (GWT)

IE

Firefo
x

i
Safar e
m
Chro

Opera

63


GWT Debugging in the browser

64


Quotes to remember

“The fastest code is that
which does not run.”
Joel Webber
GWT co-creator

65


Google Apps + your apps

Your custom
Our Google Apps applications

Google's scalable serving architecture

66


Google Apps integration

http://appid.appspot.com/

http://yourapp.yourdomain.com/

67


2. Log in to your Google Apps domain as an
admin

68


3. Accept terms and specify web address

69


4. The new feature is available in your control
panel

70


5. Additional configuration is available for each feature

71


Google Apps + App Engine

72


Federate your on-premise data

73


Secure Data Connector (SDC)

74


Secure Data Connector

75


Secure Data Connector

and 50+ more...

76


Using Secure Data Connector

77


Installation
- Determine access rules
- Configure and install SDC

77


Installation
Getting ready to serve
- SDC opens SSL tunnel

77


Installation
Getting ready to serve
- SDC opens SSL tunnel
Serving
- User request sent to App Engine
- User authenticated
- App makes request through tunnel
- SDC performs access checks
- Results returned
77


Database export / import

78


Cron support

- clean-up
DB
- prepare r
eports
- batch ma
il
- perform
calcs
- call mom

79


Cron support
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
  <cron>
   <url>/recache</url>
   <description>Repopulate the cache every 2 minutes</description>
   <schedule>every 2 minutes</schedule>
  </cron>
  <cron>
   <url>/weeklyreport</url>
   <description>Mail out a weekly report</description>
   <schedule>every monday of month 08:30</schedule>
   <timezone>America/New_York</timezone>
  </cron>
</cronentries>

("every"|ordinal) (days) "of" (monthspec) (time)

every 5 minutes
every 12 hours
2nd,third mon,wed,thu of march 17:00
every monday of month 09:00
1st monday of sep,oct,nov 17:00

80


Distributed Meme: Divide & Conquer
Task Queues

Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas

Offline Processing on
App Engine: A Look Ahead
Brett Slatkin
May, 2009

81


Tasks Queues - Motivation

• Why do background processing?
o Do work continuously without user requests
o Incrementally process data, compute results
o Smooth out load patterns, lower user latency

• Simple idea in general:
1.Describe the work you want to do now
2.Save the description somewhere
3.Have something else execute the work later

• Work executed in the order received (best-effort FIFO)
• If execution fails, work will be retried until successful



Task Queue Benefits

• Benefits
o Asynchronous
Why do work now when we can do it later?
o Low-latency (for users)
Tasks are light-weight; ~3x faster than Datastore
o Reliable
Once written, a task will eventually complete
o Scalable
Storage of new tasks has no contention
Parallelizable with multiple workers

• Many features can extend this basic concept



How do traditional task queues work?


How do traditional task queues work? (2)

• Polling has problems:
o Worker sits in a loop polling the front of the queue
Not event driven; wasted work
o Workers stay resident when there's no work to do
Wastes machine resources
o Fixed number of workers
Admins must manually add more workers to keep
up or queue will grow without bounds

• Limited optimization possible
o Many systems fake a polling interface with something
event-driven under the hood
o Long-lived, hanging connections


How does our Task Queue API work? (2)


How does our Task Queue API work? (3)


Your application's health

88


App Engine's health history

89


Scalable serving architecture

creative commons licensed photograph from cote 90


Distributed frontends

91



- Edge Caching

91



- Edge Caching
- Priority routing

91



- Edge Caching
- Priority routing
- Geographically close
connection endpoint 91


Life of a request

App App's
User Engine Runtime
Frontend

92


Life of a request

App App's
User Engine Runtime
Frontend

93


Scales dynamically / automatically

User
User

App App's
User Engine Runtime
Frontend

User
User

94


Scales dynamically / automatically

User
App's
User Runtime

App App's
User Engine Runtime
Frontend

App's
Runtime
User
User

95


Multiple versions
http://fredsa.appspot.com/

Version
1.1
App
User Datastore
Engine
Frontend

Version
2.1

http://2.1.fredsa.appspot.com/

96



http://labs.google.com/papers/bigtable.html
97


Bigtable :
A distributed, sharded, sorted array
Row key Row data

98


Bigtable :
A distributed, sharded, sorted array
Row key Row data

Shard 1

Shard 2
.
.

Shard n

99


Datastore design

- Distributed
- Bigtable + entity groups
- ACID transactions
- Optimistic concurrency
- Entities + indexes
- Protobuf encoded entities
100


Datastore properties

- Core value types
- List properties
- Text & binary blobs
- Reference

101


What's different?

102

Datastore - what's new

- Distributed
- Scales to 'internet scale'
- No deadlocks

103


Datastore - what's different

- No inner/outer/natural joins
- Dense index scans
- Per entity metadata
- Soft schema
- No more DDL

104


Datastore - what's familiar

- Indexes
- ACID transactions
- GQL
SELECT * FROM Story
WHERE title = 'App Engine Launch'
AND author = :current_user
AND rating >= 10
ORDER BY rating, created DESC
105


The new ACID

Atomic Associative
Consistent Commutative
Isolated Idempotent
Durable Distributed

106


Entities

107


Entity groups

108


Amortizing costs

Count(COUNT)
Transaction
SUM(Order Amount)

Create
Update
Delete

Customer Order

109


An evolving platform

110

23 months in review
Apr 2008 Python launch
May 2008 Memcache, Images API
Jul 2008 Logs export
Aug 2008 Batch write/delete
Oct 2008 HTTPS support
Dec 2008 Status dashboard, quota details
Feb 2009 Billing, larger files
Apr 2009 Java launch, DB import, cron support, SDC
May 2009 Key-only queries
Jun 2009 Task queues
Aug 2009 Kindless queries
Sep 2009 XMPP
Oct 2009 Incoming email
Dec 2009 Blobstore
Feb 2010 Datastore cursors, Async Urlfetch
111


App Engine
Roadmap

- Support for mapping operations across
datasets
- Alerting system for exceptions in your
application
- Datastore dump and restore facility
112


Wrap up

113

Always free to get started

~5M pageviews/month
• 6.5 CPU hrs/day
• 1 GB storage
• 650K URL Fetch calls/day
• 2,000 recipients emailed
• 1 GB/day bandwidth
• 100,000 tasks enqueued
• 650K XMPP messages/day
114


Purchase additional resources *

* free monthly quota of ~5 million page views still in full effect
115


Thank you

Read more
http://code.google.com/appengine/

Contact info
Patrick Chanezon
Developer Advocate
chanezon@google.com
http://twitter.com/chanezon

Questions
?
116


Thanks
To Alon Levi, Fred Sauer, Brett Slatkin and others for their slides


Google App Engine - Devfest India 2010

Recommandé

Recommandé

Contenu connexe

Similaire à Google App Engine - Devfest India 2010

Similaire à Google App Engine - Devfest India 2010 (20)

Plus de Patrick Chanezon

Plus de Patrick Chanezon (20)

Dernier

Dernier (20)

Google App Engine - Devfest India 2010