Google App Engine Update: What's New and How to Optimize Performance

Google App Engine
…is an expression of how Google
thinks about infrastructure.

DevNexus App Engine Update
Mar 21, 2012

David Chandler
Developer Advocate
drfibonacci@google.com

Friday, March 23, 2012

Agenda
App Engine intro
‣ 60 second demo
Whatʼs new in App Engine?
‣ SLA, premier accounts, security audits
‣ Backends, pull queues
‣ HRD, query planner, XG transactions
‣ Cloud SQL, Files API
Load testing recipe
Minimize response time, cost
Case study (Mike Lawrence)
Wrap up / Questions?


Cloud in a box
Easy to build
‣ Local SDK - Java, Python, Go
‣ Lots of APIs

Easy to maintain
‣ Admin Console
‣ Site Reliability Engineers
‣ No clunky knobs/dials

Easy to scale
‣ Google scale infrastructure
‣ No limits to how much data your can put in the datastore


Two minute demo

4


Out of labs... weʼre growing up!

✓ Supported product

✓ 99.95% uptime guarantee (SLA)

✓ Paid operational support

✓ Platform certiﬁcations

✓ Growing fast...

5


3B+ Requests / day
300k+ Apps
100k+ Developers

2008 2009 2010 2011
High
Memcache Task Replication
Queues Datastore
HTTPS Appstats Go Runtime
Channel
XMPP Namespaces API ISAE3402
Python Runtime
Java Runtime Email API Type II
Certification
Backends

Pull Queues


Common types of App Engine apps
Mobile/Tablet
‣ App backends
‣ Cloud persistence
Social/Mobile Games
‣ Speed, scale
‣ API integrations
Consumer Web Apps
‣ Unpredictable trafﬁc
‣ Scale
Business Web Apps
‣ Enterprise
‣ Java runtime


App Engine customers

8


Social networking at scale

>75M Installs

9


Angry Birds = + + + +

10


Khan Academy

SALMAN KHAN

www.youtube.com/watch?v=gM95HHI4gLk#t=0m15s
11


The Royal Wedding

- 32k req/second peak
- 37m pageviews
- 13m visitors

12


Mobile/game backend .... PocketGems

13


WebFilings

14


Best Buy

15


Chillingo Crystal
Gaming meets Social
16

Zombie Dash Angry Birds LITE Underground Meltdown Cogs

Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds
Extreme The Fallen King


What’s New?

17

Feature overview

URL Fetch
Frontends
XMPP
Compute Backends Network
Channel API
Task Queues
Mail API
Cron

Datastore Images API
Storage
Memcache App Identity
Services
Namespaces Users API
Blobstore MapReduce API
Cloud SQL Pipeline API
Static content Prospective Search API

18


Frontends, Backends, Task Queue, Cron Compute

Pull (Task) Queues

App
App
Engine Push (Task) Queues / Cron
Engine
Client Frontend
Client Frontend

App
App App
App Engine
App
Client Engine Engine
App
Client Engine Frontend
Engine
Backend Frontend
Engine
Backend Backend
Backend

19


URL Fetch Network

HTTP/1.1
‣ GET, POST, PUT, HEAD, and DELETE
Maximum request deadlines
‣ 60s (default 5s): user facing (online) requests
‣ 10 minutes (new): cron, task queue (ofﬂine) requests
Execution options
‣ Synchronous
‣ Async


Talking to the outside world Network

XMPP / Jabber
‣ Send/receive messages
‣ Send chat invites
‣ Request presence, status
‣ Set status
Channel API
‣ Server push
Mail API
‣ Send and receive
Use cases
‣ Chat bot
‣ Realtime notiﬁcations
‣ Chat user interface


Images Services

22


Prospective search Services

Register your queries
‣ Deﬁne interesting search criteria
Create your data
‣ Users and systems interact with your app
Receive callbacks
‣ App Engine tells you about data which matches searches


Identity Services

Users API
‣ Require users to login and identify
- Email address
- Unique, stable id
‣ Servlet ﬁlter for dynamic requests

App Identity
‣ Who am I?
appIdentityService.getServiceAccountName()
‣ Assert your identity to applications
- GAE handles key security, key rotation for you
- Other Google APIs
appIdentityService.getAccessToken()
- On other systems
appIdentityService.signForApp()


MapReduce & Pipeline API Services

code.google.com/p/appengine-pipeline/
25


High Replication Datastore (HRD) Storage

Schemaless
Object datastore
‣ Entities & Entity Groups
Query engine
‣ Standard & custom indexes
Atomic transactions
Features
‣ Highly scalable
- More distributed as more data is stored
‣ Highly reliable
- Synchronous, multi-datacenter replication
‣ Managed
www.google.com/events/io/2011/sessions/more-9s-please-
under-the-covers-of-the-high-replication-datastore.html


Advanced query planner New! Storage

Datastore never requires an exploding index
‣ code.google.com/appengine/docs/python/datastore/
queries.html#Big_Entities_and_Exploding_Indexes
dev_appserver
‣ Understands query planner changes
Zigzag merge-join queries
‣ Will continue scanning for up to 30 sec
‣ No more "needs index" errors (although some may timeout)
‣ www.google.com/events/io/2009/sessions/
BuildingScalableComplexApps.html


The “old” query planner Storage

es
Index(Photo, I
owner_id,s -date)ndex
Tag -rating) = 256
Index(Photo, owner_id, 2^6*4 12
Index(Photo, 1
owner_id, -comments) = 5
Index(Photo, 2 2^7*4
owner_id, -downloads) 1024
=
Index(Photo, size, -date) 2^ 8*4
3 .
Index(Photo, size, -rating) .
Index(Photo, .
size, -comments) .
Index(Photo, size, -downloads) .
.
…
Index(Photo, owner_id, size, -date)
Index(Photo, owner_id, size, -rating)
Index(Photo, owner_id, size, -comments)
Index(Photo, owner_id, size, -downloads)
…
Index(Photo, aspect, coloration, license, owner_id, size, -date)
Index(Photo, aspect, coloration, license, owner_id, size, -rating)
Index(Photo, aspect, coloration, license, owner_id, size, -comments)
Index(Photo, aspect, coloration, license, owner_id, size, -downloads)

= 2^(no. of filters) * (no. of different orders) = 2 ^ 6 * 4 = 256 indexes


The “improved” query planner Storage

Index(Photo, owner_id, -date)
Index(Photo, owner_id, -rating)
Index(Photo, owner_id, -comments)
Index(Photo, owner_id, -downloads)
Index(Photo, size, -date)
Index(Photo, size, -rating)
Index(Photo, size, -comments)
Index(Photo, size, -downloads)
…
Index(Photo, license, -date)
Index(Photo, license, -rating)
Index(Photo, license, -comments)
Index(Photo, license, -downloads)
…
Index(Photo, tag, -date)
Index(Photo, tag, -rating)
Index(Photo, tag, -comments)
Index(Photo, tag, -downloads)
= (number of filters) * (number of orders) = 7 * 4 = 28 indexes


Query planner performance - cost trade-offs
Option #1 - Write to more indexes at write time
‣ Entity writes are expensive
‣ Possible queries must be known in advance
- Others will fail with Needs Index error
‣ Predictable / fast query time [for supported queries]

Option #2 - Scan multiple indexes at read time
‣ Entity writes much cheaper
‣ More queries possible with fewer indexes
‣ Ad hoc queries
‣ Query response time depends on shape of data
- Speciﬁc cases can be optimized by providing addʼl indexes


Using the new query planner Storage

Upgrade to 1.5.4+
Turn off auto index generation
‣ Python: dev_appserver.py --require_indexes
‣ Java: <datastore-indexes autoGenerate="false">
Remove unnecessary custom index deﬁnitions
‣ Python:
- index.yaml
‣ Java:
- WEB-INF/datastore-indexes.xml
- WEB-INF/appengine-generated/datastore-indexes-auto.xml
Test your app
Deploy to production
‣ Judiciously run: appcfg vacuum_indexes


XG Transactions New! Storage

In a single transaction…
‣ Access 1..5 entity groups - read and/or write
‣ Does not provide transactional ancestor-less query
‣ Does allow 1..5 separate ancestor queries within transaction
Concurrency exceptions possible
‣ 1st access of each entity group
‣ For reads as well as writes
Availability
‣ Only in HRD apps
‣ App Engine 1.5.5 release
Non-ancestor (transaction-less) queries may see…
‣ Partially committed non-XG transaction (same as before)
‣ Partially committed XG transactions (more likely)


XG Transactions - Java Storage

import com.google.appengine.api.datastore.*;

void transferFunds(long amt, String fromAccount, String toAccount) {
DatastoreService datastore =
DatastoreServiceFactory.getDatastoreService();
TransactionOptions options = TransactionOptions.Builder.withXG(true);
Transaction txn = datastore.beginTransaction(options);

Entity a = datastore.get(KeyFactory.createKey("Account", fromAccount));
Entity b = datastore.get(KeyFactory.createKey("Account", toAccount));

a.setProperty("balance", (Long) a.getProperty("balance") - amt);
b.setProperty("balance", (Long) b.getProperty("balance") + amt);

datastore.put(txn, a);
datastore.put(txn, b);

txn.commit();
}


Cloud SQL (labs) Storage

New! Cloud SQL support in Google Plugin for Eclipse 2.5

Client

Client

Client

Client

34


Cloud SQL (labs) Storage

Developer console
‣ Easy to use
Fully managed
‣ Site Reliability Engineers (SREs)
High availability
‣ Synchronous replication to multiple data centers
Integrated with Google App Engine
‣ Java: JDBC, Python: DB-API
‣ Use with High Replication Datastore
Migration
‣ MySQL Import / export


Google Cloud Storage / Files API Storage

Google Cloud Storage
‣ Buckets in the cloud
‣ REST API
‣ Command line
‣ Web UI tool
‣ Enable in APIs console

Use from App Engine
‣ Files API (new)
‣ No more OAuth hassle
‣ http://code.google.com/appengine/docs/java/googlestorage/
overview.html


Minimize response time

37


Appstats
“I used to be blind,
but now I can see :-)”
--An early Appstats user

38


Load testing - Guidelines
Test trafﬁc must be representative of real user trafﬁc
Ramp over a period of several minutes

his!
o ts ! s
t d hi e i etter
on’ T ic
D N
uc h b
m


Load testing - basic recipe
Send steady 5 qps trafﬁc to your site
‣ Uncover basic problems, e.g. entity contention on global data
Monitor for 15-20 minutes and expect to see:
‣ Stable request latency <1000ms (no longer an issue)
‣ Zero quota denials
‣ Low error rate
‣ Tasks queues are keeping up
‣ No elevated datastore contention
Slowly ramp up; increase trafﬁc by 2x
‣ Uncover next application bottleneck, if any
Repeat until serving desired qps
‣ Plan your next feature


Memcache

2 ms

App
App
Client Engine
App
Client Engine
App
Frontend
Engine
Frontend
Engine
Backend
Backend 20m
s

41


Enable content caching
Free lunch possibility
‣ And your users will be happier too
Use memcache
‣ Only if you care about your users
Static resource ﬁles
‣ Do not consume instance hours
HTTP/1.1 caching
Cache-Control: public, s-maxage=...
‣ Respected by browsers, ISPs and App Engine
‣ Saves bandwidth
Replace dynamic content with
‣ Static content
‣ Blobstore (custom HTTP headers)


Best practices in client-side code
Clients all
t his!
Av oid checking in at a
speciﬁc time of
day
Spread out load:
- Randomize client checkin times

Clients retr y
failures after a
t his!
And ﬁxe d interval
Avoi d a Denials-of-Ser vice
(DoS) on yo ur app after a
failure:
- Use exponential back-off:
1s, 2s, 4s, 8s, 16s, …
- Use a fuzz factor:
randomize retr y times
43


Best practices in server-side code
Reduce instance hours
‣ Reduce request latency
‣ Use Memcache
‣ Datastore/Memcache batch get/put
‣ Async URL Fetch, Datastore, Memcache
- Parallelize RPCs
‣ Enable HTTP session async writes
Spin up fewer instances
‣ Task queue rate and max_concurrent_requests
‣ X-AppEngine-FailFast

code.google.com/appengine/articles/managing-resources.html


Best practices in server-side code
Enable concurrent requests
‣ Java appengine-web.xml
‣ Python 2.7
Discounted reserved instance hours
‣ $0.08/hr → $0.05/hr
Scheduler knobs
‣ Max Idle Instances
‣ Min Pending Latency


Manage storage costs
Discard stuff you no longer need
‣ Entities
‣ Indexes
‣ Blobs
‣ Tasks
Drop unwanted indexes
‣ Vacuum custom indexes
‣ New query planner
Explicitly mark properties as unindexed
‣ To migrate existing entities, put() again with:
Python: foo = db.StringProperty(indexed=False)
Java: entity.setUnindexedProperty(“foo”, “bar”)


Manage 'datastore ops' costs
Number of write ops in SDK
Fewer indexes & indexed properties → fewer write ops
Take advantage of query planner improvements
Replace queries with
‣ keys-only queries (cheaper)
‣ fetch-by-key (optimal)
‣ datastore cursors (for pagination)


Enable warming requests
Request logs
/_ah/warmup
Default behavior
‣ Enabled for Java apps / Disabled for Python apps
‣ All jar files are indexed in memory
‣ Initializes your application and filters
- Servlets marked in <load-on-startup>
- ServletContextListener filters
Custom warmup servlet
‣ Override built-in _ah_warmup servlet
<servlet>
<servlet-name>_ah_warmup</servlet-name>
<servlet-class>foo.MyWarmupServlet</servlet-class>
</servlet>


Only upload what you need
Python
static_files, static_dir
skip_files
Java
<static-files>
<resource-files>
Diagnostics
appcfg.sh --retain_upload_dir update …


Custom error pages
Avoid

Java
<static-error-handlers>
<handler file="default_error.html"/>
<handler file="over_quota.html" error-
code="over_quota"/>
</static-error-handlers>
Python
error_handlers:
- file: default_error.html
- error_code: over_quota
file: over_quota.html


Google App Engine Update: What's New and How to Optimize Performance

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Google App Engine Update: What's New and How to Optimize Performance

Similar to Google App Engine Update: What's New and How to Optimize Performance (20)

More from David Chandler

More from David Chandler (13)

Recently uploaded

Recently uploaded (20)

Google App Engine Update: What's New and How to Optimize Performance