The document discusses Google App Engine, a platform that allows developers to build and host web applications at Google's scale. It summarizes key features of App Engine including scalability, the datastore, memcache, templates, and quotas. It also provides an example of an application built on App Engine called AllForGood, which is an open source volunteering opportunity search engine.

1. Google App Engine
Jonathan Blocksom
Stack Overflow DevDays DC, 10/26/09

2. Submit a Question: http://tinyurl.com/gae-dd09
Attention Conservation Notice
• Jonathan Blocksom
jblocksom@google.com
@jblocksom
Software Engineer
Google
Public Sector Projects Team

3. Submit a Question: http://tinyurl.com/gae-dd09
We’re lonely
Open positions at Google in DC

4. Submit a Question: http://tinyurl.com/gae-dd09
App Engine!
• Launched April 2008
• Paid Use Feb 2009
• Java support May 2009
• Latest Release (1.2.7) October 2009
• Incoming Email
• Previous Release (1.2.5) September 2009
• XMPP Support

5. Submit a Question: http://tinyurl.com/gae-dd09
Your Scalable Website?
• Brad Fitzpatrick describes LiveJournal in ’07:
net.
LiveJournal Backend: 2007
Today
(Roughly.)
BIG-IP
perlbal (httpd/proxy) Global Database
bigip1 mod_perl
bigip2 proxy1 master_a master_b
web1
proxy2 web2
proxy3 Memcached
web3 slave1 slave2 ... slave5
djabberd proxy4 mc1
web4
djabberd proxy5
... mc2 User DB Cluster 1
djabberd
webN mc3 uc1a uc1b
mc4 User DB Cluster 2
... uc2a uc2b
gearmand
Mogile Storage Nodes gearmand1 mcN User DB Cluster 3
sto1 sto2 gearmandN uc3a uc3b
Mogile Trackers
... sto8
tracker1 tracker3 User DB Cluster N
ucNa ucNb
MogileFS Database “workers”
gearwrkN Job Queues (xN)
mog_a mog_b theschwkN jqNa jqNb
slave1 slaveN
http://danga.com/words/
3

6. Submit a Question: http://tinyurl.com/gae-dd09
App Engine Scalable Website
Request App Engine Response

7. Submit a Question: http://tinyurl.com/gae-dd09
What’s Scalable?
• Google Moderator on App Engineused in
March 2009 for White House Town Hall
meeting

8. Submit a Question: http://tinyurl.com/gae-dd09
Scalable

9. Submit a Question: http://tinyurl.com/gae-dd09
Google Moderator team didn’t...
• Provision a server
• Install and configure an OS
• Install and configure a web server
• Install and configure a web framework
• Install and configure a database
• Install and configure a load balancer
• Create user accounts
• Crash in the middle of any of the above

10. Submit a Question: http://tinyurl.com/gae-dd09
App Engine Architecture

11. Submit a Question: http://tinyurl.com/gae-dd09
Static Content
• Served by Google
Static Content Servers
• Very Efficient

12. Submit a Question: http://tinyurl.com/gae-dd09
Dynamic Content
• Java xor Python
• Sandboxed
• No file writes
• No sockets
• No spawning threads/processes
• Smart about caching
• Fairly standard CGI Framework
• Measured and Instrumented

13. Submit a Question: http://tinyurl.com/gae-dd09
Datastore 1 / 3
• Based on BigTable,
transitioning to MegaStore
• ACID Compliant
• 2008 Talk on Implementation:
http://sites.google.com/site/io/under-the-
covers-of-the-google-app-engine-datastore

14. Submit a Question: http://tinyurl.com/gae-dd09
Datastore 2 / 3
• Typical types:
Int, String, DateTime,
GeoPt, PhoneNumber,
PostalAddress, Email, Link, Blob...

15. Submit a Question: http://tinyurl.com/gae-dd09
Datastore 3 / 3
• GQL:
Google Query Language
• If it’s not SQL, it must be good!
SELECT [* | __key__] FROM <kind>
[WHERE <condition> [AND <condition> ...]]
[ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]]
[LIMIT [<offset>,]<count>]
[OFFSET <offset>]
<condition> := <property> {< | <= | > | >= | = | != } <value>
<condition> := <property> IN <list>
<condition> := ANCESTOR IS <entity or key>

16. Submit a Question: http://tinyurl.com/gae-dd09
Memcache
• Arbitrary key/value cache
• Similar API to memcached
• get, set
• get_multi, set_multi for speed
• Namespace Support
• 1MB limit per entry
• Free Quota 8.6M/day, 48K/min (800/sec)

17. Submit a Question: http://tinyurl.com/gae-dd09
Users & Authentication
• @gmail.com addresses
• Apps for Domain
• Admin Privileges

18. Submit a Question: http://tinyurl.com/gae-dd09
URL Fetch
• Load URL
• Asynchronous support
• HTTP / HTTPS
• Max 10 second response (default 5)
• Max 1MB data

19. Submit a Question: http://tinyurl.com/gae-dd09
Even More...
• Scheduled Tasks
• Task Queue
• Email
• Image Manipulation cron jobs
• XMPP email
• ...
XMPP

20. Submit a Question: http://tinyurl.com/gae-dd09
Web Hooks Pattern
• Incoming data / operations to perform are
URLs with POST data
• Handled same as
web request
Scheduled Tasks
Task Queue tasks
Incoming Email
XMPP Messages

21. Submit a Question: http://tinyurl.com/gae-dd09
Templates?
• Treetop Wants to know

22. Submit a Question: http://tinyurl.com/gae-dd09
Templates!
• Django Templates built in
Variable substitution, function calls, loops, etc
• Can add your own Python temlpate lib

23. Submit a Question: http://tinyurl.com/gae-dd09
Let’s Review!

24. Submit a Question: http://tinyurl.com/gae-dd09
Developing w/ App Engine
• Local test server
• Write your code, html, etc on local machine
• Send code to App Engine
• Monitor in App Engine Admin Console

25. Submit a Question: http://tinyurl.com/gae-dd09
Dashboard Demo
• http://appengine.google.com/

26. Submit a Question: http://tinyurl.com/gae-dd09
What have we done?
• Served static web pages
• Served dynamic web pages
• Stored data (cached & persistent)
• Ran a cron Job
• Authenticated users
• We could have...
• Scheduled a task, sent or received email,
loaded a URL, modified an image,
rocked out with XMPP

27. Submit a Question: http://tinyurl.com/gae-dd09
Quotas
• 34 global quotas
in Dashboard
• Individual quotas:
request size
time
amount of data
• Exceptions
thrown when hit
• Free and Paid

28. Submit a Question: http://tinyurl.com/gae-dd09
Quota Rule of Thumb
• Less than 1000 of everything
• The Zen of Quotas:
Meet the quotas and you will be scalable

29. Submit a Question: http://tinyurl.com/gae-dd09
All For Good
• Volunteering Opportunity Search Engine
• Started March 2009 in cooperation with
White House, CNCS, Craigslist, Google...
• Released June 2009
• Open source, released under Apache License
http://code.google.com/p/allforgood/
• Front end at http://allforgood.org
API powers search results on serve.gov

30. Submit a Question: http://tinyurl.com/gae-dd09
Tech Overview
• Partners provide listings in XML or Google Spreadsheets
• AARP, United Way, AmeriCorp, HandsOnNetwork,
United Way
• AFG indexes feeds, stores outside GAE
• Initially Google Base
• Now SOLR on private server
• GAE front end
• API for external apps, partners
• Integration with Google Maps, FriendConnect, Facebook

31. Submit a Question: http://tinyurl.com/gae-dd09
Tech Overview
User or
API
Search
Queries
Backend
Volunteering Opportunities
(Base or
App Engine s SOLR)
lt
su Volunteering Opportunities
Re
memcache Listings ...
Volunteering Opportunities
Crawler
Static Content
Google Code SVN

32. Submit a Question: http://tinyurl.com/gae-dd09
Why A Different Back End?
• No full text search in App Engine Datastore

33. Submit a Question: http://tinyurl.com/gae-dd09
On The Promiscuous Nature of GAE
• GAE plays well with others
• Do full text search with something else!
• Serve large files with S3!
• Serve videos with YouTube!

34. Submit a Question: http://tinyurl.com/gae-dd09
Challenges with AFG
• Parsing questionable data
• Integration w/ Social Networks
• Google Base is an Odd Duck
• I Was New And Had No Clue What Was
Going
• App Engine Outages

35. Submit a Question: http://tinyurl.com/gae-dd09
Q &A

36. Submit a Question: http://tinyurl.com/gae-dd09
More Information
• App Engine Documentation
• Google I/O 2008 & 2009
• 2008 for intro, 2009 for deep dives / Java
• http://code.google.com/events/io/2009/
sessions.html#appengine
“App Engine Nitty-Gritty”
“Spark Plug to Drive Train”
• Google Group
• Samples
http://code.google.com/p/google-app-engine-samples/