An overview of the different Cloud technologies available from Google including App Engine, Google Storage, Google Prediction API, and BigQuery. This presentation was given to the San Diego GTUG on Aug 26th, 2011.

1. Introduction to Google's Cloud
Technologies
Chris Schalk
Developer Advocate
@cschalk

2. Agenda
● Introduction
● Introduction to Google's Cloud Technologies
● App Engine Recap
● Google's new Cloud Technologies
○ Google Storage
○ Prediction API
○ BigQuery
● Summary Q&A

3. Google's Cloud Technologies
Google App Engine
Google Google
BigQuery Prediction API
Google
Storage

4. Google App Engine
An App Engine recap...

5. Cloud Development in a Box
● Downloadable SDK
● Application runtimes
○ Java, Python
● Local development tools
○ Eclipse plugin,
AppEngine Launcher
● Specialized application
services
● Cloud based dashboard
● Ready to scale
● Built in fault tolerance, load
balancing

6. Specialized Services
Memcache Datastore URL Fetch
Mail XMPP Task Queue
Images Blobstore User Service
But, is that it?

7. No!!
Now App Engine has access to even
more Specialized Cloud Services...

8. Google's new Cloud Technologies

9. New Google Cloud Technologies
● Google Storage
○ Store your data in Google's cloud
● Prediction API
○ Google's machine learning tech in an API
● BigQuery
○ Hi-speed data analysis on massive scale
● SQL Service*
○ Relational Database in the cloud

10. Google Storage for Developers
Store your data in Google's cloud

11. What Is Google Storage?
● Store your data in Google's cloud
○ any format, any amount, any time
● You control access to your data
○ private, shared, or public
● Access via Google APIs or 3rd party tools/libraries

12. Sample Use Cases
Static content hosting
e.g. static html, images, music, video
Backup and recovery
e.g. personal data, business records
Sharing
e.g. share data with your customers
Data storage for applications
e.g. used as storage backend for Android, AppEngine, Cloud
based apps
Storage for Computation
e.g. BigQuery, Prediction API

13. Google Storage Benefits
High Performance and Scalability
Backed by Google infrastructure
Strong Security and Privacy
Control access to your data
Easy to Use
Get started fast with Google & 3rd party tools

14. Google Storage Technical Details
● RESTful API
○ Verbs: GET, PUT, POST, HEAD, DELETE
○ Resources: identified by URI
○ Compatible with S3
● Buckets
○ Flat containers
● Objects
○ Any type
○ Size: 100 GB / object
● Access Control for Google Accounts
○ For individuals and groups
● Two Ways to Authenticate Requests
○ Sign request using access keys
○ Web browser login

15. Security and Privacy Features
● Key-based authentication
● Authenticated downloads from a web browser
● Sharing with individuals
● Group sharing via Google Groups
● Access control for buckets and objects
● Set Read/Write/List permissions

16. Demo
● Tools:
○ GSUtil
○ GS Manager
● Upload / Download

17. Google Storage usage within Google
Google Google
BigQuery Prediction API
Haiti Relief Imagery USPTO data
Partner Reporting Partner Reporting

18. Some Early Google Storage Adopters

19. Google Storage - Pricing
○ Free trial quota until Dec 31, 2011
■ For first project
■ 5 GB of storage
■ 25 GB download/upload data
■ 20 GB to Americas/EMEA, 5GB APAC
■ 25K GET, HEAD requests
■ 2,5K PUT, POST, LIST* requests
○ Production Storage
■ $0.17/GB/Month (Location US, EU)
■ Upload - $0.10/GB
■ Download
■ $0.15/GB Americas / EMEA
■ $0.30/GB APAC
■ Requests
■ PUT, POST, LIST - $0.01 / 1000 Requests
■ GET, HEAD - $0.01 / 10,000 Requests
■ Up to 99.9% SLA

20. Google Storage Summary
● Store any kind of data using Google's cloud infrastructure
● Easy to Use APIs
● Many available tools and libraries
○ gsutil, GS Manager
○ 3rd party:
■ Boto, CloudBerry, CyberDuck, JetS3t, and more

21. Google Prediction API
Google's prediction engine in the cloud

22. Google Prediction API as a simple example
Predicts outcomes based on 'learned' patterns

23. How does it work?
"english" The quick brown fox jumped over the
The Prediction API lazy dog.
finds relevant
"english" To err is human, but to really foul things
features in the up you need a computer.
sample data during "spanish" No hay mal que por bien no venga.
training.
"spanish" La tercera es la vencida.
The Prediction API
later searches for ? To be or not to be, that is the
question.
those features
? La fe mueve montañas.
during prediction.

24. A virtually endless number of applications...
Customer Transaction Species Message Diagnostics
Sentiment Risk Identification Routing
Churn Legal Docket Suspicious Work Roster Inappropriate
Prediction Classification Activity Assignment Content
Recommend Political Uplift Email Career
Products Bias Marketing Filtering Counselling
... and many more ...

25. Using the Prediction API
A simple three step process...
Upload your training data to
1. Upload Google Storage
Build a model from your data
2. Train
3. Predict Make new predictions

26. Step 1: Upload
Upload your training data to Google Storage
● Training data: outputs and input features
● Data format: comma separated value format (CSV)
"english","To err is human, but to really ..."
"spanish","No hay mal que por bien no venga."
...
Upload to Google Storage
gsutil cp ${data} gs://yourbucket/${data}

27. Step 2: Train
Create a new model by training on data
To train a model:
POST prediction/v1.3/training
{"id":"mybucket/mydata"}
Training runs asynchronously. To see if it has finished:
GET prediction/v1.3/training/mybucket%2Fmydata
{"kind": "prediction#training",...
,"training status": "DONE"}

28. Step 3: Predict
Apply the trained model to make predictions on new data
POST
prediction/v1.3/training/mybucket%2Fmydata/predict
{ "data":{
"input": { "text" : [
"J'aime X! C'est le meilleur" ]}}}

29. Step 3: Predict
Apply the trained model to make predictions on new data
POST prediction/v1.3/training/bucket%2Fdata/predict
{ "data":{
"input": { "text" : [
"J'aime X! C'est le meilleur" ]}}}
{ data : {
"kind" : "prediction#output",
"outputLabel":"French",
"outputMulti" :[
{"label":"French", "score": x.xx}
{"label":"English", "score": x.xx}
{"label":"Spanish", "score": x.xx}]}}

30. Step 3: Predict
Apply the trained model to make predictions on new data
import httplib
header = {"Content-Type" : "application/json"}#...put new data in JSON
format in params variable
conn = httplib.HTTPConnection("www.googleapis.com")conn.request
("POST",
"/prediction/v1.3/query/bucket%2Fdata/predict", params, header)print
conn.getresponse()

31. Demo
● Command line Demos
○ Training a model
○ Checking training status
○ Making predictions
● A complete Web application using the JavaScript
API for Prediction

32. Prediction API Capabilities
Data
● Input Features: numeric or unstructured text
● Output: up to hundreds of discrete categories
Training
● Many machine learning techniques
● Automatically selected
● Performed asynchronously
Access from many platforms:
● Web app from Google App Engine
● Apps Script (e.g. from Google Spreadsheet)
● Desktop app

33. Prediction API - key features
● Multi-category prediction
○ Tag entry with multiple labels
● Continuous Output
○ Finer grained prediction rankings based on multiple labels
● Mixed Inputs
○ Both numeric and text inputs are now supported
Can combine continuous output with mixed inputs

34. Google BigQuery
Interactive analysis of large datasets in Google's cloud

35. Introducing Google BigQuery
● Google's large data adhoc analysis technology
○ Analyze massive amounts of data in seconds
● Simple SQL-like query language
● Flexible access
○ REST APIs, JSON-RPC, Google Apps Script

36. Why BigQuery?
Working with large data is a challenge

37. Many Use Cases ...
Interactive Trends
Spam
Tools Detection
Web Network
Dashboards Optimization

38. Key Capabilities of BigQuery
● Scalable: Billions of rows
● Fast: Response in seconds
● Simple: Queries in SQL
● Web Service
○ REST
○ JSON-RPC
○ Google App Scripts

39. Using BigQuery
Another simple three step process...
Upload your raw data to
1. Upload Google Storage
Import raw data into
2. Import
BigQuery table
3. Query Perform SQL queries on
table

40. Writing Queries
Compact subset of SQL
○ SELECT ... FROM ...
WHERE ...
GROUP BY ... ORDER BY ...
LIMIT ...;
Common functions
○ Math, String, Time, ...
Statistical approximations
○ TOP
○ COUNT DISTINCT

41. BigQuery via REST
GET /bigquery/v1/tables/{table name}
GET /bigquery/v1/query?q={query}
Sample JSON Reply:
{
"results": {
"fields": { [
{"id":"COUNT(*)","type":"uint64"}, ... ]
},
"rows": [
{"f":[{"v":"2949"}, ...]},
{"f":[{"v":"5387"}, ...]}, ... ]
}
}
Also supports JSON-RPC

42. Security and Privacy
Standard Google Authentication
● Client Login
● OAuth
● AuthSub
HTTPS support
● protects your credentials
● protects your data
Relies on Google Storage to manage access

43. Large Data Analysis Example
Wikimedia Revision History
Wikimedia Revision history data from: http://download.wikimedia.
org/enwiki/latest/enwiki-latest-pages-meta-history.xml.7z

44. Using BigQuery Shell
Python DB API 2.0 + B. Clapper's sqlcmd
http://www.clapper.org/software/python/sqlcmd/

45. BigQuery from a Spreadsheet

46. BigQuery from a Spreadsheet

47. Recap
● Google App Engine
○ Application development platform for the
cloud
● Google Storage
○ High speed cloud data storage on Google's
infrastructure
● Prediction API
○ Google's machine learning technology able to
predict outcomes based on sample data
● BigQuery
○ Interactive analysis of very large data sets
○ Simple SQL query language access

48. Further info available at:
● Google App Engine
○ http://code.google.com/apis/storage
● Google Storage for Developers
○ http://code.google.com/apis/storage
● Prediction API
○ http://code.google.com/apis/predict
● BigQuery
○ http://code.google.com/apis/bigquery

49. Thank you!
Questions?
● @cschalk