6. AzureのService Model
単純なサービスの構造をモデリングする
Public
Internet
テンプレートは自動的に
Frontend
Background
サービスモデルにマップされる
Web
Role
Process
Role
Load
Balancer
Fundamental
Services
Load Balancer Channel
Endpoint
Interface
Directory Resource
14. Transactions Across Datacenters
（Weekend Project）
o この障害事故の少し前、2009年5月27日 Google IO
で、Ryan Barrettは、自らのWeekend Projectとして、
データセンターをまたいだシステムを提案していた。
o http://www.google.com/intl/ja/events/io/2009/
sessions/TransactionsAcrossDatacenters.html
n what if your entire datacenter falls off the face of the
earth? This talk will examine how current large scale
storage systems handle fault tolerance and
consistency, with a particular focus on the App
Engine datastore. We'll cover techniques such as
replication, sharding, two phase commit, and
consensus protocols (e.g. Paxos), then explore how
they can be applied across datacenters.
19. 2009年9月、Bigtableの見直し
o 2009年9月14日、Ryan Barrettは、“Migration to a
Better Datastore”を発表
http://googleappengine.blogspot.jp/2009/09/
migration-to-better-datastore.html
o Megastore replication saves the day!
Megastore is an internal library on top of
Bigtable that supports declarative schemas,
multi-row transactions, secondary indices, and
recently, consistent replication across
datacenters.
23. Google App Engine
for Business
o 2010年5月19日 Google I/Oで、企業が直面している
問題を解決するために、新しいバージョンのGAE4Bをス
クラッチから開発すると発表。
o GoogleとVMwareとの協業。Google I/Oでの、
VMware CEO Paul Maritzの発言
http://www.youtube.com/
GoogleDevelopers#p/c/02292AD8CFFE1349/7/
KzTgzKkBtqE
29. GAE1.5の発表 2011年5月10日
o Backends: developer-controlled, long-running,
addressable sets of instances
o Pull Queues:you can write a Backend to do
some background processing and pull 1, 10, or
100s of tasks off the Pull Queue
o High Replication Datastore as default:
setting HRD as the default for all new apps
created, lowering the price of HRD storage
from $0.45 down to $0.24, and encouraging
everybody to begin plans to migrate.
http://googleappengine.blogspot.jp/2011/05/app-engine-150-release.html
59. Datastoreでは、
SQLのサブセットが使える
o Filters
n SELECT * FROM Table WHERE A=1 AND
(B=2 OR C=3)
o Sorting
n SELECT * FROM Table ORDER BY A, B
DESC
o Projections / Index-Only Queries
n SELECT A, B FROM Table
61. High Replication
Master/Slave
Performance
Put/delete latency
1/2x–1x
1x
Get latency
1x
1x
Query latency
1x
1x
Consistency
Put/get/delete
Strong
Strong
Most queries
Eventual
Strong
Ancestor queries
Strong
Strong
Occasional planned read-only period
No
Yes
Unplanned downtime
Extremely rare; no Rare; possible to
data loss.
lose a small %
of writes
occurring near
downtime
(recoverable
after event)
62. Free quota per app Pricing if you
per day
exceed your free
quota
Hosting
Free quota per app Price
per day
On-demand Frontend 28 free instance $0.08 / hour
Instances
hours
Reserved Frontend $0.05 / hour
Instances
High Replication 1G
$0.24 / G / month
Datastore
Outgoing Bandwidth
1G
$0.12 / G
Incoming Bandwidth
1G
Free
APIs
Datastore API
50k free read/write/ $0.10/100k write ops
small
$0.07/100k read ops
$0.01/100k small ops
Blobstore API
5G
$0.13 / G / month
63. App Engine MapReduce
A model to do efficient distributed
computing over large data sets.
2011年5月11日発表
http://code.google.com/p/appengine-
mapreduce/
http://www.youtube.com/watch?
v=EIxelKcyCC0
73. Google Cloud SQL
Google Cloud SQL は、Googleクラウド内の
MySQLデータベースである。2011年10月6日、発表。
https://developers.google.com/cloud-sql/
74. Google Cloud SQLの特徴
o MySQLデータベースを、クラウド上で動かす。
o 地理的にはなれたデータセンター間で、データベースの複
製を同期することが出来る。
o mysqldumpを利用して、データベースのimport,
exportが出来る。
o Java, PythonからのAPIが使える。
o コマンドライン・ツールがある。
o Google API Consoleで、SQLが使える。
75. 制限
o 一つのインスタンスのサイズは、10GBまで。
o ユーザー定義関数は、サポートされていない。
o MySQLのレプリカはサポートされない。
o 次のＳＱＬ文は、サポートされない。
n LOAD DATA INFILE
n SELECT ... INTO OUTFILE/DUMPFILE
n INSTALL/UNINSTALL PLUGIN ...
n CREATE FUNCTION ...
n LOAD_FILE()
76. Packages Billing Plan
Tier
RAM Included Included I/O Charge
Storage per Day per Day
D1
0.5GB
1GB
850K
$1.46
D2
1GB
2GB
1.7M
$2.93
D4
2GB
5GB
4M
$5.86
D8
4GB
10GB
8M
$11.71
Per Use Billing Plan
Resource
Charge
D1 Database Instance (0.5GB RAM)
$0.10 per hour
D2 Database Instance (1GB RAM)
$0.19 per hour
D4 Database Instance (2GB RAM)
$0.38 per hour
D8 Database Instance (4GB RAM)
$0.77 per hour
1GB Storage
$0.24 per month
I/O
$0.10 per Million
77. Google Cloud Storage
Fast, scalable, highly available object store
2011年10月11日発表。
https://developers.google.com/appengine/
docs/python/googlestorage/
79. Storage
Monthly Usage
Price (per GB)
First 0 - 1TB
$0.12
Next 9TB
$0.105
Next 90TB
$0.095
Next 400TB
$0.085
Additional Storage
Contact us
Network
Monthly Network Network Network
Usage
(Egress) - (Egress) - (Ingress)
Americas and Asia-Pacific
EMEA* (per GB)
(per GB)
0 - 1TB
$0.12
$0.21
Free
Next 9TB
$0.11
$0.18
Next 90TB
$0.08
$0.15
Additional Data
Transfer
Contact us
80. Requests
PUT, POST, GET GET, HEAD DELETE Requests
bucket**, GET Requests
service** (per 10,000
Requests requests/month)
(per 1,000
requests/month)
$0.01
$0.01
Free
81. クラウド内サービスを結合する
Pipeline API
The Google App Engine Pipeline API
connects together complex, time-consuming
workflows. 2011年5月28日 発表
http://code.google.com/p/appengine-
pipeline/
https://developers.google.com/events/io/
sessions/gooio2012/307/
82. Overview
o Google App Engine Pipeline API は、複雑で時間を
消費するワークフロー（人間の仕事も含む）を、一つに結
合する。
o Pipeline APIの目標は、柔軟さ、ワークフローの再利用、
そして、テスト可能性である。
o このAPIの第一のユースケースは、App Engineの様々
のMapReduceを、計算パイプラインに結合することであ
る。
84. immediate jobs
class DiffJob extends Job2<Integer, Integer, Integer> {
@Override
public Value<Integer> run(Integer a, Integer b) {
return immediate(a - b);
}
}
class MultJob extends Job2<Integer, Integer, Integer> {
@Override
public Value<Integer> run(Integer a, Integer b) {
return immediate(a*b);
}
}
85. Generator Jobs
class ComplexJob extends Job3<Integer, Integer, Integer, Integer> {
@Override
public Value<Integer> run(Integer x, Integer y, Integer z) {
DiffJob diffJob = new DiffJob();
MultJob multJob = new MultJob();
FutureValue<Integer> r = futureCall(diffJob, immediate(x), immediate(y));
FutureValue<Integer> s = futureCall(diffJob, immediate(x), immediate(z));
FutureValue<Integer> t = futureCall(multJob, r, s);
FutureValue<Integer> u = futureCall(diffJob, t, immediate(2));
return u;
}
}
FutureValue<Integer> r =
futureCall(diffJob, immediate(x), immediate(y));
87. Running Pipelines
PipelineService service =
PipelineServiceFactory. newPipelineService();
String pipelineId =
service. startNewPipeline(
new ComplexJob(), 11, 5, 7);
// Later, check on the status and get the final output
JobInfo jobInfo = service. getJobInfo(pipelineId);
JobInfo.State state = jobInfo. getJobState();
if (JobInfo.State.COMPLETED_SUCCESSFULLY
== state){
System.out.println("The output is " +
jobInfo. getOutput());
}
91. class ExternalAgentJob extends Job1<Integer, String> {
@Override
public Value<Integer> run(String userEmail) {
// Invoke ComplexJob on three promised values
PromisedValue<Integer> x = newPromise(Integer.class);
PromisedValue<Integer> y = newPromise(Integer.class);
PromisedValue<Integer> z = newPromise(Integer.class);
FutureValue<Integer> intermediate =
futureCall(new ComplexJob(), x, y, z);
// Kick off the process of retrieving the data from the external agent
getIntFromUser("Please give 1st int", userEmail, x.getHandle());
getIntFromUser("Please give 2nd int", userEmail, y.getHandle());
getIntFromUser("Please give 3rd int", userEmail, z.getHandle());
// Send the user the intermediate result and ask for one more integer
FutureValue<Integer> oneMoreInt = futureCall(new PromptJob(),
intermediate, immediate(userEmail));
// Invoke MultJob on intermediate and oneMoreInt
return futureCall(new MultJob(), intermediate, oneMoreInt);
}
92. public static void getIntFromUser(String prompt, String userEmail,
String promiseHandle) {
// 1. Send the user an e-mail containing the prompt.
// 2. Ask user to submit one more integer on some web page.
// 3. promiseHandle is a query string argument
// 4. Handler for submit invokes
submitPromisedValue(promiseHandle, value)
}
}
class PromptJob extends Job2<Integer, Integer, String> {
@Override
public Value<Integer> run(Integer intermediate, String userEmail) {
String prompt = "The intermediate result is " + intermediate + "."
+ " Please give one more int";
PromisedValue<Integer> oneMoreInt = newPromise(Integer.class);
ExternalAgentJob.getIntFromUser(prompt,
userEmail, oneMoreInt.getHandle());
return oneMoreInt;
}
}
93. Pythonでの定義
class Add(pipeline.Pipeline):
def run(self, a, b):
return a + b
class Multiply(pipeline.Pipeline):
def run(self, a, b):
return a * b
class LinearFunc(pipeline.Pipeline):
def run(self, x, slope=1, offset=0):
# y = m*x + b
mx = yield Multiply(x, slope)
yield Add(mx, offset)
http://code.google.com/p/appengine-pipeline/wiki/GettingStarted
94. Pythonでの呼び出し
# Create it like an object
job = LinearFunc(6, slope=3.5, offset=7)
job.start()
pipeline_id = job.pipeline_id
# Access outputs later
job = LinearFunc.from_id(pipeline_id)
if job.has_finalized:
job.outputs.default.value == 28 # True
120. SELECT timestamp, title, COUNT(*) as cnt
FROM publicdata:samples.wikipedia
WHERE LOWER(title) CONTAINS ‘speed’ AND vp_namespace = 0
GROUP BY title, timestamp ORDER BY cnt DESC LIMIT 20;
Wikipediaの中から、’speed’という
ことばを含むタイトルを持つ論文を
探し出す。
123. SELECT corpus, sum(word_count) AS wordcount
FROM publicdata:samples.shakespeare GROUP BY
corpus ORDER BY wordcount DESC;
Which are the longest works?
