10. CouchDB MapReduce
Incremental view index update
“possible to use M/R to query huge partitioned clusters in
realtime, instead of having to wait for a whole M/R job to
complete or having stale, occasionally updated indexes.”
~Damien Katz
http://damienkatz.net/2008/02/incremental_map.html
implementation details
{ omitted } this is Lightning Talk :-P
see : http://horicky.blogspot.com/2008/10/couchdb-
implementation.html
12. fluent-plugin-couch
fluentd (buffered) output plugin
store fluent-event as CouchDB Document to CouchDB
install
$ gem install fluent-plugin-couch
<match couch.**>
configuration type couch
database fluent
host ixixi.cloudant.com
port 80
# for Basic Authentication (optional)
user ixixi
password **********
# Other buffer configurations....
</match>
13. Demo
count tweets in public timeline (use Streaming API)
group by timezone and language in real-time.
M/R view update
(incremental)
bulk insert
Streaming API fluentd flush interval => 5sec.
buffer
15. why need queue?
to stop and update receiver application
to consider the state of receiver servers
Receiver servers ( on which heavy processes are running)
may not be able to deal with messages.
fluentd
16. Amazon SQS
Amazon Simple Queue Service (SQS)
Distributed Queue Service
since 2006 ( the oldest service of AWS )
Automatic Replication in data centers
High Availability
High Scalability
Secure
17. why SQS?
Queue Server requires high availability
Generally, high availability queue server is
very expensive
Amazon SQS is inexpensive
$0.000001( per request )
free tier (~100,000 requests/month)
$~0.201 /GB (Data transfer out)
18. fluent-plugin-sqs
fluentd (buffered) output plugin
store fluent-event to Amazon SQS
install
$ gem install fluent-plugin-sqs
configuration
<match sqs.**>
type sqs
aws_key_id {AWS_KEY_ID}
aws_sec_key {AWS_SEC_KEY}
queue_name fluent-queue
sqs_endpoint sqs.ap-northeast-1.amazonaws.com
# Other buffer configurations....
</match>
20. Amazon SNS
Amazon Simple Notification Service (SNS)
multi-protocol
e-mail,email(json),http(s),SMS,SQS(!!)
Installation not required
push-base architecture
high availability
very inexpensive
26. next feature...?
fluent-plugin-couch
“update document”
e.g.) To handle events such as updating the status of the server.
can refer to previous documents.
∵ CouchDB is MVCC based , automatic versioning.
fluent-plugin-sqs
input-plugin
fluent-plugin-sns
customize formatting & multi formatting