Celluloid - Beyond Sidekiq

Celluloid: Beyond Sidekiq
Marcelo Pinheiro
@salizzar - https://github.com/salizzar
Sunday, October 20, 13

Keynote Soundtrack


Topics
Intro to Actor Model
Rubinius: cores 4 Ruby
Celluloid: AM in Ruby
How it works
Supervisors, Supervision Groups
Linking, Observers
Futures, Pools, Notiﬁcations
Celluloid + Rails = ?
Simple Fractal Demo


Carl Hewitt paper from 1973
[OT] Two processors with
native support:
J-Machine
Cosmic Cube
Inspired well-known
languages:
Smalltalk
Erlang (embraces at all)


Actor Model provides a abstraction to concurrency
Threads
Locks
Gained popularity today with multi-core programming
challenge
Better CPU use
More CPUs
Massive concurrency


Everything is a actor
Actors communicates between self by asynchronous
message exchange
Does sound familiar?
Mailbox to buﬀer incoming messages
Mailbox processing with pattern matching
Each actor runs as independent, lightweight process
No shared state


It sounds good, but is not a silver bullet (as
anything in Planet Earth)
Data *must* be immutable
Requires multi-core support to be eﬀective
You need to be comfortable to change your way
of thinking :)


Created by Evan Phoenix
Implements concurrency support for Ruby (no GIL)
Better Garbage Collector
Uses LLVM for aggressive bytecode optimization with JIT
(Just-In-Time) machine code compiler
Written mainly in C++
Ruby STDLIB in pure Ruby (rubysl*)
Wraps native gems with FFI


Supports MRI 2.0 features, focusing on 2.1
horizon
Rubinius implementation was created RubySpec,
a executable speciﬁcation for Ruby Programming
Language
Used by JRuby too


To install in your computer:
$ rbenv install rbx-2.0.0
$ rvm install rbx
To install in your server:
No available OSs packages at this moment :’(
But I create one for Debian Wheezy 64 bits :D
https://github.com/salizzar/rubinius-debian
Next target: CentOS ;)


HOT NEWS: Rubinius X was created in last week
Created by Brian Shirai
Roadmap to modernize Ruby
@polemiquinho: Ruby is dying


Created by Tony Arcieri
Inspired from Erlang
concurrency approach
Gooby pls, only thread-safe
libs
Requires Ruby 1.9 support
MRI >= 1.9
JRuby >= 1.6
Rubinius >= 1.2


Celluloid contains a lot of features, check on Github Wiki Page
Main Features
Automatic Synchronization
Celluloid manages method dispatch and threads
Abstraction layer to Threads / Fibers, don’t worry to manage it
Fault-Tolerance
Let it crash Erlang philosophy
Celluloid oﬀers mechanisms to handle crashed actors


require 'celluloid'
class FredFlinstone
include Celluloid
def scream(to)
@scream = "#{to}#{to[-1] * 10}"
@screamed_at = Time.now
end
def resume
"Screamed [#{@scream}] at #{@screamed_at}"
end
end



[1] pry(main)> fred = FredFlinstone.new
=> #<Celluloid::ActorProxy(FredFlinstone:0x1164f94)>
[2] pry(main)> fred.async.scream("Wilma")
=> nil
[3] pry(main)> fred.resume
=> "Screamed [Wilmaaaaaaaaaaaa] at 2013-10-13 17:12:59 -0300"


Fault-Tolerance
Let-it-crash Erlang philosophy
A crashed actor *must* be handled or your application will
be down
Celluloid have the following mechanisms:
Supervisors
Supervision Groups
Linking


Celluloid: Supervisors
How actors crash? Simple, unhandled exceptions
Warning #1: async calls that raises an error
crashes the message receiver; posterior calls NOT
RAISES ANYTHING
Warning #2: each actor spawn a native Thread,
that is not automatically cleaned by GC; you
MUST explicitly terminate this if not crashed
Supervise to the rescue


require 'celluloid'
class Devops
include Celluloid
def initialize(name)
@name = name
end
def up_to_no_good
@bad_cmd = 'rm-f /'
@command = `#{@bad_cmd}`,
@executed_at = Time.now
end
end


[1] pry(main)> supervisor = Devops.supervise "salizzar"
=> #<Celluloid::ActorProxy(Celluloid::SupervisionGroup:0x104af14) (...)>
[2] pry(main)> salizzar = supervisor.actors.first
=> #<Celluloid::ActorProxy(Devops:0x104c33c) @name="salizzar">
[3] pry(main)> salizzar.async.up_to_no_good
=> nil
[4] pry(main)> E, [2013-10-13T17:37:37.296348 #4829] ERROR -- : Devops
crashed!
Errno::ENOENT: No such file or directory - rm-f /
! (pry):9:in ``'
! (pry):9:in `up_to_no_good'
! (... backtrace error here ...)
[5] pry(main)> salizzar
=> #<Celluloid::ActorProxy:0x104be64>
[6] pry(main)> salizzar.terminate
Celluloid::DeadActorError: actor already terminated
from /vagrant/vendor/bundle/ruby/1.9.1/gems/celluloid-0.15.2/lib/celluloid/
proxies/actor_proxy.rb:66:in `terminate!'
[7] pry(main)> salizzar = supervisor.actors.first
=> #<Celluloid::ActorProxy(Devops:0xe81368) @name="salizzar">


Celluloid: Supervision
Groups
Supervise many actors at once
Able to supervise other groups too
You can create pools of supervised actors
Transparent GC cleaning (automagically
terminates all supervised actors)


Groups
require 'celluloid'
class QuarryWorker
include Celluloid
def initialize(sound) ; @sound = sound ; end
def explode ; puts @sound.upcase ; end
end
class EyeOfSauron < Celluloid::SupervisionGroup
supervise FredFlinstone, as: :fred
pool
QuarryWorker, as: :quarry_pool, args: [ 'boom' ]
end


Groups
[1] pry(main)> eye_of_sauron = EyeOfSauron.run!
=> #<Celluloid::ActorProxy(EyeOfSauron:0x17b608c) (...)>
[2] pry(main)> fred = Celluloid::Actor[:fred]
=> #<Celluloid::ActorProxy(FredFlinstone:0x17abf38)>
[3] pry(main)> quarry_pool = Celluloid::Actor[:quarry_pool]
=> #<Celluloid::ActorProxy(QuarryWorker:0x14a2cc4) @sound="boom">
[4] pry(main)> quarry_pool.async.explode
BOOOM
=> nil
[5] pry(main)> fred.async.scream("Barney")
=> nil
[6] pry(main)> fred.resume
=> "Screamed [Barneyyyyyyyyyyy] at 2013-10-13 18:40:12 -0300"


Celluloid: Linking
Suppose that you have two interdependent actors
and wants to be notiﬁed if one fails
Association by linking actor that commonly dies
and the receiver enables a simple callback when
failure occurs
Useful to catch exceptions


Celluloid: Linking
require 'celluloid'
class RobertoBaggio
include Celluloid
class KickedFarAwayError < StandardError; end
def kick_penalty
raise(KickedFarAwayError, "OH MAMMA MIA! :'(")
end
end


Celluloid: Linking
require 'celluloid'
class GalvaoBueno
include Celluloid
trap_exit :penalty_kick
def penalty_kick(player, reason)
puts "#{player.inspect} will kick and... #{reason.class}!"
2.times { puts "ACABOOOOOOU! "; sleep(1) }
3.times { puts "EH TETRAAAA! "; sleep(1) }
end
end


Celluloid: Linking
[1] pry(main)> galvao = GalvaoBueno.new
=> #<Celluloid::ActorProxy(GalvaoBueno:0x19808e0)>
[2] pry(main)> baggio = RobertoBaggio.new
=> #<Celluloid::ActorProxy(RobertoBaggio:0x1984fe4)>
[3] pry(main)> galvao.link(baggio)
=> #<Celluloid::ActorProxy(RobertoBaggio:0x1984fe4)>
[4] pry(main)> baggio.async.kick_penalty
E, [2013-10-13T18:46:41.512914 #4972] ERROR -- : RobertoBaggio
crashed!
RobertoBaggio::KickedFarAwayError: OH MAMMA MIA! :'(
! (...) backtrace here (...)
#<Celluloid::ActorProxy(RobertoBaggio) dead> will kick and...
RobertoBaggio::KickedFarAwayError!
ACABOOOOOOU!
=> nil
ACABOOOOOOU!
EH TETRAAAA!
EH TETRAAAA!
EH TETRAAAA!


Celluloid: Futures
Lazy computation: calling method with .future
returns a Future object, that will be executed
when .value is called
When value is required, Celluloid synchronously
call method and returns
Transparent error raising


Celluloid: Futures
require 'celluloid'
require 'restclient'
class LazyConsumer
include Celluloid
def retrieve
RestClient.get('http://www.locaweb.com.br').body
end
end


Celluloid: Futures
[1] pry(main)> consumer = LazyConsumer.new
=> #<Celluloid::ActorProxy(LazyConsumer:0xf38284)>
[2] pry(main)> future = consumer.future.retrieve
=> #<Celluloid::Future:0x00000001f925d0>
[3] pry(main)> future.value
=> "<!DOCTYPE html>rn<html dir="ltr" lang="pt-BR">r
n<head>rnt <meta http-equiv="content-type" content=
"text/html; charset=utf-8" />rn <meta name="robots"
content="index, follow" /> (...) HTML (...)"


Celluloid: Pools
Generalized pool mechanism
Default size: number of processors (Celluloid.cores)
Delegates method call to a worker in pool to execute
In MRI, performance is OK with async I/O
Two tips:
Synchronous call with concurrent actors accessing pool
Asynchronous call or Futures with parallel computation


Celluloid: Pools
require 'celluloid'
require 'complex'
class ComplexFactory
def create(seed)
factor = seed + 0.1
a, b = 2.times.collect { srand() % factor }
Complex(a, b)
end
end


Celluloid: Pools
[1] pry(main)> pool = ComplexFactory.pool
=> #<Celluloid::ActorProxy(ComplexFactory:0x12d21d8)>
[2] pry(main)> (1..8).collect do |i|
[3] pry(main)*
pool.future.create(i)
[4] pry(main)* end.collect(&:value)
=> [
(0.9320213390604204+0.6495975396265634i),
(0.2192850934232697+1.3652569533895615i),
(0.9544363040629293+1.5916591822692894i),
(1.9770264674909868+2.9614915840843354i),
(3.2629217852664585+1.3563783097433732i),
(3.4472746283258061+2.7785173492567807i),
(6.3050303460818099+0.3322862800371151i),
(4.1685797344882974+4.5646328695454597i),
(7.4257865062947812+6.2157473844634782i)
]


Celluloid: Notiﬁcations
typeof Observer Pattern
Subscribe / Publish topics to be handled by actors
without need to explicitly make them known
Ideal for long-lived subscriptions, be careful with
short-lived


require 'celluloid'
class HardWorker
include Celluloid, Celluloid::Notifications
def work(factor)
fibo = lambda do |x|
return x if (0..1).include?(x)
fibo[x - 1] + fibo[x - 2]
end
result = fibo.call(factor)
publish('factorial_created', factor: factor, value: result)
true
end
end


require 'celluloid'
class LazyStudent
include Celluloid, Celluloid::Notifications
def on_creation(*args)
data = args.last
puts "Factorial of #{data[:factor]} is #{data[:value]}"
end
end


[1] pry(main)> student = LazyStudent.new
=> #<Celluloid::ActorProxy(LazyStudent:0xda16c8)>
[2] pry(main)> student.subscribe('factorial_created', :on_creation)
=> #<Celluloid::Notifications::Subscriber:0x00000001ac20f0
@actor=#<Celluloid::ActorProxy(LazyStudent:0xda16c8)>,
@method=:on_creation,
@pattern="factorial_created">
[3] pry(main)> worker = HardWorker.new
=> #<Celluloid::ActorProxy(HardWorker:0xce6d64)>
[4] pry(main)> fibo = [ 30, 25, 28, 34, 11, 5 ]
=> [30, 25, 28, 34, 11, 5]
[5] pry(main)> fibo.each { |i| worker.async.work(i) }
=> [30, 25, 28, 34, 11, 5]
Factorial of 30 is 832040
Factorial of 5 is 5


Celluloid: and Rails?
+


=

?

Honestly? Forget it when using MRI.
Rack not handle Fibers well
Since Celluloid uses Fibers a lot, it may be a big problem
Fibers in MRI 1.9 have 4kb of stack size, when exceeds Celluloid mad
With apps that uses ActiveRecord, you will work hard due for
concurrency issues related to DB connection management
Sidekiq have a AR middleware to handle DB connections
Some people runs Rails apps mounted in a Reel (Celluloid::IO
webserver) using reel-rack, but only with MRI 2.0 because of Fiber
stack size


It sounds very bad for MRI, but if you use JRuby or
Rubinius...
No fear, but checks Celluloid Gotchas Github Page for
your sanity :)
In general, you can use it if:
Precise Timing is not a requirement (ﬁre and forget)
Work is CPU bound
Work can be parallelizable


Here are some useful links talking about it:
http://rubyrogues.com/088-rr-concurrency-andcelluloid-with-tony-arcieri/
https://groups.google.com/d/topic/celluloidruby/y5gSm2VjVJw
https://github.com/celluloid/celluloid/wiki/
Gotchas


Celluloid: Fractal Demo
Time to show!
http://fractal.rubinius.salizzar.net
Source code available on:
https://github.com/salizzar/celluloid-fractal


Questions?


Thank you!


Celluloid - Beyond Sidekiq

Recommandé

Recommandé

Contenu connexe

Similaire à Celluloid - Beyond Sidekiq

Similaire à Celluloid - Beyond Sidekiq (20)

Dernier

Dernier (20)

Celluloid - Beyond Sidekiq