Os Whitaker

Keeping Your Workers In Line: ,[object Object],[object Object],use TheSchwartz;

Once upon a time... In a galaxy far far away Users wanted features Like subscription based notifications, Pinging external services, And other things which Love to tie up webserver processes And are generally too slow / blocking to execute synchronously with web requests

(Yes, that was really my best slide-foo, so bear with me)

So we needed to solve the problems these features create

Which can easily result in a mess

Contacting External Web Services ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Processing Uploaded Media ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Initial Solution: GhettoQueue ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Incoming data from lots of different transports ,[object Object],[object Object],[object Object],[object Object]

Initial Solution: Lots of Daemons! ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Events, Subscriptions, Notifications? ,[object Object],[object Object],[object Object],[object Object],[object Object]

Problems With These Approaches ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Implementation ,[object Object],[object Object],[object Object]

Topology ,[object Object],[object Object],[object Object]

Schwartz Database ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Schwartz Workers ,[object Object],[object Object],[object Object],[object Object]

Full Topology: With Application

Request Cycle ,[object Object],[object Object],[object Object],[object Object],[object Object]

Workers can define other per-worker behaviors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Other fun features ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Using TheSchwartz in production ,[object Object]

Schwartz Database Configuration ,[object Object],[object Object],[object Object],[object Object],[object Object]

Schwartz Database Configuration … .. And so on, adding clusters as needed

Tools and monitoring ,[object Object],[object Object],[object Object],[object Object],[object Object]

Schwartzmon Example lj@ljadmin1:~$ schwartzmon --dsn=DBI:mysql:theschwartz_livejournalhost=10.191.90.101 --user=lj -f errors Thu Jul 26 19:22:13 2007 [2116902910]: Connection failed to domain 'imagemenagerie.com', MXes: [imagemenagerie.com] Thu Jul 26 19:22:14 2007 [2120335058]: Connection failed to domain 'thedashcat.net', MXes: [thedashcat.net] Thu Jul 26 19:22:15 2007 [2126277932]: Connection failed to domain 'cox.net', MXes: [mx.west.cox.net mx.east.cox.net] Thu Jul 26 19:22:16 2007 [2126007758]: Connection failed to domain 'cox.net', MXes: [mx.west.cox.net mx.east.cox.net] Thu Jul 26 19:22:16 2007 [2126309296]: Permanent failure TO [hey2a@cs.com]: 550 MAILBOX NOT FOUND Thu Jul 26 19:22:17 2007 [2126309446]: Permanent failure TO [joe_junkpan@hotmail.com]: 550 Requested action not taken: mailbox unavailable Thu Jul 26 19:22:18 2007 [2126308836]: Error during DATAEND phase to [ourxtrees@yahoo.com]: 451 go ahead Message temporarily deferred - [250] Thu Jul 26 19:22:18 2007 [2126188996]: Connection failed to domain 'buffyboarders.zzn.com', MXes: [c2mailmx.mailcentro.com c2mds.mailcentro.com] Thu Jul 26 19:22:21 2007 [2125661508]: Connection failed to domain 'cox.net', MXes: [mx.east.cox.net mx.west.cox.net]

Ljworkerctrl example ,[object Object]

Ljworkerctrl example ,[object Object],-

Questions? http://code.sixapart.com/svn/TheSchwartz/trunk

Os Whitaker

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Os Whitaker

Similaire à Os Whitaker (20)

Plus de oscon2007

Plus de oscon2007 (20)

Dernier

Dernier (20)

Os Whitaker