WebCo Way CParty 2009

The WebCo Way:
O que fazemos &
Como fazemos

Manoel Lemos [CTO & Founder]

Campus Party 2009 - Desenvolvimento
São Paulo, SP, Brasil

Acompanhe e participe

#cparty
http://live.blogblogs.com.br/

BlogBlogs Status
Business Numbers Trafﬁc Numbers
115.000 users 7.5M PageViews / month
200.000 blogs 3.5M Uniques / month
2.500.000 posts Top 5000 Alexa Global
250 users / day Top 120 Alexa Brasil
150 blogs / day 50M Widget
Impressions / month
7.000 posts / day

Scaling is a process
complicated and painful!
Its impossible to predict with accuracy
There are hidden bottlenecks
Sometimes you need to step back
Sometimes you need to re-factor everything
There are no “silver-bullets”
Complexity will grow with you

Growth never happens in a
single dimension
Trafﬁc : more users, more transactions, more downloads...
Size : database, memory, bandwidth...
Complexity : algorithms, architecture, infrastructure...
Organization : team, communication, processes...
Business : models, partners, suppliers...
Mostly the issues grow faster than linearly!!!

httpd
mongrel
mongrel
mongrel

mysql

httpd
mongrel
mongrel
mongrel
memcached
mysql

load balancing

httpd httpd
mongrel mongrel
mongrel mongrel
mongrel mongrel
memcached memcached
mysql

load balancing

httpd httpd httpd
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing

httpd httpd httpd

mysql

load balancing

httpd httpd httpd


mysql

load balancing

httpd httpd httpd

mysql

load balancing 1 (web)

httpd httpd httpd

mysql

load balancing 1 (web) load balancing 2 (widgets)

httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached

mysql



mysql (master)

mysql (slave)



mysql mysql (master)

mysql (slave)




mysql (slave)

memcached memcached

memcached memcached




mysql (slave)

memcached memcached Robot Robot





mysql (slave)

Indexer Indexer
(Search) (Search)





mysql (slave)

Indexer Indexer
(Search) (Search)

httpd httpd
mongrel mongrel
mongrel mongrel
mongrel mongrel
mongrel mongrel
mongrel mongrel

load balancing 3 (API)


mongrels mongrels mongrels mongrels mongrels mongrels


mysql (slave)

Indexer Indexer
(Search) (Search)

memcached memcached Robot Robot mongrels mongrels

load balancing 3 (API)

load balancing

mongrels mongrels mongrels mongrels

mongrels mongrels mongrels mongrels

Robot

Robot

Indexer
memcached memcached
(Search)

mysql mysql
(master) (master)

Indexer
memcached memcached
(Search)

load balancing load balancing

mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels


Robot Robot

Robot Robot

Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

mysql mysql mysql mysql
(master) (master) (master) (master)

Indexer Indexer
(Search) (Search)



Eureca!! Ideas...

Single Sign On
Shared Resources
Robot Robot

Integrated Search
and more...
Robot Robot

Indexer Indexer
(Search) (Search)


Indexer Indexer
(Search) (Search)




#howto ?
Robot Robot

Robot Robot

Indexer Indexer
(Search) (Search)


Indexer Indexer
(Search) (Search)


mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels





Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql
(master) (slave) (slave)

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

DNA
Credentials
(PHP)

ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

DNA DNA
Credentials Proﬁle
(PHP) (PHP)

ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

DNA DNA DNA
Credentials Proﬁle Avatars
(PHP) (PHP) (PHP)

ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

DNA DNA DNA DNA DNA
Credentials Proﬁle Avatars Scoring SocialGraph
(PHP) (PHP) (PHP) (C++) (Python)

ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)

ActiveDNA (REST)

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)




ActiveDNA (REST)

load balancing


ActiveDNA (REST)

load balancing

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)



humm...

PHP... ActiveDNA (REST)

WTF??!! load balancing

Credentials
(PHP)

ActiveDNA (REST)

load balancing

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)



humm...

PHP... ActiveDNA (REST)

WTF??!! load balancing


ActiveDNA (REST)

load balancing

Indexer Indexer
Robot Robot
(Search) (Search)

mysql mysql mysql

Indexer Indexer
Robot Robot
(Search) (Search)

Active DNA
The problem : time-to-market & development time
The solution : ready-for-use SWAT DNA components
The layer : Active DNA, a lightweight, RESTful, SWAT
DNA client API translating REST calls into fully realized
Ruby objects
Pros
Complete abstraction of the component’s complexity.
It’s just Ruby, very small learning curve for devs.

Active DNA
Application

REST API
SWAT DNA Component

Active DNA
Application

REST Client

REST API
SWAT DNA Component

Active DNA
Application

Query Runner
REST Client

REST API
SWAT DNA Component

Active DNA
Application

Query Runner Query Translator
REST Client

REST API
SWAT DNA Component

Active DNA
Application

Component Layer
Query Runner Query Translator
REST Client

REST API
SWAT DNA Component

class ActiveDNA::ProfileEntry < ActiveDNA::Base

uses :profile

key :key, :suffix => :entry

depends_on :profile_id

attributes :type, :group, :level, :value => :json, :is_unique => :boolean

validates_presence_of :value, :message => quot;The value must be informedquot;
validates_presence_of :profile_id, :message => quot;Profile must be informedquot;
validates_presence_of :group, :message => quot;The group must be informedquot;

validates_inclusion_of :type,
:in => %w(boolean email float general guid integer url),
:message => quot;The type must be informedquot;

end

class RegistrationController < ApplicationController

def create_credential(profile_id)

returning credential = ActiveDNA::Credential.new do

credential.credential_identifier = self.identifier
credential.credential_type = self.identifier_type ||
IDENTIFIER_TYPE_EMAIL
credential.credential_status = quot;enabledquot;
credential.user_id = Profile.find(:first, :conditions => { :profile_id
=> profile_id }).user_id
credential.password = self.password if self.identifier_type ==
IDENTIFIER_TYPE_EMAIL
credential.save!

end

end

end

Technical Learnings
GIT
Distributed version control that works
No branches on SVN to dozens of branches on Git
Fine-grained control of revisions
Fewer conﬂicts
A branch per story

Technical Learnings
Multiple Environments
Local ➙ Sandbox
Integration (QA)
Staging ➙ Production
All referenced in Git branches
~$ sudo cap deploy

Technical Learnings

master

rc

stable

Technical Learnings
story-1
king
local
master

rc

stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master

rc

stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master
qa
rc

stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master
qa
rc
staging
stable production

Test all the f**king time!!!

TDD/BDD is a muscle, you need to exercise it
Without coverage, there is no understanding
Knowledge socialization
Continuous integration

Continuous Integration

Dedicated integration environment with automated
integration and tests execution with the most up to
dated code base.
Automated Reporting
Test Cases Results
Tests Coverage

Looking Forward
Queues are the new black : starling & others
Asynchronous is the newer black : workling & others
Erlang + Cloud = massive continuous computing space
Distributed objects : Distributed search
Predictive caching
Algorithms & Computer Science : latent semantics,
vector space model, etc

Three Lines on Scrum

Fastest way to go to nowhere
Scrum without a present Product Owner
Slowest way to go to somewhere
Scrum without a dedicated Scrum Master
Embrace it fully with the best team.

Valeu!!! ;-)
mlemos@webcointernet.com

http://brasigo.com.br/
http://blog.brasigo.com.br/

http://blogblogs.com.br/
http://blog.blogblogs.com.br/

Estamos contratando novos Jedis:

Envie seu CV para
vagas@blogblogs.com.br

ou deixe seus dados no
Lounge da Abril Digital

WebCo Way CParty 2009

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

More from Manoel Lemos

More from Manoel Lemos (8)

Recently uploaded

Recently uploaded (20)

WebCo Way CParty 2009

Editor's Notes