2. Team Lead
Scrum Master
Head of Open Source Labs
@ThorstenRinne
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 2
1th 1
3. Last year a new customer
came up with an idea...
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 3
1th 1
4. A Social Network!
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 4
1th 1
5. A Social Network?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 5
1th 1
6. for friends.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 6
1th 1
7. for business.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 7
1th 1
8. for videos.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 8
1th 1
9. for photos.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 9
1th 1
10. for real estate?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 10
1th 1
11. The idea:
Connect people, buildings
and businesses.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 1
1th 1 1
12. People
Building Business
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
13. People
... is living at ...
Building Business
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
14. People
... is living at ...
... is the home of ...
Building Business
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
15. People
... is living at ... ... is an employee of ...
... is the home of ...
Building Business
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
16. People
... is living at ... ... is an employee of ...
... is the home of ...
... has as employee ...
Building Business
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
17. People
... is living at ... ... is an employee of ...
... is the home of ...
... has as employee ...
Building Business
... has its office at ...
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
18. People
... is living at ... ... is an employee of ...
... is the home of ...
... has as employee ...
... is the office location of ...
Building Business
... has its office at ...
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 12
1th 1
19. We thought:
Nice idea, let‘s start!
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 13
1th 1
20. 4 Requirements
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 14
1th 1
21. 1
Agile software
development with Scrum
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 15
1th 1
23. 1 product owner (customer)
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 16
1th 1
24. 1 product owner (customer)
1 scrum master (myself)
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 16
1th 1
25. 1 product owner (customer)
1 scrum master (myself)
1 development team
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 16
1th 1
26. Scrum
Return
Gift wrap
Cancel
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
27. Scrum
Sprint
2 weeks
Return
Gift wrap
Cancel
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
28. Scrum
Sprint
2 weeks
Sprint goal
Return
Gift wrap
Cancel
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
29. Scrum
Sprint
2 weeks
Sprint goal
Return
Sprint
backlog
Gift wrap
Cancel
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
30. Scrum
Sprint
2 weeks
Sprint goal
Return
Sprint
Potentially shippable
backlog product increment
Gift wrap
Cancel
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
31. Scrum
Sprint
2 weeks
Sprint goal
Return
Sprint
Potentially shippable
backlog product increment
Gift wrap
Cancel Coupons
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
32. Scrum
Sprint
2 weeks
Sprint goal
Return
Sprint
Potentially shippable
Cancel backlog product increment
Coupons
Gift wrap
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
33. Scrum 24 hours
Sprint
2 weeks
Sprint goal
Return
Sprint
Potentially shippable
Cancel backlog product increment
Coupons
Gift wrap
Product
backlog
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 17
1th 1
34. 2
Scalability
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 18
1th 1
35. Classic web architecture
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 19
1th 1
36. Browser
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 20
1th 1
37. Browser
Appserver 1 Appserver 2
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 20
1th 1
38. Browser
Appserver 1 Appserver 2
DB Master DB Slave
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 20
1th 1
39. We built a simple, scalable
n-tier architecture.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 21
1th 1
57. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 39
1th 1
58. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
59. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
60. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
61. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
62. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
63. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
64. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
65. 3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
66. Multiple Frontnodes
3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
67. Multiple Frontnodes
3
Backend Sharding
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 40
1th 1
68. Infrastructure
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 41
1th 1
69. One Ubuntu package to
rule them all.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 42
1th 1
70. Development VMs
Jenkins CI Server
Selenium Server
Monitoring Server
Testing Server
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 43
1th 1
71. http://www.flickr.com/photos/nirak/644337888/
Only on production:
Frontend Nodes 11.04
Backend Nodes 10.04 LTS
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 44
1th 1
73. http://www.flickr.com/photos/nirak/644337888/
Mayflower DMZ
Developer Laptopamazon AWS
with Ubuntu VM
(Git repo + complete
Eucalyptus Cloud
software stack)
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 46
1th 1
74. http://www.flickr.com/photos/nirak/644337888/
Mayflower DMZ
Gitorious Server amazon AWS
Jenkins CI Server
Puppet Master
Eucalyptus Cloud
Selenium Server
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 47
1th 1
75. http://www.flickr.com/photos/nirak/644337888/
2 Frontnodes
3 Backendnodes
Mayflower DMZ
Development Monitor
Staging Monitor
amazon AWS
Eucalyptus Cloud
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 48
1th 1
76. http://www.flickr.com/photos/nirak/644337888/
Mayflower DMZ
2 Frontnodes amazon AWS
2 Backendnodes
Drupal Cloud
Eucalyptus
Live Monitor
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 49
1th 1
79. Application
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 52
1th 1
80. Do you remember?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 53
1th 1
81. Frontend 1 Frontend 2
Varnish Varnish
nginx nginx
PHP 5.3 PHP 5.3
Backendnode 1 Backendnode 2
Backendnode 3
MySQL Master MySQL Slave
RabbitMQ
MogileFS MogileFS
Apache Solr
Memcached Memcached
PHP 5.3
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 54
1th 1
82. Backendnode 1 Backendnode 2
Backendnode 3
MySQL Master MySQL Slave
RabbitMQ
MogileFS MogileFS
Apache Solr
Memcached Memcached
PHP 5.3
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 55
1th 1
83. Backendnode 3
RabbitMQ
Backendnode 1 Apache Solr
Backendnode 2
MySQL Master
MogileFS PHP 5.3
MySQL Slave
MogileFS
Memcached Memcached
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 55
1th 1
85. Apache Solr XML REST API
Profiles Profiles Profiles Autosuggest
DE EN i18n
Lucene Core Lucene Core Lucene Core Lucene Core
Various transformations such as
lowercasing, removing plurals, or
stemming to increase relevancy.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 57
1th 1
86. Search string: „mapolis“
LL(1) parser
„q=mapolis&sort=name desc&fl=name,id,country“
Search queries tokenized
with a LL(1) Parser
implemented in PHP.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 58
1th 1
87. Profile information
autosuggestions
user generated content
Apache
MySQL
Solr
Indexing is done with
asynchronous jobs with
RabbitMQ.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 59
1th 1
91. c
c
yn
n
sy
as
PHP
Publish Routes Consumes PHP CLI
Exchange Queue
Application Script
Apache
MySQL
Solr
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 63
1th 1
92. Why RabbitMQ?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 64
1th 1
93. Backendnode 1 Backendnode 2
Backendnode 3
MySQL Master MySQL Slave
RabbitMQ
MogileFS MogileFS
Apache Solr
Memcached Memcached
PHP 5.3
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 65
1th 1
94. Backendnode 1
MySQL Master
MogileFS
Memcached Backendnode 3
Backendnode 2
MySQL Slave
RabbitMQ
PHP 5.3
MogileFS
Memcached
Apache Solr
PHP 5.3
PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 65
1th 1
95. MogileFS
an open source distributed filesystem
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 66
1th 1
96. Application level
No single point of failure
Automatic file replication
Shared nothing architecture
Local filesystem agnostic
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 67
1th 1
97. Client
(API: Perl)
Trackers
Trackers Storage Nodes
Storage Nodes
Trackers Storage Nodes
(mogilefsd)
(mogilefsd) (mogstored)
(mogstored)
(mogilefsd) (mogstored)
Tracker‘s database
(MySQL)
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 68
1th 1
99. Application
Host 1 Host 2
R/W
MySQL Replication MySQL
Master Slave
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 70
1th 1
100. Application
Keepalive
Host 1 Host 2
W
R/W R
MySQL Replication MySQL
Master Slave
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 70
1th 1
101. Some notes on our
database architecture.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 71
1th 1
102. We use MySQL as
key/value storage.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 72
1th 1
103. We implemented all
relations in the application.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 73
1th 1
104. We already prepared the
database design for sharding
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 74
1th 1
105. Benefits:
Very simple, scalable and
flexible architecture
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 75
1th 1
106. Drawbacks:
More complexity in the
PHP code. But that‘s okay.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 76
1th 1
107. Frontend 1 Frontend 2
Varnish Varnish
nginx nginx
PHP 5.3 PHP 5.3
Backendnode 1 Backendnode 2
Backendnode 3
MySQL Master MySQL Slave
RabbitMQ
MogileFS MogileFS
Apache Solr
Memcached Memcached
PHP 5.3
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 77
1th 1
108. Frontend 1 Frontend 2
Varnish Varnish
nginx nginx
PHP 5.3 PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 78
1th 1
109. Frontend 2
Varnish
Frontend 1 nginx
PHP 5.3
Varnish
nginx
PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 78
1th 1
111. Doctrine 2
is a data-mapper,
not ActiveRecord.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 80
1th 1
112. We use Doctrine 2 as ORM.
And we think it‘s great.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 81
1th 1
113. /**
ORM with Doctrine 2
* @Entity
* @Table(name="role")
*
*/
class Mapolis_Entity_Role
{
/**
* @Id
* @Column(name="id", type="integer")
* @generatedValue(strategy="IDENTITY")
*/
protected $_roleId;
/**
* @Column(name="parent_id", type="integer")
* @var int
*/
protected $_parentId;
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 82
1th 1
114. Working with objects
/**
* Saves the personal interests of user.
*
* @param Mapolis_Entity_Person $profile
*
* @return void
*/
public function savePersonalInterests(
Mapolis_Entity_Person $profile
)
{
$this->getManager()
->getRepository(self::ENTITY_PERSON)
->update($profile);
}
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 83
1th 1
115. Working with objects
/**
* Deletes a building.
*
* @param Mapolis_Entity_Building $building
*
* @return boolean
*/
public function deleteBuilding($building)
{
$this->getManager()->remove($building);
return $this->getManager()->flush();
}
★ Transactional write-behind (batch writes)
★ Very fast for small UnitOfWorks
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 84
1th 1
116. Working with objects
/**
* Show recent invitations
*
* @return array of $count recent invitations
*/
public function recent($inviter, $count = 5)
{
$invitations = $this->getManager()
->getRepository(self::ENTITY_INVITATION)
->findBy(
array('_inviterId' => $inviter),
array('_modified' => 'DESC'),
$count
);
return $invitations;
}
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 85
1th 1
118. Why did we choose Zend
Framework?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 87
1th 1
119. 60 colleagues using Zend
Framework in other projects.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 88
1th 1
120. We use Zend Framework
more like a component
library.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 89
1th 1
121. We use Doctrine 2 instead
of Zend_Db_Table.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 90
1th 1
122. We built our own
Autoloader.
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 91
1th 1
123. What did we use from
Zend Framework?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 92
1th 1
124. Zend_Application
Zend_Controller
Zend_View
Zend_Validate
Zend_Translation
Zend_Locale
Zend_Cache
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 93
1th 1
125. Frontnode 1
Varnish
nginx
PHP 5.3
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 94
1th 1
126. PHP 5.3 from Ubuntu 11.04
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 95
1th 1
129. HTTP Request
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 98
1th 1
130. HTTP Request
GET mapolis.com 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 98
1th 1
131. HTTP Request
GET mapolis.com 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 98
1th 1
132. HTTP Request
GET mapolis.com 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 98
1th 1
133. HTTP Request
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 99
1th 1
134. HTTP Request
GET mapolis.com/de/start 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 99
1th 1
135. HTTP Request
GET mapolis.com/de/start 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 99
1th 1
136. HTTP Request
GET mapolis.com/de/start 80
Varnish HTTP accelerator
mapolis mapolis
Core Drupal
nginx on port 81 nginx on port 81
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 99
1th 1
137. Frontend
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 100
1th 1
139. JavaScript Framework
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 102
1th 1
140. Supported browsers
7 3.6 14 5.0
8 7 (latest) 5.1
9 (latest) (latest)
(10) (iOS)
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 103
1th 1
141. Development
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 104
1th 1
142. The Team:
1 Scrum Master
2 DevOps
4 PHP/JS Developers
1 Frontend Developer
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 105
1th 1
143. Development Tools
Used IDE:
Zend Studio, PhpStorm, vi and Textmate
Database change management:
Liquibase
CSS helper:
SASS / Compass
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 106
1th 1
144. Development Laptop
Ubuntu VM with
Local Git repo
Complete software stack
ant <everything>
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 107
1th 1
145. Some ant commands
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
146. Some ant commands
$ ant db-development
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
147. Some ant commands
$ ant db-development
$ ant sync
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
148. Some ant commands
$ ant db-development
$ ant sync
$ ant phpunit
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
149. Some ant commands
$ ant db-development
$ ant sync
$ ant phpunit
$ ant jslint
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
150. Some ant commands
$ ant db-development
$ ant sync
$ ant phpunit
$ ant jslint
$ ant deb
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
151. Some ant commands
$ ant db-development
$ ant sync
$ ant phpunit
$ ant jslint
$ ant deb
$ ant build-vm
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 108
1th 1
152. Typical commit workflow
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 109
1th 1
153. Typical commit workflow
$ vi Foo.php
$ ant phpunit
$ ant jslint
$ git add Foo.php
$ git commit -m ”Bar”
$ git pull
$ git push
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 109
1th 1
154. ant phpunit
<target
name="phpunit"
depends="db-test"
description="Run all tests">
<exec executable="phpunit"
failonerror="true">
<arg line="--configuration $
{basedir}/tests/phpunit-verbose.xml $
{basedir}/tests/AllTests.php" />
</exec>
</target>
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 1
1th 1 10
155. One click deployment
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 1 1
1th 1 1
156. And how does it look now?
Building a Cloud-based social network I Mayflower GmbH I 1 October 201 I 1
1th 1 12