Slides from our talk "Service discovery and configuration provisioning" presented by Mariusz Gil at PHP Benelux 2016 Apache Zookeeper or Consul are almost completely unknown in the PHP world, although its use solves a lot of typical problems. In a nutshell, they are a central services of provisioning configuration information, distributed synchronization and coordination of servers/processes. It simplifies the processes of application configuration management, so it is possible to change its settings and operation in real time (eg. feature flagging). During the presentation the typical cases of use of Zookeeper/Consul in PHP applications will be presented, both strictly web and workers running from the CLI.

1. Service discovery and
configuration provisioning
mariuszgil srcministry
Mariusz Gil

5. What is an application? Source code?

6. Is it a complete solution for client?

9. How to manage the configuration?

10. Old school approach…

11. <?php
define('DB_HOST', 'localhost');
define('DB_NAME', 'northwind');
define('DB_USER', 'root');
define('DB_PASS', 'your_password');
// This connection allows your application to be used for multi-lingual websites.
function _connect_to_mysql($is_utf8 = null)
{
$mysql_link = mysql_connect(DB_HOST, DB_USER, DB_PASS)
or die("Could not connect to database server");
mysql_select_db(DB_NAME, $mysql_link)
or die("Could not select database");
if (is_null($is_utf8))
{
/* This sets collation for the connection to utf8_general_ci
because it is the default collation for utf8.
This enables multi-lingual capability in database.
*/
mysql_query("SET NAMES 'utf8'");
}
return $mysql_link;
}

12. <?php
// ...
// ** MySQL settings - You can get this info from your web host ** //
define('DB_NAME', 'database');
/** MySQL database username */
define('DB_USER', 'username');
/** MySQL database password */
define('DB_PASSWORD', 'password');
/** MySQL hostname */
define('DB_HOST', 'localhost');
/** Database Charset to use in creating database tables. */
define('DB_CHARSET', 'utf8');
/** The Database Collate type. Don't change this if in doubt. */
define('DB_COLLATE', '');

13. <?php
/**
* The base configuration for WordPress
*
* The wp-config.php creation script uses this file during the
* installation. You don't have to use the web site, you can
* copy this file to "wp-config.php" and fill in the values.
*
* @link https://codex.wordpress.org/Editing_wp-config.php
* @package WordPress
*/
// ** MySQL settings - You can get this info from your web host ** //
define('DB_NAME', 'database');
/** MySQL database username */
define('DB_USER', 'username');
/** MySQL database password */
define('DB_PASSWORD', 'password');
/** MySQL hostname */
define('DB_HOST', 'localhost');
/** Database Charset to use in creating database tables. */
define('DB_CHARSET', 'utf8');
/** The Database Collate type. Don't change this if in doubt. */
define('DB_COLLATE', '');

14. bash-3.2$ ack DB_NAME
wp-includes/load.php
350: $wpdb = new wpdb( DB_USER, DB_PASSWORD, DB_NAME, DB_HOST );
wp-content/plugins/xcloner-backup-and-restore/restore/XCloner.php
580: $config_data = str_replace("define('DB_NAME', '",
"define('DB_NAME', ‚".$_REQUEST[mysql_db]."');
#", $config_data)
wp-content/plugins/w3-total-cache/lib/W3/Db.php
165: parent::__construct(DB_USER, DB_PASSWORD, DB_NAME, DB_HOST);

15. How we can improve it?

16. <?php
use PimpleContainer;
$container = new Container();
$container['db'] = function() {
$host = 'localhost';
$dbName = 'wordpress';
$user = 'root';
$pass = '';
return new PDO("mysql:host={$host};dbname={$dbName}", $user, $pass);
};

17. <?php
$dice = new DiceDice();
$rule = [
//Mark the class as shared so the same instance is returned each time
'shared' => true,
//The constructor arguments that will be supplied when the instance is created
'constructParams' => [
'mysql:host=127.0.0.1;dbname=mydb',
'username',
'password',
],
];
//Apply the rule to the PDO class
$dice->addRule('PDO', $rule);
//Now any time PDO is requested from Dice, the same instance will be returned
//And will have been constructed with the arguments supplied in 'constructParams'
$pdo = $dice->create('PDO');

18. <?php
$builder = new DIContainerBuilder();
$builder->addDefinitions([
'foo' => 'hello ',
'bar' => 'world',
]);
$builder->addDefinitions([
'foo' => DIdecorate(function ($previous, InteropContainerContainerInterface $container) {
return $previous . $container->get('bar');
}),
]);
$builder->addDefinitions([
'values' => [
'value 1',
'value 2',
],
]);
$builder->addDefinitions([
// with the same name
'stdClass' => DIobject('stdClass'),
// with name inferred
__NAMESPACE__ . 'ObjectDefinitionClass1' => DIobject(),
// with a different name
'object' => DIobject(__NAMESPACE__ . 'ObjectDefinitionClass1'),
]);
$container = $builder->build();

19. # Swiftmailer Configuration
swiftmailer:
transport: "%mailer_transport%"
host: "%mailer_host%"
username: "%mailer_user%"
password: "%mailer_password%"
spool: { type: memory }
# KNP Menu configuration
knp_menu:
twig:
template: knp_menu.html.twig
templating: false
default_renderer: twig
# KNP Paginator configuration
knp_paginator:
page_range: 5
default_options:
page_name: page
# LiipImagine configuration
liip_imagine:
filter_sets:
square_200:
quality: 100
filters:
thumbnail: { size: [200,
200], mode: outbound }
# Configuration parameters
parameters:
database_host: 127.0.0.1
database_port: null
database_name: database
database_user: root
database_password: null
mailer_transport: smtp
mailer_host: 127.0.0.1
mailer_user: null
mailer_password: null
secret: 092402938a039b72278b9b05da3f4d4

22. Is captcha feature enabled?
Is remember-be feature enabled?
Is remind-me feature enabled?
Is 2-factor authentication enabled?
Is brute-force protection enabled?
How many attempts per login allowed?
What is delay between failed logins?
Is hide-errors protection enabled?
…
Is login feature enabled at all?

23. What is database host?
What is database port?
What is database user?
What is database password?
What is database name?
What is database encoding?
Is database cache enabled?
What is cache host?
What is cache port?
Where the sessions are stored?

24. Standard feature
Single form
Many configuration problems

25. How to update configs?

26. mysql> DESC wp_options;
+--------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------------+------+-----+---------+----------------+
| option_id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| option_name | varchar(64) | NO | UNI | | |
| option_value | longtext | NO | | NULL | |
| autoload | varchar(20) | NO | | yes | |
+--------------+---------------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)
mysql> SELECT * FROM wp_options WHERE option_name LIKE '%enabled%' LIMIT 1;
+-----------+----------------------+--------------+----------+
| option_id | option_name | option_value | autoload |
+-----------+----------------------+--------------+----------+
| 87 | link_manager_enabled | 0 | yes |
+-----------+----------------------+--------------+----------+
1 row in set (0.00 sec)

27. You need to deploy application to
release changes in configuration

28. host A

29. host A host B

30. Another approach to maintain configs

31. Decouple your application from
hardcoded configuration details

32. by

33. Service discovery and
configuration provisioning tool

35. Service
discovery
Failure
detection
Key Value
storage
Multi DC
ready

36. host A host B host C host D
192.168.100.56
192.168.100.120

37. host A host B host C host D
192.168.100.56
192.168.100.120

38. host A host B host C host D
mysql.service.consul.
redis.service.consul.
agentserver agent agent

39. host A host B host C host D
mysql.service.consul.
redis.service.consul.
server agent agent agent

40. Managing services

41. # curl -X PUT -d '{
"service": {
"name": "memcached",
"tags": ["cache"],
"port": 11211,
"check": {
"name": "memcached status",
"script": "echo stats | nc 127.0.0.1 11211",
"interval": "5s"
}
}
}' http://192.168.33.10:8500/v1/catalog/service/memcached
# curl -X GET http://192.168.33.10:8500/v1/catalog/service/memcached
[{„Node":"vagrant-ubuntu-
trusty-64","Address":"192.168.33.10","ServiceID":"memcached","ServiceName":"memcached","ServiceTags
":["cache"],"ServiceAddress":"","ServicePort":11211,"ServiceEnableTagOverride":false,"CreateIndex":
19,"ModifyIndex":22}]

42. <?php
require '../vendor/autoload.php';
$sf = new SensioLabsConsulServiceFactory(array(
'base_url' => 'http://192.168.33.10:8500',
));
$catalog = $sf->get('catalog');
$response = $catalog->service('memcached')->json();
# Result data
array (
0 =>
array (
'Node' => 'vagrant-ubuntu-trusty-64',
'Address' => '192.168.33.10',
'ServiceID' => 'memcached',
'ServiceName' => 'memcached',
'ServiceTags' =>
array (
0 => 'cache',
),
'ServiceAddress' => '',
'ServicePort' => 11211,
'ServiceEnableTagOverride' => false,
'CreateIndex' => 19,
'ModifyIndex' => 60,
),
)

43. DNS interface for services and tags

44. <node>.node[.dc].<domain>

45. [tag.]<service>.service[.dc].<domain>

46. # dig @192.168.33.10 -p 8600 memcached.service.consul
; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> @192.168.33.10 -p 8600 memcached.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58136
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;memcached.service.consul. IN A
;; ANSWER SECTION:
memcached.service.consul. 0 IN A 192.168.33.10
;; Query time: 4 msec
;; SERVER: 192.168.33.10#8600(192.168.33.10)
;; WHEN: Fri Jan 29 16:40:37 UTC 2016
;; MSG SIZE rcvd: 82

47. # dig @192.168.33.10 -p 8600 memcached.service.consul SRV
; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> @192.168.33.10 -p 8600 memcached.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10299
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;memcached.service.consul. IN SRV
;; ANSWER SECTION:
memcached.service.consul. 0 IN SRV 1 1 11211 vagrant-ubuntu-trusty-64.node.dc1.consul.
;; ADDITIONAL SECTION:
vagrant-ubuntu-trusty-64.node.dc1.consul. 0 IN A 192.168.33.10
;; Query time: 4 msec
;; SERVER: 192.168.33.10#8600(192.168.33.10)
;; WHEN: Fri Jan 29 16:41:06 UTC 2016
;; MSG SIZE rcvd: 182

48. PHP Benelux Fries Example

49. Service 1 Service 2

50. # curl -X PUT -d '{
"Datacenter": "dc1",
"Node": "vagrant-ubuntu-trusty-64",
"Address": "192.168.33.10",
"Service": {
"ID": "fries-1",
"Service": "fries",
"Tags": [
"fries"
],
"Address": "192.168.33.10",
"Port": 1111
}
}' http://192.168.33.10:8500/v1/catalog/register
true
# curl -X PUT -d '{
"Datacenter": "dc1",
"Node": "vagrant-ubuntu-trusty-64",
"Address": "192.168.33.10",
"Service": {
"ID": "fries-2",
"Service": "fries",
"Tags": [
"fries"
],
"Address": "192.168.33.10",
"Port": 2222
}
}' http://192.168.33.10:8500/v1/catalog/register
true

51. # dig @192.168.33.10 -p 8600 fries.service.consul SRV
; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> @192.168.33.10 -p 8600 fries.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30707
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;fries.service.consul. IN SRV
;; ANSWER SECTION:
fries.service.consul. 0 IN SRV 1 1 1111 vagrant-ubuntu-trusty-64.node.dc1.consul.
fries.service.consul. 0 IN SRV 1 1 2222 vagrant-ubuntu-trusty-64.node.dc1.consul.
;; ADDITIONAL SECTION:
vagrant-ubuntu-trusty-64.node.dc1.consul. 0 IN A 192.168.33.10
vagrant-ubuntu-trusty-64.node.dc1.consul. 0 IN A 192.168.33.10
;; Query time: 4 msec
;; SERVER: 192.168.33.10#8600(192.168.33.10)
;; WHEN: Fri Jan 29 23:21:22 UTC 2016
;; MSG SIZE rcvd: 310

53. Storing values

54. # curl -X GET http://192.168.33.10:8500/v1/kv/foo
[{"LockIndex":0,"Key":"foo","Flags":128,"Value":"YmFy","CreateIndex":123,"ModifyIndex":130}]
# curl -X PUT -d 'bar' http://192.168.33.10:8500/v1/kv/foo
true
# curl -X GET http://192.168.33.10:8500/v1/kv/foo
[{"LockIndex":0,"Key":"foo","Flags":0,"Value":"YmFy","CreateIndex":123,"ModifyIndex":123}]
# curl -X PUT -d 'bar' http://192.168.33.10:8500/v1/kv/foo?flags=128
true

55. <?php
require '../vendor/autoload.php';
$sf = new SensioLabsConsulServiceFactory(array('base_url' => 'http://192.168.33.10:8500'));
$kv = $sf->get('kv');
$kv->put('foo', 'bar' . time());
$response = $kv->get('foo')->json();
$kv->delete('foo');
// Result data
array (
0 =>
array (
'LockIndex' => 0,
'Key' => 'foo',
'Flags' => 0,
'Value' => 'YmFyMTQ1NDA4NTQ3MA==',
'CreateIndex' => 123,
'ModifyIndex' => 147,
),
)

56. Distributed locks

57. <?php
require '../vendor/autoload.php';
$sf = new SensioLabsConsulServiceFactory(array('base_url' => 'http://192.168.33.10:8500'));
$session = $sf->get('session');
$kv = $sf->get('kv');
// Start a session
$sessionId = $session->create()->json()['ID'];
// Lock a key / value with the current session
$lockAcquired = $kv->put('session/a-lock', 'a value', ['acquire' => $sessionId])->json();
if (false === $lockAcquired) {
$session->destroy($sessionId);
echo "The lock is already acquire by another node.n";
exit(1);
}
# YOUR CRITICAL SECTION CODE HERE...
$kv->delete('session/a-lock');
$session->destroy($sessionId);

58. Useful to avoid dogpile effect

59. Even more useful with leader election

60. Possibilities...

63. Task #1
Replace one search engine to another
Without downtime

64. <?php
$backend = $userId % 100 < $settings->get('search/elastic/access_threshold', 0)
? $container->get('search.elastic')
: $container->get('search.sphinx');
$suggestions = $backend->search($query);
# curl -X PUT -d '5' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '10' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '15' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '25' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '50' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '75' http://192.168.33.10:8500/v1/kv/search/elastic/threshold
# curl -X PUT -d '50' http://192.168.33.10:8500/v1/kv/search/elastic/threshold

65. <?php
$engine = $userId % 100 < $settings->get('search/elastic/access_threshold')
? 'elastic'
: 'sphinx';
$backend = $container->get('search.' . $engine);
$minimalQueryLength = $settings->get('search.' . $engine . '.minimal_length', 3);
$maximalNumberOfSuggestion = $settings->get('search.' . $engine . '.suggestions_limits', 10);
if (length($query) >= $minimalQueryLength) {
$suggestions = $backend->search($query, $maximalNumberOfSuggestion);
}

66. Task #2
Add CPU/memory to the server
In the middle of the night

67. <?php
if ($settings->get('recommendations/enabled')) {
# Fetch recommendations for user based on request
# ...
}
# curl -X PUT -d '0' http://192.168.33.10:8500/v1/kv/recommendations/enabled
# halt
# ADD MORE CPU/MEMORY
# curl -X PUT -d '1' http://192.168.33.10:8500/v1/kv/recommendations/enabled

68. # curl -X PUT -d '{
> "Datacenter": "dc1",
> "Node": "vagrant-ubuntu-trusty-64",
> "ServiceID": "memcached"
> }' http://192.168.33.10:8500/v1/catalog/deregister
true
# curl -X PUT -d '{
> "Datacenter": "dc1",
> "Node": "vagrant-ubuntu-trusty-64",
> "Address": "192.168.33.10",
> "Service": {
> "ID": "memcached",
> "Service": "memcached",
> "Tags": [
> "cache"
> ],
> "Address": "192.168.33.10",
> "Port": 11211
> }
> }' http://192.168.33.10:8500/v1/catalog/register
true

69. <?php
require '../vendor/autoload.php';
$sf = new SensioLabsConsulServiceFactory(array(
'base_url' => 'http://192.168.33.10:8500',
));
$catalog = $sf->get('catalog');
$serviceDefinitions = $catalog->service('memcached')->json();
$memcached = new Memcache();
foreach ($serviceDefinitions as $serviceDefinition) {
$memcached->addserver($serviceDefinition['ServiceAddress'], $serviceDefinition['ServicePort']);
}

70. Long running background workers

73. Watch the Watches. React

74. host A host B host C host D
agentserver
web app
agent
background worker
agent
background worker
watch for event
make changes
& fire event

75. API methods

76. Agent
/v1/agent/checks
/v1/agent/services
/v1/agent/members
/v1/agent/self
/v1/agent/maintenance
/v1/agent/join/<address>
/v1/agent/force-leave/<node>>
/v1/agent/check/register
/v1/agent/check/deregister/<checkID>
/v1/agent/check/pass/<checkID>
/v1/agent/check/warn/<checkID>
/v1/agent/check/fail/<checkID>
/v1/agent/service/register
/v1/agent/service/deregister/<serviceID>
/v1/agent/service/maintenance/<serviceID>

77. Catalog
/v1/catalog/register
/v1/catalog/deregister
/v1/catalog/datacenters
/v1/catalog/nodes
/v1/catalog/services
/v1/catalog/service/<service>
/v1/catalog/node/<node>

78. Health Checks
/v1/health/node/<node>
/v1/health/checks/<service>
/v1/health/service/<service>
/v1/health/state/<state>

79. Network coordinates
/v1/coordinate/datacenters
/v1/coordinate/nodes

80. Key / Values
/v1/kv/<key>

81. Events
/v1/event/fire/<name>
/v1/event/list

82. ACL
/v1/agent/checks
/v1/agent/services
/v1/agent/members
/v1/agent/self
/v1/agent/maintenance
/v1/agent/join/<address>
/v1/agent/force-leave/<node>>
/v1/agent/check/register
/v1/agent/check/deregister/<checkID>
/v1/agent/check/pass/<checkID>
/v1/agent/check/warn/<checkID>
/v1/agent/check/fail/<checkID>
/v1/agent/service/register
/v1/agent/service/deregister/<serviceID>
/v1/agent/service/maintenance/<serviceID>

83. Prepared queries
/v1/query
/v1/query/<query>
/v1/query/<query or name>/execute

84. Sessions
/v1/session/create
/v1/session/destroy/<session>
/v1/session/info/<session>
/v1/session/node/<node>
/v1/session/list
/v1/session/renew

85. Server status
/v1/status/leader
/v1/status/peers

86. Alternative solutions

91. Choose one or implement your own
If you need it. Really need it

92. Thanks! Enjoy config management!
https://joind.in/talk/198e7
mariuszgil srcministry