1. Reducing the Operations Burden and
Increasing QoS with
WSO2 Platform Management
and Administration Tactics
Sanjaya Ratnaweera
Amani Soysa

2. Agenda
● Common Deployment Patterns
● Deployment Synchronization
● Registry Mounting
● Setting up a worker-manager separated Appserver cluster -
Demo/Hands-on
● Deployment Automation
● Server Monitoring
● Production Deployment Guidelines
● Common issues in Production Setups

3. WSO2 Product deployment patterns
● Standalone
● Clustered
● Cloud based

4. Standalone Deployment
● User engages with one or many independent nodes

5. Clustered Deployment
● User engages with the load balancer
● Highly available
● Concept of worker and manager nodes

6. Worker-Manager Separation ?
➔ Proper separation of concerns
◆ worker node - can ONLY deploy artifacts and read
configurations
◆ manager node - authorized to add new artifacts and
make configuration changes
➔ Lower memory foot in the worker nodes
◆ lesser OSGi bundles
➔ Improved security
◆ management nodes can be behind the internal firewall &
be exposed to clients running within the organization
only, while worker nodes can be exposed to external
clients.

7. Cloud Based Deployment
● Extension of the Clustered deployment
● Highly scalable
● Span across multiple IaaSs

8. Why Deployment synchronization ?
• Artifact distribution and deployment should be transparent to
the end-users
• Manual artifact copying is acceptable for some extent in
standalone product clusters.
• Will that work in an elastically scaling cloud environment ??

9. Deployment Synchronization
● Distribute deployment artifacts and related meta-data across
the cluster
● Make the cluster nodes consistent
● Automated synchronization based on SVN
● Management nodes commits and worker nodes check-out

11. Deployment Synchronization
<--Manager Node -->
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl>
<SvnUser>wso2</SvnUser>
<SvnPassword>wso2123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
<-- Worker Node -->
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl>
<SvnUser>wso2</SvnUser>
<SvnPassword>wso2123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

12. Registry Mounting
● An approach to share registry space across cluster nodes
● Local Data repository
○ Should not be shared
● Configuration registry
○ Shared across multiple nodes of the same product
● Governance registry
○ Shared across product platform

13. Registry Mounting

14. Worker-Manager Separated Cluster -
Demo/Hands-On
● Setting up user management and registry databases
● Configuring the Elastic loadbalancer
● Configuring WSO2 AS as a management node
● Configuring WSO2 AS as a worker node

16. Deployment Automation

17. Configuration Management

18. No matter how small you are
adapt yourself to configuration management

19. Abstracted Configuration
node001 {
appserver {
version => "5.2.1",
target => "/opt/appserver",
cluster_domain => "mgt",
}
}
node002 {
elb {
version => "2.1.0",
offset => "0",
target => "/opt/elb",
user => "root",
}
}

20. Patch Management

21. Orchestration

22. Powered by Puppet

24. https://github.com/wso2/Puppet-Modules

25. Server and Health Monitoring
● JMX based monitoring
○ JConsole, Graphite
● Message Tracing Tools
○ Wireshark, tcpmon
● System monitoring tools
○ Ganglia, Cacti, Nagios
● Alerting
○ Email, SMS, Phone-call alerts

26. Production Deployment Guidelines
● Changing the default administrator credentials
● Replacing SSL certificates/keys
● Hardening the Operating System
● Turning off unused services/ports
● Switching the default H2 registry database to a production
ready database
● Using secure vault to encrypt passwords in configuration files
● Tune database connection pools

27. Production Deployment Guidelines contd..
● Disabling HTTP access logs
● Tuning heap memory
● Tuning GC
● Running servers as background processes
○ nohup, YAJSW

28. Patching
● Follow the recommended procedure
● Automate the patch applying process
● Test the patch in dev and staging environments
○ Automated process is a must
● Update production servers and restart in a RR manner
● Maintain a log for patches

30. Common errors in production
● IO errors - Connection reset by peer, Broken pipe, Too many
open files, I/O reactor has been shut down
● Database connection pool errors
● OutOfMemoryErrors

31. Production Troubleshooting Tips
● carbondump
○ WSO2_HOME/bin/carbondump.{sh,bat}
● Replicate the issue in staging
● Report JIRAs with detailed information
○ Upload carbondump output to the provided FTP server
○ Attach the relevant deployment artifacts and logs

32. Sanjaya Ratnaweera
Blog: http://samudura.org
Follow: @sanjayar
Amani Soysa
Blog: http://sparkletechthoughts.
blogspot.com/
Follow: @poohdedoo

33. Thank You