Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics
1. Reducing the Operations Burden and
Increasing QoS with
WSO2 Platform Management
and Administration Tactics
Sanjaya Ratnaweera
Amani Soysa
2. Agenda
● Common Deployment Patterns
● Deployment Synchronization
● Registry Mounting
● Setting up a worker-manager separated Appserver cluster -
Demo/Hands-on
● Deployment Automation
● Server Monitoring
● Production Deployment Guidelines
● Common issues in Production Setups
5. Clustered Deployment
● User engages with the load balancer
● Highly available
● Concept of worker and manager nodes
6. Worker-Manager Separation ?
➔ Proper separation of concerns
◆ worker node - can ONLY deploy artifacts and read
configurations
◆ manager node - authorized to add new artifacts and
make configuration changes
➔ Lower memory foot in the worker nodes
◆ lesser OSGi bundles
➔ Improved security
◆ management nodes can be behind the internal firewall &
be exposed to clients running within the organization
only, while worker nodes can be exposed to external
clients.
7. Cloud Based Deployment
● Extension of the Clustered deployment
● Highly scalable
● Span across multiple IaaSs
8. Why Deployment synchronization ?
• Artifact distribution and deployment should be transparent to
the end-users
• Manual artifact copying is acceptable for some extent in
standalone product clusters.
• Will that work in an elastically scaling cloud environment ??
9. Deployment Synchronization
● Distribute deployment artifacts and related meta-data across
the cluster
● Make the cluster nodes consistent
● Automated synchronization based on SVN
● Management nodes commits and worker nodes check-out
12. Registry Mounting
● An approach to share registry space across cluster nodes
● Local Data repository
○ Should not be shared
● Configuration registry
○ Shared across multiple nodes of the same product
● Governance registry
○ Shared across product platform
14. Worker-Manager Separated Cluster -
Demo/Hands-On
● Setting up user management and registry databases
● Configuring the Elastic loadbalancer
● Configuring WSO2 AS as a management node
● Configuring WSO2 AS as a worker node
25. Server and Health Monitoring
● JMX based monitoring
○ JConsole, Graphite
● Message Tracing Tools
○ Wireshark, tcpmon
● System monitoring tools
○ Ganglia, Cacti, Nagios
● Alerting
○ Email, SMS, Phone-call alerts
26. Production Deployment Guidelines
● Changing the default administrator credentials
● Replacing SSL certificates/keys
● Hardening the Operating System
● Turning off unused services/ports
● Switching the default H2 registry database to a production
ready database
● Using secure vault to encrypt passwords in configuration files
● Tune database connection pools
28. Patching
● Follow the recommended procedure
● Automate the patch applying process
● Test the patch in dev and staging environments
○ Automated process is a must
● Update production servers and restart in a RR manner
● Maintain a log for patches
29.
30. Common errors in production
● IO errors - Connection reset by peer, Broken pipe, Too many
open files, I/O reactor has been shut down
● Database connection pool errors
● OutOfMemoryErrors
31. Production Troubleshooting Tips
● carbondump
○ WSO2_HOME/bin/carbondump.{sh,bat}
● Replicate the issue in staging
● Report JIRAs with detailed information
○ Upload carbondump output to the provided FTP server
○ Attach the relevant deployment artifacts and logs