[API World 2021 ] - Understanding Cloud Native Deployment
WSO2 Dep Sync for Artifact Synchronization of Cluster Nodes
1. DepSync for Deployment Artifact
Synchronization Between the Cluster
Nodes
Pradeep Fernando Kasun Gajasinghe
Senior Software Engineer Software Engineer
pradeep@wso2.com kasung@wso2.com
2. Agenda
• The need for deployment synchronization
• Brief introduction to WSO2 product clusters
• Demo - Complete Application server cluster with DepSync +
fronted by WSO2 ELB
3. Why Deployment Synchronization?
• Artifact distribution and deployment should be transparent to
the end-users
• Manual artifact copying is acceptable for some extent in
standalone product clusters.
• Will that work in an elastically scaling cloud environment ??
4. Typical WSO2 Product Cluster
• Manager and Worker nodes.
• Manager is for administration purposes - artifact upload,
applying security,etc
• Worker nodes responsible for serving requests.
• The setup is only valid for products that support artifact
deployment:
• Application Server
• Enterprise Service Bus
• etc
6. Why Management & Worker Node Separation
Crash Course
• Proper separation of concerns - management nodes
specialize in management of the setup while worker
nodes specialize in serving requests to deployment
artifacts
• Only management nodes are authorized to add new
artifacts into the system or make configuration changes
• Worker nodes can only deploy artifacts & read
configuration
• Lower memory foot in the worker nodes because the
management console related OSGi bundles are not loaded
• Improved security - management nodes can be behind the
internal firewall & be exposed to clients running within
the organization only, while worker nodes can be exposed
to external clients.
7. Deployment Synchronization
• Distributing deployment artifacts & related metadata to
all nodes in a homogeneous cluster is a typical
requirement for a clustered deployment of any
middleware platform
• Provides a consistent, and reliable cluster
• Automated synchronization without need of any user
interaction
• Artifact Metadata contains service policies, and other
important service configuration details
8. Deployment Synchronizer
• WSO2 products provides this through Deployment
Synchronizer (DepSync) mechanism
• A new feature addition to WSO2 products
• Synchronization done using a central repository, usually,
a SVN server
• We call it SVN-based Deployment Synchronizer
• Extensible - may be a Git based one in the future.
10. What it synchronizes?
• Web Applications
• JAXWS / JAXRS Applications
• Proxy Services
• Data Services
• BPEL
• Basically all the deployable artifacts in axis2 repo of
Carbon products (ex. repository/deployment/server)
• Service Metadata – (details on policies, transports etc.)
12. Multicast vs. WKA
Multicast
• Cluster is going to be set up in a network where
multicasting is allowed
Well-Known Address
• Cloud based deployment
• Members are distributed across datacenters & regions
• Multicasting blocked
13. DepSync and Multi-Tenancy
• DepSync synchronizes the artifacts among the product
cluster in a tenant aware manner.
• Ghost deployment is supported in the depsync aware
manner in order reduce the initial deployment and
request serving latencies.
16. Process
1. Set up the Environment for Deployment Synchronizer
2. Configure Load Balancer
3. Configure Management Node - Application Server
4. Configure Worker Node - Application Server
17. PART 1
Set up the Environment for
Deployment Synchronizer
18. Set up the Environment for Deployment
Synchronizer
• We will setup the SVN-based DepSync
• For this, you need the following
o svnkit osgi bundle
o TortoiseSVN / Silk SVN (Windows), Subversion
command line package(Unix) -optional
19. Set up the Environment for Deployment
Synchronizer
• Download svnkit bundle from here - http://goo.gl/CVR2F
• If you will be using a remote svn location for depsync, you may
skip the creation of local svn repository.
• To create a local svn repository,
o if you have svn command line client, enter following
command.
svnadmin create C:demosvnrepo
o if you are using TortoiseSVN, then first create a folder at
any place you like. Say - C://demo/svnrepo . Now go in to
that folder, right-click -> TortoiseSVN -> "Create Repsitory
here"
o Now, the SVN URL for this location is -
file:///C:/demo/svnrepo
20. Set up the Environment for Deployment
Synchronizer
• We will use this svn url when configuring the manager
and worker nodes. So, remember the location where you
created this. DepSync config will also be configured there.
23. Configure Load Balancer -
Loadbalancer.conf
• Download and extract wso2elb-2.0.0.zip
o Let the extracted directory be WSO2_LB_HOME
• Open loadbalancer.conf file in the
WSO2_LB_HOME/repository/conf
24. Configure Load Balancer
• Configuring the WSO2 Elastic Load Balancer is now
complete
• To start the server
o Go to the WSO2_LB_HOME/bin folder using command
line
o Enter the command `wso2server.bat`
o Notice the logs printed by TribesClusteringAgent
26. Configure Management Node
• Download and extract wso2as-5.0.0.zip
o Rename extracted directory to wso2as-5.0.0-manager
o Let the extracted directory be WSO2_AS_MGR_HOME
27. Configure Management Node -
axis2.xml
• Enable clustering at axis2 level.
<clustering
class="org.apache.axis2.clustering.tribes.TribesClusteringAgent"
enable="true">
• Change the membershipScheme of clustering to wka (well-
known address)
<parameter name="membershipScheme">wka</parameter>
• Set the cluster domain. This must be the same which we
defined in loadbalancer.conf
<parameter name="domain">wso2.as.domain</parameter>
28. Configure Management Node -
axis2.xml
• Set localMemberHost
<parameter
name="localMemberHost">mgt.appserver.wso2.com</parameter>
• Set localMemberPort
<parameter name="localMemberPort">4250</parameter>
• Add a new property named "subDomain"and set it to
'mgt' inside the parameter "properties"
<parameter name="properties">
<property name="backendServerURL"
value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="mgt"/>
</parameter>
29. • Add load balancer as a well-known member.
o Use the IP/hostname and port defined in LB's
axis2.xml
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4000</port>
</member>
</members>
30. Configure Management Node –
catalina-server.xml
• Since the mgt node is fronted by an LB, we need to
configure
proxy ports associated with HTTP and HTTPS connecters.
o Defaults for http and https are 8280 and 8243
respectively.
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="8280“
---
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443“
proxyPort="8243“
---
31. Configure Management Node –
carbon.xml
• Set port offset to avoid port conflicts with other nodes
(LB).
<Offset>1</Offset>
• Update mgtHostName and HostName elements in
carbon.xml as shown below
<HostName>appserver.wso2.com</HostName><MgtHostName>mgt.appserver
.wso2.com</MgtHostName>
• Copy svnkit bundle to repository/components/dropins
32. Configure Management Node -
Deployment Synchronizer
• The deployment synchronizer config is in carbon.xml.
• Uncomment the DeploymentSynchronizer xml segment.
• Set AutoCommit to true.
• Set the SVNUrl
• No need to set username/password if you didn't set it for local
svn repo
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>file:///C:/webinar-setup/demo/final/svnrepo</SvnUrl>
<SvnUser>username</SvnUser>
<SvnPassword>password</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
33. Configure Management Node
• Configuring the WSO2 Application Server is now complete
• To start the server
o Go to the WSO2_AS_MGR_HOME/bin folder using
command line
o Enter the command `wso2server.bat`
o Notice the logs printed by TribesClusteringAgent
and corresponding cluster joining logs in load
balancer.
• You can now login to management console via
https://mgt.appserver.wso2.com:8243/carbon/
• Notice that requests we sent to services (including WSDL)
will be directed to worker nodes.
36. Configure Worker Node
• We can use a copy of a management node, and convert it
to a worker node easily since it will have most of the
settings
• So, instead of using a fresh appserver copy, make a copy of
the management node we just created
• Rename the new folder as wso2as-5.0.0-worker
• Let this directory be WSO2_AS_WORKER_HOME
37. Configure Worker Node -
axis2.xml
• In addition to the settings we already have, do the
following.
• Set the localMemberPort to 4251
<parameter name="localMemberPort">4251</parameter>
• This node belongs to the "worker"subDomain, so, change
the property "subDomain" to worker (instead of mgt)
<parameter name="properties">
<property name="backendServerURL"
value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL"
value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="worker"/>
</parameter>
38. Configure Worker Node –
carbon.xml
• First set a new port offset
<Offset>2</Offset>
• Comment out/remove the mgtHostName configuration
• Set the AutoCommit property inside Deployment Synchronizer
to false. This MUST be done because the worker nodes of a
cluster should NOT commit (write) artifacts
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>file:///C:/webinar-setup/demo/final/svnrepo</SvnUrl>
<SvnUser>username</SvnUser>
<SvnPassword>password</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
39. Configure Worker Node –
carbon.xml
•
• Now, it's time to start the server.
o NOTE: The workerNode system property must be set
when starting the worker nodes every time.
wso2server.bat -DworkerNode=true
40. Testing the cluster
There are many ways to test our cluster deployment. Let's
follow a simpler path.
• Log in to the management console of Application Server
management node
• Deploy a new Axis2 Web service (Go to Manage --> Axis2
Services --> Add --> AAR service)
• Once the service is deployed in the management node go
to the services list and click on Tryit
• Invoke the service
