How to use clustering and replicating cache to improve performance and high availability of WSO2 API Manager

Performance is very critical factor when it comes to real product deployments. So most of use use cache to avoid unnecessary web service calls database calls. In WSO2 API Manager api gateway is the most critical thing if we consider performance. In real production systems we separate api manager in to several parts based on their functionalities.
01. API Gateway (All api calls hits gateway validation throttling happen here)
02. API Keymgt (Security validation part)
01. API Store (Allows users to subscribe apis / allow self registrations)
01. API Publisher (Allow publishers to create and publish new API's)
01. API Registry (Control registry)

So as you see API Gateway hits all inbound calls to apis. So most of production systems we recommend users to have 2 nodes clustered instead having one node. In this case you have to configure load balancer carefully to distribute load between both nodes. If you are using F5 like hardware load balancer make it session aware to get optimum performance and if you are using WSO2 Elastic Load Balancer it will take care of session awareness as well as service awareness for you. Both nodes should share same configurations and db and most importantly cache and carbon context. Because carbon context carries throttling related sensitive information. Also we use cache to cache keys to avoid unnecessary web service calls.

So first we will we how to enable clustering for nodes. Replace clustering section in repository/conf/axis2/axis2.xml with following configuration.

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
        <parameter name="AvoidInitiation">true</parameter>       
        <parameter name="membershipScheme">multicast</parameter>
        <parameter name="domain">wso2.carbon.domain</parameter>
        <parameter name="synchronizeAll">true</parameter>
        <parameter name="maxRetries">10</parameter>
        <parameter name="mcastAddress">228.0.0.4</parameter>
        <parameter name="mcastPort">45564</parameter>
        <parameter name="mcastFrequency">500</parameter>
        <parameter name="memberDropTime">3000</parameter>
        <parameter name="localMemberPort">4000</parameter>
        <parameter name="preserveMessageOrder">true</parameter>
        <parameter name="atmostOnceMessageSemantics">true</parameter>
        <parameter name="properties">
      <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
        <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
        </parameter>
        <groupManagement enable="false">
       <applicationDomain name="apache.axis2.application.domain"
      description="Axis2 group"
     agent="org.apache.axis2.clustering.management.DefaultGroupManagementAgent"/>
        </groupManagement>    
        <nodeManager class="org.apache.axis2.clustering.management.DefaultNodeManager"
                     enable="true"/>  
        <stateManager class="org.apache.axis2.clustering.state.DefaultStateManager"
                      enable="true">
            <replication>
                <defaults>
                    <exclude name="local_*"/>
                    <exclude name="LOCAL_*"/>
                </defaults>
                <context class="org.apache.axis2.context.ConfigurationContext">
                    <exclude name="local_*"/>
                    <exclude name="UseAsyncOperations"/>
                    <exclude name="SequencePropertyBeanMap"/>
                </context>
                <context class="org.apache.axis2.context.ServiceGroupContext">
                    <exclude name="local_*"/>
                    <exclude name="my.sandesha.*"/>
                </context>
                <context class="org.apache.axis2.context.ServiceContext">
                    <exclude name="local_*"/>
                    <exclude name="my.sandesha.*"/>
                </context>
            </replication>
        </stateManager>
    </clustering>

Do this for both nodes. Refer  this article on cluster configurations if you need more information.

Now we need to enable replicate cache between two nodes.

1.In your setup caching synchronization should be done based on a predefined multicast ip and a port.
The caching synchronization is done by Jgroups multicasting - and you can specify a port and the address if needed as system properties by updating wso2server.sh.

-Djgroups.udp.mcast_addr="224.10.10.10" \
-Djgroups.udp.mcast_port="5555" \

Add following configurations to repository/conf/etc/cache.xml

<configuration>
        <clusterName>apimanager-cache</clusterName>
        <cacheMode>replicated</cacheMode>
        <sync>true</sync>
        <l1>
            <enabled>true</enabled>
            <lifespan>60000</lifespan>
        </l1>
    </configuration>

2. Specify valid<HostName>in carbon.xml
3. If you do not want to replicate user-core caches among nodes, add unique cache identifiers for each node as a child under the corresponding

<UserStoreManager>

(JDBCUserStoreManager or AD) of user-mgt.xml

<Property name="UserCoreCacheIdentifier">node2</Property>

i.e:- Since we use separate user stores for publisher and store, user-store caches should not be replicated among those nodes. Hence, we can add two different cache identifiers for each publisher and store nodes.
In store:

<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
<Property name="UserCoreCacheIdentifier">node2</Property>

In publisher:

<UserStoreManager class="org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager">
<Property name="UserCoreCacheIdentifier">node1</Property>

4. Start servers with -Dbind.address= system property

Repeat same for other node as well. If you done configurations propery you will see cluster joining messages in output log of api gateway nodes as follows.

[2012-09-04 16:22:39,265]  INFO - TribesMembershipListener New member 192.168.115.218:4000(wso2.carbon.domain) joined cluster.

[2012-09-04 16:22:41,061]  INFO - TribesClusteringAgent Local Member 192.168.115.218:40(wso2.carbon.domain)

[2012-09-04 16:22:41,062]  INFO - TribesUtil Members of current cluster

[2012-09-04 16:22:41,062]  INFO - TribesUtil Member1 192.168.115.218:4000(wso2.carbon.domain)

If you enabled logs for jgroup you will see following messages as well.

[2012-09-04 16:22:27,278]  INFO - JGroupsTransport ISPN000094: Received new cluster view: [sanjeewa-TECRA-M11-23170|2] [sanjeewa-TECRA-M11-23170, sanjeewa-TECRA-M11-37541, sanjeewa-TECRA-M11-17759]

Please note that keeping other configurations as default is fine. But you have to configure gateway, store, publisher, registry, keymgt in a way they aware about each others. With this type of deployment you will be able to achieve great performance. To fine tune API gateway node please refer my previous blog post which describe how improve response time of api calls.