Sanjeewa Malalgoda's Blog: 2012

Usage of WSO2 API Manager's login api

Login uri in API Manager gateway is acting as another API. Actually what it does is expose OAuth2 login functionality as API. Once we send request with /login as context gateway will send to https://localhost:9443/oauth2endpoints/token(key management nodes token end point) which is configurable. OAuth2 end point is responsible for all OAuth related functionality. Most common use case of using this end point is obtaining new user token. We can use any client application for this such as advanced rest client curl or any other application. Lets see how we can obtain new user token.

When we create new application we can get consumer key and consumer secret key pair. Also each user is getting application token which can be expire after some time. So if some user want to generate new user token for his application they can use this end point. Each user is having username and password. We have to use all these parameters to obtain new token.

If you are using advanced rest client set parameters as follows
URL - https://127.0.0.1:8243/login
Method - POST
Body - grant_type=password&username=admin&password=admin&scope=PRODUCTION
Also add following headers
01.Content-Type - application/x-www-form-urlencoded
02.Also add Basic Authorization header by setting consumer key as Username and consumer secret as Password

If you are using curl command use following instructions
generate following string by concatenating consumer key and secret key.
Consumerkey:consumersecret
Encode it as Base64 encoded string. It will give some string. For this example it would be KBjmo8mUxLCs_7l9CZXgg9fs6Pcap_P7bTvVxz7tXO2use4emXxENSga
Use following curl command to obtain new user token

curl -k -v -d "grant_type=password&username=username&password=password" -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization :Basic KBjmo8mUxLCs_7l9CZXgg9fs6Pcap_P7bTvVxz7tXO2use4emXxENSga" https://127.0.0.1:8243/login

From this service call you will get access token(token which can use to access application apis), refresh token(token which can use to refresh token) and validity period(validity period of issued token) inside response body. So that is the basic usage of login api of WSO2 API manager login api.

URL Template, URL Mapping and their usages in API Manager

Here in this post i wanted to discuss about the differences between url-template and url-mapping. Also its really important understanding query parameters and path parameters and their usages. This is very important when it comes to API management as each API resource map to some sort of url template or url mapping. Most API management tool do not allow you to define resources in a very specific way. All of them allow you to define very simple url patterns like url mapping does. But with that you cannot validate complex url patterns before it send to actual back end service. We can use ril-template for that type of complex mappings. Lets see how we can do it.

URL Template

A URL Template is a way to specify a URL that includes parameters that must be substituted before the URL is resolved. The syntax is usually to enclose the parameter in Braces ({example}). The convention is for a parameter to not be Percent encoded unless it follows a Question Mark (?). Advantage of url template is you can define path parameters inside url template definition. See following example. We can define url template with exact number of parameters and their locations inside url and etc. Also see sample url which maps to that url template.

uri-template="/view/{symbol,symbol1}/test/{symbol3,symbol4}/test1"

Sample URL /view/12,12/test/14,15/test1

If someone has url like

/view/12,12/test/14,15/test1/test2 or /view/12,12/test-change/14,15/test1

API gateway will simply reject that request saying no matching resource found. This is really usefull if you need to validate exact URL before it hits your API.

From next version of WSO2 API Manager(After 1.1.0) you will be able to define path parameters and query parameters both in url template. See following example.

uri-template="/view/{symbol,symbol1}/test/{symbol3,symbol4}/test1/*

This * mark says that you can have anything after test1/ so you can have any query paramaters after that point.

Sample URL /view/12,12/test/14,15/test1/data?name=sanjeewa&mood=hap

URL Mapping

URL Mapping allows users to create constant user friendly URLs and map them to resources. As API creator create the URLs, they can define human readable names for them. The self defined URLs can be published externally and thereby made available to API consumers. Most importantly you cannot have path parameters inside url mapping definition. You can have some sort for context for given resource. If you create API from WSO2 API Manager publisher UI it will automatically created API mapping for your API. If you need to define url template you have to go to sequence editor and edit it manually.

Query parameters and path parameters

Lets see what is query parameter and what is path parameter. In the World Wide Web, a query string is the part of a Uniform Resource Locator (URL) that contains data to be passed to web applications.

A URI path parameter is part of a path segment that occurs after its name. Path parameters offer a unique opportunity to control the representations of resources. If you have multiple parameters in between resources you can separate them by , or ;.

How to protect you'r API's from common attacks using WSO2 API Manager

WSO2 API Manager is a complete solution for publishing APIs, creating and managing a developer community and for scalably routing API traffic. It leverages proven, production-ready, integration, security and governance components from the WSO2 Enterprise Service Bus, WSO2 Identity Server, andWSO2 Governance Registry. In addition, as it is also powered by the WSO2 Business Activity Monitor, the WSO2 API Manager is ready for massively scalable deployment immediately.

Here in this article i will briefly describe how to protect your actual back end from higher loads of requests and some sort of DOS attacks. Most importantly you can secure, protect and shape up traffic using WSO2 API manager.

In a denial of service attack, the user can send several requests to actual back end service and overload it. Sometimes these requests may have have false return addresses, so the server can't find the user when it tries to send the response back. Also service tries to process so many requests at same time which causes to reduce performance and consume CPU and memory a lot. When server close the connection due to failure, the attacker sends a new batch of forged requests, and the process begins again--tying up the service indefinitely.

One of the more common methods of blocking a "denial of service" attack is to set up a filter or pattern recognition mechanism in front of server before request hits actual back end service. The filter can look for attacks by noticing patterns of incoming traffic. If a pattern comes in frequently, the filter can block messages containing that pattern.

So now we will see how we can achieve this from WSO2 API Manager. If you are familiar with API's you know what API does. If you are using WSO2 API Manager you can engage different throttling policy per each API. Lets say you have one back end which cannot handle 1000 requests same time or so. In that case you can limit concurrent access to that service using API manager. What you have to do is define a throttling policy saying number of concurrent requests and engage it to your API pointing to actual back end service.

01. Create throttling policy. See following throttling policy which allows 1000 concurrent requests to service.

<wsp:Policy xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
xmlns:throttle="http://www.wso2.org/products/wso2commons/throttle"
            wsu:Id="WSO2MediatorThrottlingPolicy">
    <throttle:MediatorThrottleAssertion>
        <throttle:MaximumConcurrentAccess>1000</throttle:MaximumConcurrentAccess>
        <wsp:Policy>
            <throttle:ID throttle:type="IP">other</throttle:ID>           
        </wsp:Policy>
    </throttle:MediatorThrottleAssertion>
</wsp:Policy>

02. Upload it as registry resource. For this you can use resource link available in management console of API manager. Go to goverence/apimgt/applicationdata path and upload created policy file. Please make sure to give same path for policy key in API definiton on next step

03. Engage policy to your API as follows. For this you can use source view of synapse configuration editor. Click on source view of API manager management console side menu then you can find configuration related to each and every API created. Add it to your API definition as follows. For example i will take login API.

<?xml version="1.0" encoding="UTF-8"?><api xmlns="http://ws.apache.org/ns/synapse"

name="_WSO2AMLoginAPI_" context="/login">
    <resource methods="POST" url-mapping="/*">
        <inSequence>
            <send>
                <endpoint>
                    <address uri="https://localhost:9493/oauth2/token"/>
                </endpoint>
            </send>
        </inSequence>
        <outSequence>
            <send/>
        </outSequence>
    </resource>
    <handlers>
 <handler class="org.wso2.carbon.apimgt.gateway.handlers.throttling.APIThrottleHandler">
       <property name="id" value="A"/>
       <property name="policyKey" value="gov:/apimgt/applicationdata/throttle.xml"/>
       </handler>

<handler class="org.wso2.carbon.apimgt.gateway.handlers.ext.APIManagerExtensionHandler"/>
    </handlers>
</api>

04. test the service

Other common use case is IP address based throttling. For this you may want to limit number of requests send by some client IP(Let say 10 call from single client). So we can use policy shown below. If client from 10.1.1.1 ip address it will allow only 1 API call per minute. If it is any other IP address it will allow only 2 API calls per minute. This is so cool. Isn't it? You can secure, protect, traffic shaping, load balancing using single product

<wsp:Policy xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"

xmlns:throttle="http://www.wso2.org/products/wso2commons/throttle">   
<throttle:MediatorThrottleAssertion>    
<wsp:Policy>            
<throttle:ID throttle:type="IP">10.1.1.1</throttle:ID>            
<wsp:Policy>                
<throttle:Control>                    
<wsp:Policy>                        
<throttle:MaximumCount>1</throttle:MaximumCount>                        
<throttle:UnitTime>60000</throttle:UnitTime>                    
</wsp:Policy>                
</throttle:Control>           
</wsp:Policy>        
</wsp:Policy>
     
<wsp:Policy>            
<throttle:ID throttle:type="IP">other</throttle:ID>            
<wsp:Policy>                
<throttle:Control>                    
<wsp:Policy>                        
<throttle:MaximumCount>2</throttle:MaximumCount>                        
<throttle:UnitTime>60000</throttle:UnitTime>                   
 </wsp:Policy>                
</throttle:Control>            
</wsp:Policy>        
</wsp:Policy>    
</throttle:MediatorThrottleAssertion></wsp:Policy>

WSO2 API Manager - How to use thrift for api validation call between Gateway and Keymgt servers

If we deployed API Manager in a distributed way we have separate keymgt server for validation and authentication purpose. Normally when some api call hits API gateway it do some security check. For that verification we pass access token, api, api version. So once any API call hits gateway it will extract those parameters and do check whether this token is valid one or not. Its very simple and straight forward. For that gateway calls to Keymgt server per each call if cache is not available at gateway side. As we all know web service call is always bit costly operation. So we thought its better if we can put thrift there. So thrift implementation for validation call will be available in next generally available release of API Manager 1.0.1 (not in 1.0.0 release).

So here in this post i will describe how to add configurations for that change. Here i will describe how should we do this for distributed setup. Let say we have 2 gateway nodes 2 keymgt nodes in four different machines. So ideally 2 thrift servers should run in both keymgt nodes and 2 clients should run on gateway nodes. So when api call comes gateway does a call to keymgt and load balancer sent it to one keymgt node based on load balancer algorithm.

If each node runs on different machines we can have identical configuration for both gateway nodes and same applies to keymgt as well. This

So Add following entries to APIKeyManager section of api-manager.xml file available in /repository/conf (gateway node). So ServerURL parameter says where is keymgt server is running in addition to that we have to specify thrift client port(it is the same as thrift server port of keymgt node). Actually its not necessary to run thrift server at gateway node but we will keep it as completeness.

&lt;ServerURL>https://192.168.113.209:9493/services/&lt;/ServerURL>

&lt;KeyValidatorClientType>ThriftClient&lt;/KeyValidatorClientType>
&lt;ThriftClientPort>10397&lt;/ThriftClientPort>
&lt;ThriftClientConnectionTimeOut>10000&lt;/ThriftClientConnectionTimeOut>
&lt;ThriftServerPort>10399&lt;/ThriftServerPort>

So Add following entries to APIKeyManager section of api-manager.xml file available in /repository/conf (keymgt node). Here important parameter is thrift server port as gateway tries to connect it.

&lt;KeyValidatorClientType>ThriftClient&lt;/KeyValidatorClientType>
&lt;ThriftClientPort>10398&lt;/ThriftClientPort>
&lt;ThriftClientConnectionTimeOut>10000&lt;/ThriftClientConnectionTimeOut>
&lt;ThriftServerPort>10397&lt;/ThriftServerPort>

So we have done the configurations. Then copy necessary thrift jar files to repository/component/dropins folder and restart both servers.If you want to go back to web service call change key validator client type in to WSClient

WSO2 API Manager advanced validation information caching configurations

If we deployed API Manager in a distributed way we have focus on few important things. Caching is one of the most important thing. Normally when some api call hits API gateway it do some security check. For that verification we pass access token, api, api version. So once any API call hits gateway it will extract those parameters and do check whether this token is valid one or not. Its very simple and straight forward. But as we know gateway hits all most all the traffic goes to actual apis. So this verification process should faster one. For that we are using cache. So we cache validation information object with the key containing token, api name and version. This cache can store at gateway side or keymgt side. Please note that these configurations are valid only after API Manager 1.0.0 release and its not available on 1.0.0 main release. Here we can consider 2 main scenarios.

01. Cache at gateway side.

If cache is available at gateway side when request hits gateway it first populate cached entry for given token and if it is not available on cache it will cal to keymgt server. Normally this happens through web service call. so once keymgt returns validation information we can store it in gateway. This is obviously fastest way of doing this as this reduces web service calls to keymgt server.

02. Cache at keymgt side.

For each call hits gateway it calls to keymgt server and there is a cache at keymgt side. So if entry is available in cache it returns else do database call and check the validity of the token. This method has known performance issue as it there is a web service call to keymgt server form gateway for each api call. But advantage of this method is we do not have to store any security related information at gateway side.

For each validation information object we have JWT token. Normally it also got cached with validation information object. But in some scenarios you might want to generate JWT per each call. So we can do that by using configuration. But please note that we can do this only when cache resides at keymgt side.

If you didn't set any of these configurations (by default) cache is available at gateway side and JWT is also cached with validation information object we are using this as default configuration as its the fastest way. If you need to change default configuration follow given instructions.

Move cache to keymgt server form gateway

Disable the caching at gateway side by adding following entry to APIGateway section of api-manager.xml file available in /repository/conf (gateway node)

<EnableGatewayKeyCache>false</EnableGatewayKeyCache>

Enable ketmgt server side caching by adding following entry to APIKeyManager section of api-manager.xml file available in /repository/conf (keymgt node)

<EnableKeyMgtValidationInfoCache>true</EnableKeyMgtValidationInfoCache>

If you want to disable or enable JWT caching at keymgt server side add following entry to APIKeyManager section of api-manager.xml file available in /repository/conf (keymgt node).

<EnableJWTCache>false</EnableJWTCache>

Also please note that you have to add following entry to the root level of api-manager.xml file available in /repository/conf (keymgt node) to generate JWT token at keymgt server side.

<APIConsumerAuthentication.EnableTokenGeneration>
true
</APIConsumerAuthentication.EnableTokenGeneration>

WSO2 API Manager Introduction-1

WSO2 API Manager is a complete solution for publishing APIs, creating and managing a developer community and for scalably routing API traffic. It leverages proven, production-ready, integration, security and governance components from the WSO2 Enterprise Service Bus, WSO2 Identity Server, andWSO2 Governance Registry. In addition, as it is also powered by the WSO2 Business Activity Monitor, the WSO2 API Manager is ready for massively scalable deployment immediately. Here in this post i will briefly describe key functionalities and terms in WSO2 API manager.

API Gateway
This is responsible for securing, protect, manage, and scale API calls. The API gateway is a simple API proxy which intercepts API requests and applies policies such as throttling and security checks. It is also instrumental in gathering API usage statistics. We are using set for handlers to security validation and throttling purposes. It will pass web service call to actual back end after these validation steps. If it is token request call then directly pass to keymgt server to handle it. We can add remove these handlers using source view of synapse configuration. When it comes to production deployments this server hits higher number of requests as all api calls goes through this server. So we have to handle this very carefully. Fine tuning and additional configuration changes according to environment is must to achieve highest performance

API Publisher
This enables API providers to easily publish their APIs, share documentation,provision API keys, and gather feedback on APIs features, quality and usage. You can create new apis by pointing to actual back end service and also define rate limiting policies available for this api.
Here are the some of publisher features.
Create, manage API and publish them to gateway.
Monitor API consumers, behaviour, response time, last accessed time and etc.
List the active subscriptions per each API.
Manage life cycles versions and policies. Based on the state of API you can set it to any of following state.

CREATED: API metadata has been added to API store, but it is not visible to subscribers
yet, nor deployed to the API gateway

PUBLISHED: API is visible in API store, and eventually ( if the “Push to Gateway” option is selected at publishing time)

DEPRECATED: API is still deployed into API gateway (available at runtime to existing users) but not visible to subscribers. An API can automatically be deprecated when a new version is published.

RETIRED: API is not unpublished from the API gateway and retired.

BLOCKED: Access is temporarily blocked.

API Store
Provides a space for consumers to self-register, discover APIs functionality, subscribe to APIs, evaluate them and interact with API publishers. Here users can come and view existing apis created and self sign in. After that they can create their own application by bundle multiple apis together to one application. Once user creates application he can generate 3 type of keys.
Application Token - This is security token that can use to call created application API’s
Consumer key, Consumer secret key – These key pair can use to generate a new access token for given application and user.
Later we will see how we can use these key to make actual API call. Here are the feature list for API store.
View top used, new featured APIs and search by name, tag or creator.
API rate, share comments, feature requests and participate forum discussions.
Create, manage application, add remove APIs to application.
Generate application keys consumer and consumer secret keys for application.
Download help and documents contact owners of API and ask questions.
Self-signup for API consumption.

API Key Manager Server
This is responsible for all security and key related operations. Normally when gateway hits API call it calls keymgt service and verify the validity of token provided. And if gateway gets login call it directly forward it to keymgt server. If we discuss little bit about this login call its basically use for get a new access token. So for this we have to pass user name, password, consumer key and consumer secret key you got when you register your application. All tokens use for validation are based on OAuth 2.0.0 protocol. All secure authorization of APIs is provided using the OAuth 2.0 standard for key management. The API gateway supports API authentication with OAuth 2.0, and it enables IT organizations to enforce rate limits and throttling policies for APIs by consumer.

There are 3 types of users we can consider in API story. Here are those. These roles are based on the privileges they have and tasks they perform.

Creator( The technical owner of an API)
A creator will typically be a person in a technical role who understands the technical aspects of the API (interfaces, documentation, versions, how it will be exposed by API gateway) and uses the API publisher web application to provision APIs into the API store. The creator will use the API store to consult ratings and feedback provided by API users. Creator can add APIs to the store but cannot manage their lifecycle

Publisher( The business owner of an API)
The publisher typically manages a set of APIs across the enterprise or business unit and controls the API lifecycle and monetization aspects. The publisher is also interested in usage patterns for APIs and as such has access to all API statistics.

Consumer(The developer or consumer of an API)
the consumer uses the API store to discover APIs, consult the documentation and forums as well as rate/comments on the API. He/she subscribes to APIs to obtain an API key. Consumers can bundle multiple API’s together into one application and use it. Actual application consumers are user devices it may be computer application or mobile application.
See the following diagram to understand this concept clearly.

How to enable/disable cache for API Key validation information and JWT token in WSO2 API Manager

<EnableGatewayKeyCache>false</EnableGatewayKeyCache>

So above configuration will disable API gateway side ley validation information cache. Then it will call to keymgt per each incoming request.

Also we can cache JWT token with api key validation information. We can do this in keymgt node. Lets see how we could do this. This is also important as JWT generation is bit costly operation.
Add the following lines under tag in api-manager.xml found under /repository/conf.

<EnableJWTCache>false</EnableJWTCache>

So above configuration disable JWT cache which means generate new JWT token per each call to keymgt server.

How to add user claims as custom header of login response of WSO2 API Manager

Sometimes users may want to get some user claims as response headers of login call response message. Let say if you have your own implementation of JDBC User store manager you might have different user claims. So here we will see how we can use configurations to retrieve them as headers. This is bit complex use case which might not use by each and every one. I did this post because we support this feature and for my future reference.

In the identity.xml add following section (with the claims you need)

<OAuth>.....</OAuth>

When you define the claim - DisplayName will appear as the HTTP Header name in the response.

<RequiredRespHeaderClaimUris>
<ClaimUri>http://wso2.org/claims/emailaddress</ClaimUri>
<ClaimUri>http://wso2.org/claims/gender</ClaimUri>
 </RequiredRespHeaderClaimUris>

Send curl request as follows :
curl -v -X POST -H "Authorization: Basic Z3pidExIS3NsTVU2dm82dXVmZExcvxcvcZDRU9ZYTpkUzFydFRSWjlVMnIwX241bzkyRk9WTlpldFVh" -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" -d "grant_type=password&username=admin&password=admin" http://localhost:8280/login/

This is the response.
< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8
< Cache-Control: no-store
< Email: test@test.com
< Gender: male
< Pragma: no-cache
< Server: WSO2 Carbon Server
< Date: Wed, 12 Sep 2012 09:54:30 GMT
< Transfer-Encoding: chunked

How to use clustering and replicating cache to improve performance and high availability of WSO2 API Manager

Performance is very critical factor when it comes to real product deployments. So most of use use cache to avoid unnecessary web service calls database calls. In WSO2 API Manager api gateway is the most critical thing if we consider performance. In real production systems we separate api manager in to several parts based on their functionalities.
01. API Gateway (All api calls hits gateway validation throttling happen here)
02. API Keymgt (Security validation part)
01. API Store (Allows users to subscribe apis / allow self registrations)
01. API Publisher (Allow publishers to create and publish new API's)
01. API Registry (Control registry)

So as you see API Gateway hits all inbound calls to apis. So most of production systems we recommend users to have 2 nodes clustered instead having one node. In this case you have to configure load balancer carefully to distribute load between both nodes. If you are using F5 like hardware load balancer make it session aware to get optimum performance and if you are using WSO2 Elastic Load Balancer it will take care of session awareness as well as service awareness for you. Both nodes should share same configurations and db and most importantly cache and carbon context. Because carbon context carries throttling related sensitive information. Also we use cache to cache keys to avoid unnecessary web service calls.

So first we will we how to enable clustering for nodes. Replace clustering section in repository/conf/axis2/axis2.xml with following configuration.

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
        <parameter name="AvoidInitiation">true</parameter>       
        <parameter name="membershipScheme">multicast</parameter>
        <parameter name="domain">wso2.carbon.domain</parameter>
        <parameter name="synchronizeAll">true</parameter>
        <parameter name="maxRetries">10</parameter>
        <parameter name="mcastAddress">228.0.0.4</parameter>
        <parameter name="mcastPort">45564</parameter>
        <parameter name="mcastFrequency">500</parameter>
        <parameter name="memberDropTime">3000</parameter>
        <parameter name="localMemberPort">4000</parameter>
        <parameter name="preserveMessageOrder">true</parameter>
        <parameter name="atmostOnceMessageSemantics">true</parameter>
        <parameter name="properties">
      <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
        <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
        </parameter>
        <groupManagement enable="false">
       <applicationDomain name="apache.axis2.application.domain"
      description="Axis2 group"
     agent="org.apache.axis2.clustering.management.DefaultGroupManagementAgent"/>
        </groupManagement>    
        <nodeManager class="org.apache.axis2.clustering.management.DefaultNodeManager"
                     enable="true"/>  
        <stateManager class="org.apache.axis2.clustering.state.DefaultStateManager"
                      enable="true">
            <replication>
                <defaults>
                    <exclude name="local_*"/>
                    <exclude name="LOCAL_*"/>
                </defaults>
                <context class="org.apache.axis2.context.ConfigurationContext">
                    <exclude name="local_*"/>
                    <exclude name="UseAsyncOperations"/>
                    <exclude name="SequencePropertyBeanMap"/>
                </context>
                <context class="org.apache.axis2.context.ServiceGroupContext">
                    <exclude name="local_*"/>
                    <exclude name="my.sandesha.*"/>
                </context>
                <context class="org.apache.axis2.context.ServiceContext">
                    <exclude name="local_*"/>
                    <exclude name="my.sandesha.*"/>
                </context>
            </replication>
        </stateManager>
    </clustering>

Do this for both nodes. Refer this article on cluster configurations if you need more information.

Now we need to enable replicate cache between two nodes.

1.In your setup caching synchronization should be done based on a predefined multicast ip and a port.
The caching synchronization is done by Jgroups multicasting - and you can specify a port and the address if needed as system properties by updating wso2server.sh.

-Djgroups.udp.mcast_addr="224.10.10.10" \
-Djgroups.udp.mcast_port="5555" \

Add following configurations to repository/conf/etc/cache.xml

<configuration>
        <clusterName>apimanager-cache</clusterName>
        <cacheMode>replicated</cacheMode>
        <sync>true</sync>
        <l1>
            <enabled>true</enabled>
            <lifespan>60000</lifespan>
        </l1>
    </configuration>

2. Specify valid<HostName>in carbon.xml
3. If you do not want to replicate user-core caches among nodes, add unique cache identifiers for each node as a child under the corresponding

<UserStoreManager>

(JDBCUserStoreManager or AD) of user-mgt.xml

<Property name="UserCoreCacheIdentifier">node2</Property>

i.e:- Since we use separate user stores for publisher and store, user-store caches should not be replicated among those nodes. Hence, we can add two different cache identifiers for each publisher and store nodes.
In store:

<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
<Property name="UserCoreCacheIdentifier">node2</Property>

In publisher:

<UserStoreManager class="org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager">
<Property name="UserCoreCacheIdentifier">node1</Property>

4. Start servers with -Dbind.address= system property

Repeat same for other node as well. If you done configurations propery you will see cluster joining messages in output log of api gateway nodes as follows.

[2012-09-04 16:22:39,265] INFO - TribesMembershipListener New member 192.168.115.218:4000(wso2.carbon.domain) joined cluster.

[2012-09-04 16:22:41,061] INFO - TribesClusteringAgent Local Member 192.168.115.218:40(wso2.carbon.domain)

[2012-09-04 16:22:41,062] INFO - TribesUtil Members of current cluster

[2012-09-04 16:22:41,062] INFO - TribesUtil Member1 192.168.115.218:4000(wso2.carbon.domain)

If you enabled logs for jgroup you will see following messages as well.

[2012-09-04 16:22:27,278] INFO - JGroupsTransport ISPN000094: Received new cluster view: [sanjeewa-TECRA-M11-23170|2] [sanjeewa-TECRA-M11-23170, sanjeewa-TECRA-M11-37541, sanjeewa-TECRA-M11-17759]

Please note that keeping other configurations as default is fine. But you have to configure gateway, store, publisher, registry, keymgt in a way they aware about each others. With this type of deployment you will be able to achieve great performance. To fine tune API gateway node please refer my previous blog post which describe how improve response time of api calls.

How to fine tune WSO2 API Manager gateway

Here in this article i will briefly describe how we can tune up gateway to get response with minimum time. Fine tuning is depend on the load on you system.

following configurations for get minimum response time when you have 350 to 30000 gateway calls per second(in a cluster with maximum 10 nodes).

Memory allocated for the server increase by modifying /bin/wso2server.sh with the following setting.

Default setting : -Xms256m -Xmx512m -XX:MaxPermSize=256m.
New setting : -Xms2048m -Xmx2048m -XX:MaxPermSize=1024m

NHTTP and PT HTTP transports

In /repository/conf/nhttp.properties set the following

http.socket.timeout=60000
snd_t_core=200
snd_t_max=250
snd_io_threads=16
lst_t_core=200
lst_t_max=250
lst_io_threads=16

Please note that number of threads above should equal to number of processor cores in your system.These parameters vary on the load on machine processing time and etc. So you can change them and observe how results behave. Based on observations you can decide optimum parameters. Also apply followings to keymgt node to get maximum out come for login requests.

Axis2Client.xml 
<parameter name="defaultMaxConnPerHost">1000</parameter> 
<parameter name="maxTotalConnections">30000</parameter> 

masterdatasource.properties 
<maxActive>250</maxActive> 
<testOnBorrow>false</testOnBorrow> 
<validationInterval>120000</validationInterval> 

Mysql maximum connections 
mysql> show variables like "max_connections"; 
 max_connections was 151 
 set to global max_connections = 250; 


CatlinaServer.sh 
maxThreads="750" 
minSpareThreads="150" 
disableUploadTimeout="false" 
enableLookups="false" 
connectionUploadTimeout="120000" 
maxKeepAliveRequests="600" 
acceptCount="600" 

Open Files Limit 
Set open files limit to 200000 by editing 
/etc/sysctl.conf 
sudo sysctl -p

Configure WSO2 ESB with jms, accept message to ESB and persist to database

Required
WSO2 ESB 4.0.3
Activemq-5.6.0

Steps
Download ActiveMQ from here http://activemq.apache.org/
Download WSO2 ESB from here http://wso2.com/products/enterprise-service-bus/
Download Mysql and connector jars from http://dev.mysql.com/downloads/
Setup ActiveMQ
Extract ActiveMQ to /opt folder
/opt/apache-activemq-5.6.0
run activeMQ using type command /opt/apache-activemq-5.6.0/bin/activemq

Setup WSO2 ESB
First we need to copy following activeMQ jars available in /opt/apache-activemq-5.6.0/lib into repository/lib folder of ESB distribution home. From here onwards i will refer that place as ESB_HOME.

Copy these 2 jars
geronimo-j2ee-management_1.1_spec-1.0.1.jar
activemq-core-5.6.0.jar

Since we are using mysql database download and copy mysql jar as well
mysql-connector-java-5.1.12-bin.jar

Now we need to configure jms transport from ESB. open ESB_HOME/repository/conf/axis2.xml file and Enable jms transport receiver by uncommenting following section

<transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
<parameter name="myTopicConnectionFactory" locked="false">
<parameter name="java.naming.factory.initial"
locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
<parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</
parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName"
locked="false">TopicConnectionFactory</parameter>
<parameter name="transport.jms.ConnectionFactoryType" locked="false">topic</
parameter>
</parameter>
<parameter name="myQueueConnectionFactory" locked="false">
<parameter name="java.naming.factory.initial"
locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
<parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</
parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName"
locked="false">QueueConnectionFactory</parameter>
<parameter name="transport.jms.ConnectionFactoryType"
locked="false">queue</parameter>
</parameter>
<parameter name="default" locked="false">
<parameter name="java.naming.factory.initial"
locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
<parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</
parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName"
locked="false">QueueConnectionFactory</parameter>
<parameter name="transport.jms.ConnectionFactoryType"
locked="false">queue</parameter>
</parameter>
</transportReceiver>

Enable jms transport sender by uncomment following configuration

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

So now we have enabled jms transport for WSO2 ESB.
Now we will define new proxy service to consume messages through jms transport. Create new proxy service as follows.

step5: edit out sequence(we don’t have to do anything here)

Now we have completed proxy service.Next we will see how to invoke this service.

First build sample
Go to ESB_HOME/samples/axis2Server/src/SimpleStockQuoteService
run ant command > ant
Next we have to start Axis2Service for that go to ESB/samples/axis2Server.
run command > sh axis2Server.sh

So now we have proxy service with following sub items
01.Log full message
02.Extract some parameter in message body using xpath expression
xmlns:m="http://services.samples"
xmlns:m0="http://schemas.xmlsoap.org/soap/envelope/" expression="//m0:Body/m:placeOrder/
m:order/m:symbol" type="VARCHAR"
03.Persist that parameter to database using db report mediator(We can write our own mediator
using class mediator to store message or part of it in any given store)
04.Send to some back end service or drop message(in this particular use case we can drop
message since we are not going to send it to actual back end anyway i have added that as well)
Here is the synapse configuration for proxy service.

<proxy name="StockQuoteProxy" transports="jms" startOnLoad="true" trace="disable">
<target>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
</endpoint>
<inSequence>
<log level="full"/>
<dbreport>
<connection>
<pool>
<password>root</password>
<user>root</user>
<url>jdbc:mysql://localhost/testdb</url>
<driver>com.mysql.jdbc.Driver</driver>
</pool>
</connection>
<statement>
<sql>call addmessage(?)</sql>
<parameter xmlns:m="http://services.samples" xmlns:m0="http://
schemas.xmlsoap.org/soap/envelope/" expression="//m0:Body/m:placeOrder/m:order/
m:symbol" type="VARCHAR"/>
</statement>
</dbreport>
<property name="OUT_ONLY" value="true"/>
</inSequence>
<outSequence><send/></outSequence>
</target>
<publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/xml</default>
</rules>
</parameter>
</proxy>

Create test_db and execute following commands to create table and stored procedure.

CREATE TABLE `testdb`.`messages` (
`message_id` int(10) NOT NULL AUTO_INCREMENT,
`message` text,
PRIMARY KEY (`message_id`)
) ENGINE=MyISAM AUTO_INCREMENT=44 DEFAULT CHARSET=latin1
CREATE DEFINER=`root`@`localhost` PROCEDURE `testdb`.`addmessage`(message_txt
VARCHAR(10))
INSERT INTO messages (message) values (message_txt)
Now we can invoke proxy service from the axis2 Client available inside wso2 ESB.
Goto ESB_HOME/samples/axis2Client and type following command.
>ant jmsclient -Djms_type=pox -Djms_dest=dynamicQueues/StockQuoteProxy -
Djms_payload=dddddd
So now you will see message log in ESB console and then persist extracted parameter to
database and message send to actual back end. If you want to drop message drop it from proxy
configuration.
ESB-log
=======
[2012-08-28 12:11:43,798] INFO - LogMediator To: http://localhost:9000/services/
SimpleStockQuoteService, WSAction: urn:placeOrder, SOAPAction: urn:placeOrder,
MessageID: urn:uuid:ed0de324-79eb-4101-9a0b-c201971910fd, Direction: request
To : http://localhost:9000/services/SimpleStockQuoteService
WSAction : urn:placeOrder
SOAPAction : urn:placeOrder
MessageID : urn:uuid:ed0de324-79eb-4101-9a0b-c201971910fd
Body : 70.25</
m0:price>200MSFT</
m0:placeOrder>
[2012-08-28 12:11:43,801] INFO - PlaceStockOrderTask placed order symbol:MSFT
quantity:200 price:70.25
[2012-08-28 12:11:45,213] INFO - LogMediator To: , MessageID: ID:sanjeewa-TECRA-
M11-47367-1346181105012-1:1:1:1:1, Direction: request, Envelope: encoding='utf-8'?>

150.33783033599516
6095
MSFTddiiActual axis2 backend log
==================
Tue Aug 28 12:11:45 PDT 2012 samples.services.SimpleStockQuoteService :: Accepted order
#76 for : 6095 stocks of MSFTddii at $ 150.33783033599516
Table entry in database
=================
If you go to database and see there will be entry for stock symbol you entered