Tuesday, September 4, 2018

Batch API - API chaining scenario


In this example, we have two back­end services and one proxy service. WSDLs for the
examples can be found in the  zip file attached. This service is in the WSO2 API manager and it is exposed to customers. This service accepts
an ID and a credit amount for its credit operation. A request coming to this service is served by
two back­end services.
PersonInfoService
The PersonInfoService provides the name and address information about a requestor when the
ID is given. So, this is the first service being called by the CreditProxy service deployed in the API
The CreditService is the actual service that does the crediting. It is called by the CreditProxy
after getting the required information from the PersonInfoService.
The zip file attached which contains all the necessary configurations for the scenario. The
scenario needs WSO2 API Manager and WSO2 Application Server. The files in the zip should
be copied or deployed into these servers. Now, let's look at the contents of the zip file and
how to use them to create the scenario.
Please use this[1] link to download all resources for this example.
[1]https://drive.google.com/file/d/0B3OmQJfm2Ft8ZWRxRlUxd1VTeXM/edit?usp=sharing

Back-End services
Deploy the esb-samples-1.0-SNAPSHOT.jar as a jar service in to the WSO2 App Server. This
jar has two POJO services. If you need to try this with your backend services you can try
that as well. The required classes that need to be exposed as web services are
● org.wso2.esb.samples.CreditService
● org.wso2.esb.samples.PersonInfoService

API Manager Configuration
1. Copy the contents of following folders in the sample zip to the
repository/deployment/server/synapse-configs/default/ of WSO2 API manager pack.
            /local­entries/xslt.xml
/proxy­services/CreditProxy.xml
            /endpoints/CreditEpr.xml
            /endpoints/PersonInfoEpr.xml
/sequences/creditSeq.xml
/sequences/personInfoSeq.xml
Here i have attached API configs as well. But i recommend you to create 2 APIs for
services using API publisher UI. Both of them should be pointed to actual services hosted in
WSO2 Application Server.
2. Copy the personToCredit.xslt in the sample zip to wso2am-1.6.0/resources/
directory of WSO2 API Manager.
3. Copy the CreditProxy.wsdl in the sample zip to the wso2am-1.6.0/resources/
directory of the WSO2 API Manager.
4. If you are running the WSO2 API Manager in the sample machine as the WSO2
Application Server, change the ports of the WSO2 API Manager in
respository/conf/transport-mgt.xml. Otherwise there will be port conflicts.
Now we have the complete configuration for the scenario, let's go through the API Manager
configuration step by step to understand how it works.
Here we have one main entry point(credit Proxy or credit API).
● Receive a request with only credit amount and ID of the requestor.
● Send the id to the PersonInfoService to get the Address and Name of the requestor
● Use the credit amount, ID, address and name to create a request to the credit service
and call the CreditService
The request to the proxy service is:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:sam="http://samples.esb.wso2.org">
    <soapenv:Body>
        <sam:credit>
            <sam:id>99990000</sam:id>
            <sam:amount>1000</sam:amount>
        </sam:credit>
    </soapenv:Body>
</soapenv:Envelope>
The information in the original request is required to call the two back­end services. When we do
the first request, the information from the original request will be lost if we don’t preserve it. So,
we need to store the required information from the original request in the message context. In
this case, we are only going to store the ID and the credit amount in the message context. But, it
is possible to store any part of the message in the context. For example, in some scenarios, the
whole message is stored.
We store the request ID and amount by extracting them using property mediator with XPath.
<property xmlns:sam="http://samples.esb.wso2.org" name="ORG_ID"
expression="//sam:credit/sam:id"/>
<property xmlns:sam="http://samples.esb.wso2.org" name="ORG_AMOUNT"
expression="//sam:credit/sam:amount"/>
After this step, the ID and the credit amount are available in the Message Context properties
named ORG_ID and ORG_AMOUNT respectively.
Now, we are ready to do the first request to the PersonInfoService. We need the XML for this
request and, so, will insert it into the message using the Enrich Mediator.
    <source type="inline" clone="true">
        <sam:get xmlns:sam="http://samples.esb.wso2.org">
        <sam:id>?</sam:id>
        </sam:get>
    <target type="body"/>
Then we will log message details and send request to PersonInfoEpr end point. Also please note
that we have set receive sequence here. Then response will directly goto mentioned
sequence(personInfoSeq). See following configuration.
<log level="full">
                    <property name="sequence" value="inSequence ­ request for PersonInfoService"/>
Send request to PersonInfoEpr
                <send receive="personInfoSeq">
                    <endpoint key="PersonInfoEpr"/>
                </send>
Here you will see personInfoSeq configuration. There we have used xslt transformation to
convert message to desired format for credit service. There also you will noticed that response
will direct to creditSeq. If you need to do some change for final response message you are free
    <sequence name="personInfoSeq">
        <xslt key="xslt">
    <property name="amount" expression="get­property('ORG_AMOUNT')"/>
        <property name="Action" value="urn:credit"/>
        <send receive="creditSeq">
            <endpoint key="CreditEpr"/>
Here is the creditSeq configuration. There we log message and send response to client directly.
    <sequence name="creditSeq">
        <log level="full"/>
So this will cover complete service chaining scenario using batch API. According to your specific
scenarios you might have to add more complex logics. I have tested this scenario with WSO2
API Manager 1.5.0 and Application Server 5.2.1.
When you invoke this proxy service create 2 APIs to credit service and personal information
service. Then subscribe both of them to single Application.
Then generate tokens for that application and invoke proxy service with that(for this example i
used proxy service if you need create API for that).
Inside proxy service we will extract auth headers and store it inside message context then we
use that token to each and every following API calls. See following configuration.
<property name="Auth" expression="get­property('transport','Authorization')"/>
We used above to extract auth header and store it for following API calls. This is how we inject
auth headers for each API call.
<property name="Authorization" expression="get­property('Auth')" scope="transport"
type="STRING"/>
If you enabled API manager wire logs you will see following complete logs for each incoming and
outgoing messages. When you try this sample always set your API  urls to eprs and use them
inside your configuration. Then once you changed API with newer version or url do don't have to
change all the places.
[2014­02­05 15:38:45,775]  INFO ­ SequenceDeployer Sequence: main has been updated from the file:
/home/sanjeewa/work/workflow/wso2am­1.6.0­1/repository/deployment/server/synapse­configs/default/sequences/main.xml
[2014­02­05 15:38:46,684] DEBUG ­ wire >> "POST /services/CreditProxy.CreditProxyHttpSoap12Endpoint HTTP/1.1[\r][\n]"
[2014­02­05 15:38:46,684] DEBUG ­ wire >> "Content­Type: application/soap+xml; charset=UTF­8; action="urn:credit"[\r][\n]"
[2014­02­05 15:38:46,684] DEBUG ­ wire >> "Cookie: menuPanel=visible; menuPanelType=main;
region2_humantask_menu=visible; region2_bpel_instances_menu=visible; region3_registry_menu=none; i18next=en­US;
JSESSIONID=1579A304FBF0FE9AA2BEE6B9A7C9F85A;
requestedURI="../../carbon/service­mgt/index.jsp?region=region1&item=services_list_menu";
current­breadcrumb=manage_menu%2Cservices_menu%2Cservices_list_menu%23;
MSG13915816906030.6012256733392928=true; region1_configure_menu=none; region4_monitor_menu=none;
region5_tools_menu=none[\r][\n]"
[2014­02­05 15:38:46,684] DEBUG ­ wire >> "User­Agent: Axis2[\r][\n]"
[2014­02­05 15:38:46,684] DEBUG ­ wire >> "Host: sanjeewa­ThinkPad­T530:8280[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "130[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><p:credit
xmlns:p="http://samples.esb.wso2.org"><!­­0 to 1 occurrence­­><p:id>33</p:id><!­­0 to 1
occurrence­­><p:amount>2</p:amount></p:credit></soapenv:Body></soapenv:Envelope>[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "0[\r][\n]"
[2014­02­05 15:38:46,685] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,688]  INFO ­ LogMediator To: /services/CreditProxy.CreditProxyHttpSoap12Endpoint, WSAction: urn:credit,
SOAPAction: urn:credit, MessageID: urn:uuid:4f418b82­d739­4cc8­bce1­f1fbd54659b3, Direction: request, sequence =
inSequence ­ request for CreditProxy, Envelope: <?xml version="1.0" encoding="utf­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><p:credit
xmlns:p="http://samples.esb.wso2.org"><!­­0 to 1 occurrence­­><p:id>33</p:id><!­­0 to 1
occurrence­­><p:amount>2</p:amount></p:credit></soapenv:Body></soapenv:Envelope>
[2014­02­05 15:38:46,690]  INFO ­ LogMediator To: /services/CreditProxy.CreditProxyHttpSoap12Endpoint, WSAction: urn:credit,
SOAPAction: urn:credit, MessageID: urn:uuid:4f418b82­d739­4cc8­bce1­f1fbd54659b3, Direction: request, sequence =
inSequence ­ request for PersonInfoService, Envelope: <?xml version="1.0" encoding="utf­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><sam:get
xmlns:sam="http://samples.esb.wso2.org">
                            <sam:id>33</sam:id>
                        </sam:get></soapenv:Body></soapenv:Envelope>
[2014­02­05 15:38:46,691] DEBUG ­ wire << "POST /services/PersonInfoService/ HTTP/1.1[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "Cookie: menuPanel=visible; menuPanelType=main;
region2_humantask_menu=visible; region2_bpel_instances_menu=visible; region3_registry_menu=none; i18next=en­US;
JSESSIONID=1579A304FBF0FE9AA2BEE6B9A7C9F85A;
requestedURI="../../carbon/service­mgt/index.jsp?region=region1&item=services_list_menu";
current­breadcrumb=manage_menu%2Cservices_menu%2Cservices_list_menu%23;
MSG13915816906030.6012256733392928=true; region1_configure_menu=none; region4_monitor_menu=none;
region5_tools_menu=none[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "Content­Type: application/soap+xml; charset=UTF­8; action="urn:credit"[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "Host: localhost:9764[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "Connection: Keep­Alive[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "User­Agent: Synapse­PT­HttpComponents­NIO[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "124[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><sam:get
xmlns:sam="http://samples.esb.wso2.org">[\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "                            <sam:id>33</sam:id>[\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "                        </sam:get></soapenv:Body></soapenv:Envelope>[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "0[\r][\n]"
[2014­02­05 15:38:46,692] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "HTTP/1.1 200 OK[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "Content­Type: application/soap+xml;charset=UTF­8[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "Date: Wed, 05 Feb 2014 10:08:46 GMT[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "Server: WSO2 Carbon Server[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,695] DEBUG ­ wire >> "200[\r][\n]"
[2014­02­05 15:38:46,696] DEBUG ­ wire >> "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><ns:getResponse
xmlns:ns="http://samples.esb.wso2.org"><ns:return xmlns:ax2461="http://samples.esb.wso2.org/xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema­instance" xsi:type="ax2461:PersonInfo"><ax2461:address>59, Flower Road,
Lanka</ax2461:address><ax2461:id>33</ax2461:id><ax2461:name>WSO2</ax2461:name></ns:return></ns:getResponse></soa
penv:Body></soapenv:Envelope>[\r][\n]"
[2014­02­05 15:38:46,697] DEBUG ­ wire >> "0[\r][\n]"
[2014­02­05 15:38:46,697] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,706] DEBUG ­ wire << "POST /services/CreditService/ HTTP/1.1[\r][\n]"
[2014­02­05 15:38:46,706] DEBUG ­ wire << "Content­Type: application/soap+xml;charset=UTF­8[\r][\n]"
[2014­02­05 15:38:46,707] DEBUG ­ wire << "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,707] DEBUG ­ wire << "Host: localhost:9764[\r][\n]"
[2014­02­05 15:38:46,707] DEBUG ­ wire << "Connection: Keep­Alive[\r][\n]"
[2014­02­05 15:38:46,707] DEBUG ­ wire << "User­Agent: Synapse­PT­HttpComponents­NIO[\r][\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "200[\r][\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><sam:credit
xmlns:sam="http://samples.esb.wso2.org" xmlns:xsd="http://samples.esb.wso2.org/xsd"
xmlns:ax21="http://samples.esb.wso2.org/xsd">[\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "<sam:info>[\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "<xsd:amount>2</xsd:amount>[\n]"
[2014­02­05 15:38:46,708] DEBUG ­ wire << "<xsd:personInfo>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "<xsd:address>59, Flower Road, Colombo 07, Sri Lanka</xsd:address>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "<xsd:id>33</xsd:id>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "<xsd:name>WSO2</xsd:name>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "</xsd:personInfo>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "</sam:info>[\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "</sam:credit></soapenv:Body></soapenv:Envelope>[\r][\n]"
[2014­02­05 15:38:46,709] DEBUG ­ wire << "0[\r][\n]"
[2014­02­05 15:38:46,710] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "HTTP/1.1 200 OK[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "Content­Type: application/soap+xml;charset=UTF­8[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "Date: Wed, 05 Feb 2014 10:08:46 GMT[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "Server: WSO2 Carbon Server[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,712] DEBUG ­ wire >> "109[\r][\n]"
[2014­02­05 15:38:46,713] DEBUG ­ wire >> "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><ns:creditResponse
xmlns:ns="http://samples.esb.wso2.org"><ns:return>true</ns:return></ns:creditResponse></soapenv:Body></soapenv:Envelope
[2014­02­05 15:38:46,713] DEBUG ­ wire >> "0[\r][\n]"
[2014­02­05 15:38:46,713] DEBUG ­ wire >> "[\r][\n]"
[2014­02­05 15:38:46,714]  INFO ­ LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: ,
SOAPAction: , MessageID: urn:uuid:10f13946­23dc­49e0­97ab­a369c2101a9e, Direction: response, Envelope: <?xml
version="1.0" encoding="utf­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><ns:creditResponse
xmlns:ns="http://samples.esb.wso2.org"><ns:return>true</ns:return></ns:creditResponse></soapenv:Body></soapenv:Envelope
[2014­02­05 15:38:46,717] DEBUG ­ wire << "HTTP/1.1 200 OK[\r][\n]"
[2014­02­05 15:38:46,717] DEBUG ­ wire << "Content­Type: application/soap+xml;charset=UTF­8[\r][\n]"
[2014­02­05 15:38:46,717] DEBUG ­ wire << "Date: Wed, 05 Feb 2014 10:08:46 GMT[\r][\n]"
[2014­02­05 15:38:46,717] DEBUG ­ wire << "Server: WSO2­PassThrough­HTTP[\r][\n]"
[2014­02­05 15:38:46,717] DEBUG ­ wire << "Transfer­Encoding: chunked[\r][\n]"
[2014­02­05 15:38:46,717] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:46,718] DEBUG ­ wire << "109[\r][\n]"
[2014­02­05 15:38:46,718] DEBUG ­ wire << "<?xml version="1.0" encoding="UTF­8"?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap­envelope"><soapenv:Body><ns:creditResponse
xmlns:ns="http://samples.esb.wso2.org"><ns:return>true</ns:return></ns:creditResponse></soapenv:Body></soapenv:Envelope
[2014­02­05 15:38:46,718] DEBUG ­ wire << "0[\r][\n]"
[2014­02­05 15:38:46,718] DEBUG ­ wire << "[\r][\n]"
[2014­02­05 15:38:47,786]  INFO ­ SequenceDeployer Sequence: fault has been updated from the file:
/home/sanjeewa/work/workflow/wso2am­1.6.0­1/repository/deployment/server/synapse­configs/default/sequences/fault.xml

Tuesday, May 22, 2018

How to avoid getting empty response to client due to slowness in key validation call - WSO2 API Manager

Users may get empty response due to slowness in key validation call due to the possibility of timeout gateway to key manager service call. These timeouts can happen in many different ways like below.
01. Connection timeout happen during establishment phase.
We can address this using axis2 client configuration change. Default connection timeout is 60 seconds and we can change that using below property.
Connection Timeout - the time to establish a connection with the remote host
"CONNECTION_TIMEOUT"
>30000
02. Socket timeout due to inactivity to wait packets to arrive.
We can configure this value as well through axis2 client configuration change. Default value of this property is 60 seconds and we can change that as required.
Socket Timeout - this is the time of inactivity to wait for packets to arrive
"SO_TIMEOUT"
>30000
03. Delay due to slow healthy connection .
In this case connection establishment and packet sending happens as usual. But data transfer between server and client getting delayed. In this case what happens is http client keep sending data to server till it accepts them.  We can recreate this by adding slow proxy or something like that. Here important thing is if we notice this type of delay then all UI operations including logging, token retrieval, update etc will also effect. We do not have specific property to override this waiting time from http client level. After 2 minutes this throws error. But before that after 1 minute source handler getting timeout and user will get empty response as it breaks connection.
To send some sort of error to client source handler should wait more than 2 minutes without getting timeout. So if we increase passthrough level socket timeout to 3 mins or so then it will wait till key manager error comes due to data send error. Then key validation handler will send proper unclassified authentication error with error code. Increasing passthrough socket timeout will effect all APIs deployed in the system as its transport level property. But most of the cases proper clients will have timeout values in client level so they will not effect due to this changes. If some client waits forever without having timeout then they will get error like below.
{"fault":{"code":900900,"message":"Unclassified Authentication Failure","description":"Error while accessing backend services for API key validation"}}
To set socket timeout please edit following property in passthru-http.properties.
http.socket.timeout=180000
Please refer this[1] document to understand more about client properties.
[1]http://hc.apache.org/httpclient-3.x/preference-api.html#HTTP_connection_parameters

Monday, January 1, 2018

How to manage external/internal gateways and store with single publisher - WSO2 API Manager

We usually use external API stores only for advertising purposes. If they need to consume API then they have to come back to original store and subscribe there. But as we see your requirement is slightly different from that.

Let me explain you how most of our users handle API usability for internal/external users. I'm sure these external and internal users are resides in different user groups. If that is the case, then we will be able to map them to different user roles. And then we can set the visibility to only required group. For an example if we have API which expose sensitive information then we can set visibility of that API to internal_role role. Then users with internal_role role can only see that API, External users will not be able to see that API or subscribe that.

If we think of deployment then, API Store node will be deployed inside corporate network and outside network pointing to same database. Based on logged in user we can show them right APIs. This solution proposed to you assuming same data shared between both external and internal deployments.

Let me give you exact steps we need to follow,
01. Create 2 different roles for internal and external users(named internal and external).
02. Deploy API Manager internal and external gateways and configure publisher to publish them selectively.
03. Store also can deploy internally and externally pointing to same databases.
04. To create internal only API, create API with limited viability to internal role. Then publish this API to internal gateway.
When internal user logged into internal API store they will see API and can consume it(as they have internal role).
When external user logged into external API store they will not see above API and cannot use that. If they try to access API from external gateway then it will also failed as API not deployed there.
05. To create internal/external API, create API with limited viability to both internal and external role. Then publish this API to both internal and external gateways.
When external user logged into external API store they will see API and can consume it(as they have external role).
When internal user logged into internal API store they will see above API and can use that. If they try to access API from external gateway then it will also work as API deployed there.
06. To create external only API, create API with limited viability to external role. Then publish this API to external gateway.
When external user logged into external API store they will see API and can consume it(as they have external role).
When internal user logged into internal API store they will not see above API and cannot use that. If they try to access API from internal gateway then it will also failed as API not deployed there.

Sunday, October 8, 2017

How to write test case using wiremock and test Mutual SSL enabled Backend Service invocation

Add following dependencies to the pom file so we can use wiremock for testing. I specifically added below slf4j versions to dependency as they required. Also I excluded some of the components as they started sending errors. Same way if you get any errors just type mvn dependency:tree and get all dependencies. Then you can exclude problematic components.

<dependency>

             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-log4j12</artifactId>
             <version>1.7.7</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-api</artifactId>
             <version>1.7.7</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.hamcrest</groupId>
             <artifactId>hamcrest-all</artifactId>
             <version>1.3</version>
             <scope>test</scope>
        </dependency>
         <dependency>
                     <groupId>com.github.tomakehurst</groupId>
                     <artifactId>wiremock</artifactId>
                     <version>2.5.0</version>
                     <exclusions>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>slf4j-jdk14</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>jcl-over-slf4j</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>log4j-over-slf4j</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>com.fasterxml.jackson.core</groupId>
                             <artifactId>jackson-annotations</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>com.fasterxml.jackson.core</groupId>
                             <artifactId>jackson-core</artifactId>
                         </exclusion>
                     </exclusions>
        </dependency>

Following is my test class.

package org.test.testpkg;
import jdk.nashorn.internal.objects.NativeObject;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import com.github.tomakehurst.wiremock.junit.WireMockRule;
import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
import java.io.File;
public class MutualSSLTest {
    private static final Log log = LogFactory.getLog(MutualSSLTest.class);
    //Key store and client trustore paths and passwords
    private static final String KEYSTORE_FILE_PATH =
            "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "server" + File.separator + "wso2carbon.jks";
    private static final String TRUSTSTORE_FILE_PATH =
            "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "server" + File.separator + "client-truststore.jks";
    private static final String KEYSTORE_FILE_PATH_CLIENT =
             "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "client" + File.separator + "wso2carbon.jks";
    private static final String TRUSTSTORE_FILE_PATH_CLIENT =
             "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "client" + File.separator + "client-truststore.jks";
    public void testAPIProvider() {
    }
    @Rule
    public WireMockRule wireMockRule;
    @Test
    public void testMutualSSLEnabledBackend() {
//Create wiremock rule by providing SSL configuratios. Here we need to pass keystore/trustore, port and other required information.
        wireMockRule = new WireMockRule(wireMockConfig()
                .httpsPort(8081)
                .needClientAuth(true)
                .trustStoreType("JKS")
                .keystoreType("JKS")
                .keystorePath(KEYSTORE_FILE_PATH)
                .trustStorePath(TRUSTSTORE_FILE_PATH)
                .trustStorePassword("wso2carbon")
                .keystorePassword("wso2carbon"));
        wireMockRule.start();
        // Mock service for test endpoint. This will return 200 for http head method.
        wireMockRule.stubFor(head(urlEqualTo("/test"))
                .willReturn(aResponse()
                        .withStatus(200)
                        .withBody("{success}")
                        .withHeader("Content-Type", "application/json")
                ));
        try {
   //Then i will set keystore and client trustore to system properties.
            System.setProperty("javax.net.ssl.keyStoreType", "JKS");
            System.setProperty("javax.net.ssl.keyStore", KEYSTORE_FILE_PATH_CLIENT);
            System.setProperty("javax.net.ssl.keyStorePassword", "wso2carbon");
            System.setProperty("javax.net.ssl.trustStore", TRUSTSTORE_FILE_PATH_CLIENT);
            System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon");
   //Now i will invoke my utility method and call created service
            org.mozilla.javascript.NativeObject obj =
                    HostObjectUtils.sendHttpHEADRequest("https://localhost:8081/test",
                            "404");
            //Then i will assert response.
   Assert.assertEquals("success", obj.get("response"));
        } catch (Exception e) {
            e.printStackTrace();
        }
        wireMockRule.resetAll();
        wireMockRule.stop();
    }
}

In my utility class i have following method to call HTTP service and get response.

    /**
     * Validate the backend by sending HTTP HEAD
     *
     * @param urlVal - backend URL
     * @param invalidStatusCodesRegex - Regex for the invalid status code
     * @return - status of HTTP HEAD Request to backend
     */
    public static NativeObject sendHttpHEADRequest(String urlVal, String invalidStatusCodesRegex) {
        boolean isConnectionError = true;
        String response = null;
        NativeObject data = new NativeObject();
        //HttpClient client = new DefaultHttpClient();
        HttpHead head = new HttpHead(urlVal);
        //Change implementation to use http client as default http client do not work properly with mutual SSL.
        org.apache.commons.httpclient.HttpClient clientnew = new org.apache.commons.httpclient.HttpClient();
        // extract the host name and add the Host http header for sanity
        head.addHeader("Host", urlVal.replaceAll("https?://", "").replaceAll("(/.*)?", ""));
        clientnew.getParams().setParameter("http.socket.timeout", 4000);
        clientnew.getParams().setParameter("http.connection.timeout", 4000);
        HttpMethod method = new HeadMethod(urlVal);
        if (System.getProperty(APIConstants.HTTP_PROXY_HOST) != null &&
                System.getProperty(APIConstants.HTTP_PROXY_PORT) != null) {
            if (log.isDebugEnabled()) {
                log.debug("Proxy configured, hence routing through configured proxy");
            }
            String proxyHost = System.getProperty(APIConstants.HTTP_PROXY_HOST);
            String proxyPort = System.getProperty(APIConstants.HTTP_PROXY_PORT);
            clientnew.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY,
                    new HttpHost(proxyHost, Integer.parseInt(proxyPort)));
        }
        try {
            int statusCodeNew = clientnew.executeMethod(method);
            //Previous implementation
            // HttpResponse httpResponse = client.execute(head);
            String statusCode = String.valueOf(statusCodeNew);//String.valueOf(httpResponse.getStatusLine().getStatusCode());
            String reasonPhrase = String.valueOf(statusCodeNew);//String.valueOf(httpResponse.getStatusLine().getReasonPhrase());
            //If the endpoint doesn't match the regex which specify the invalid status code, it will return success.
            if (!statusCode.matches(invalidStatusCodesRegex)) {
                if (log.isDebugEnabled() && statusCode.equals(String.valueOf(HttpStatus.SC_METHOD_NOT_ALLOWED))) {
                    log.debug("Endpoint doesn't support HTTP HEAD");
                }
                response = "success";
                isConnectionError = false;
            } else {
                //This forms the real backend response to be sent to the client
                data.put("statusCode", data, statusCode);
                data.put("reasonPhrase", data, reasonPhrase);
                response = "";
                isConnectionError = false;
            }
        } catch (IOException e) {
            // sending a default error message.
            log.error("Error occurred while connecting to backend : " + urlVal + ", reason : " + e.getMessage(), e);
            String[] errorMsg = e.getMessage().split(": ");
            if (errorMsg.length > 1) {
                response = errorMsg[errorMsg.length - 1]; //This is to get final readable part of the error message in the exception and send to the client
                isConnectionError = false;
            }
        } finally {
            method.releaseConnection();
        }
        data.put("response", data, response);
        data.put("isConnectionError", data, isConnectionError);
        return data;
    }
}


Now we have successfully implemented mutual ssl test case. You can run test and verify this behavior. If you need to test negative impact then comment keystore password in client.
Then you will see errors in logs.

How to do zero downtime migration - WSO2 API Manager

Here in this post i will explain how we can do zero downtime migration in deployment.

Test Migration
First we can take backup of running system and create setup locally. Then we can perform migration and identify if there are any issues. Usually dump restore and setup take sometime based on deployment.

Then after that we can test APIs, resources etc. Also we can test extension points, integrations etc. 

Then after we completed above we can come up with exact steps on migrating. So we can run same in deployment backup(or staging deployment). Ideally it should work without issues. Then we can repeat this to verify exact steps.

Following steps need to follow(if need they can automate that)
  • Create dump of current running setup
  • Restore it to lower environment.
  • Perform migration.
  • Do sanity check

Then we need to analyze traffic pattern of the deployment for about one week or more to identify time slot for migration. After some analysis we can identify time window with minimum user interaction.

Then we need to perform all above mentioned steps in production server. But at this time we should test all use cases end to end.

Also we need to create one way synchup script to synchronize tokens to new deployment(tokens created within maintain window).
Real migration
Then comes real migration,
  • Make API Manager store and publisher read only for maintain window(say 3 hours).
  • Perform creating dump and restore to new environment.
  • Do migration in new envirionment.
  • Run synchup script to update new tokens created in original deployment.
  • Then test functionalities using automated tests and manual tests.
  • Once we are 100% confident we can do DNS switch.
  • If everything works fine open traffic for store/publisher as well.
If you wish to do gradual traffic movement to new deployment it may bit different from above above mention approach. In that case we may need 2 way synch up scripts as tokens can created in both deployments.

Wednesday, August 2, 2017

How to run WSO2 API Manager with MSSQL using docker

First you need to install docker on you local machine.

Then run following to install mssql docker image in your local machine. You can follow this article[https://hub.docker.com/r/microsoft/mssql-server-linux/ ] for more information.

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Password#123' -p 1433:1433 -d microsoft/mssql-server-linux

Then type following and verify your instance is up and running>>docker ps


CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                      NAMES03ebf2498a20        microsoft/mssql-server-linux   "/bin/sh -c /opt/mssq"   23 minutes ago      Up 23 minutes       0.0.0.0:1433->1433/tcp    



docker exec -it /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P Password#123

Download and copy the sqljdbc4 Microsoft SQL JDBC driver file to the WSO2 product's /repository/components/lib/ directory

Edit the default datasource configuration in the /repository/conf/datasources/m aster-datasources.xml file as shown below.



<datasource>

            <name>WSO2AM_DB</name>
            <description>The datasource used for API Manager database</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
           <configuration>
              <url>jdbc:sqlserver://0.0.0.0:1433;databaseName=TestDB</url>
              <username>sa</username>
              <password>Password#123</password>
              <driverClassName>com.microsoft.sqlserver.jdbc.SQLServerDriver</driverClassName>
              <maxActive>80</maxActive>
              <maxWait>60000</maxWait>
              <testOnBorrow>true</testOnBorrow>
              <validationQuery>SELECT 1</validationQuery>
              <validationInterval>30000</validationInterval>
              <defaultAutoCommit>false</defaultAutoCommit>
           </configuration>
            </definition>
        </datasource>


Now start server with following command.
sh wso2server.sh -Dsetup 

Monday, July 17, 2017

Best practices we should follow when we create API.

Best practices we should follow when we create API.

Create API for dedicated back end service. For each of the resources the HTTP methods used to perform the required application functions have to be decided. This includes the use of applicable HTTP headers.
Special behavior required by the application (e.g. concurrency control, long running requests) has to be decided.
Finally, potential error situations have to be identified and corresponding error messages have to be designed.

Proper Naming

It's important to have proper name for service and service paths. For example when we create some API related to camera related operations we can name API and resources with camera specific terms. For example we can name API as camera-api. Then when we create new resource we can create resource like capture-image, delete-image etc.
Resources must be named properly by means of URIs (Uniform Resource Identifiers). Proper naming of resources is key for an API to be easily understandable by clients.
Following are some of the guidelines for design proper API/Resource paths and names. These are not mandatory rules but whenever possible we should these best practices.

Atomic resources, collection resources and composite resources should be named as nouns because they represent "things", not "actions" (actions would lean more towards verbs as names of resources).
  • Processing function resources and controller resources should be named as verbs because they in fact represent "actions".
  • Processing function resources and controller resources should not be sub-resources of individual other resources.
  • Lower case characters should be used in names only because the rules about which URI element names are case sensitive and which are not may cause confusion.
  • If multiple words are used to name a resource, these words should be separated by dashes (i.e. "-") or some other separator.
  • Singular nouns should be used for naming atomic resources.
  • Names of collections should be "pluralized", i.e. named by the plural noun of the grouped concept (atomic resource or composite resource).
  • Use forward slashes (i.e. "/") to specify hierarchical relations between resources. A forward slash will separate the names of the hierarchically structured resources. The parent name will immediately precede the name of its immediate children.

Proper Versioning

The version of an API is specified as part of its URI. Usually this version is specified as a pair of integers (separated by a dot) referred to as the major and the minor number of the version, preceded by the lowercase letter "v". E.g. a valid version string in the base path would be v2.1 indicating the first minor version of the second major version of the corresponding API.

Proper versioning strategy for API will helpful for all API users and client to communicate with API easily and effectively. In WSO2 API Manager we do have API versioning support. Also we can copy existing API and create new version of same API. When you need to create new version of running API recommended approach is create new version from existing API API and modify it. Then before we publish new version we need to test all the functionalities of this API.
Following are some of the guidelines we need to follow when we version APIs. Again these are not mandatory rules. But it's good to follow these rules whenever possible.

In general, a version number following the semantic versioning concept has the structure major.minor.patch and the significance in terms of client impact is increasing from left to right:
  • An incremental patch number means that the underlying modification to the API cannot even be noticed by a client - thus, the patch number is omitted from the version string. Only the internal implementation of the API has been changed while the signature of the API is unchanged. From the perspective of the API developer, a new patch number indicates a bug fix, a minor internal modification, etc.
  • An incremented minor number indicates that new features have been added to the API, but this addition must be backward compatible: the client can use the old API without failing. For example, the API may add new optional parameters or a completely new request.
  • An incremented major number signals changes that are not backward compatible: for example, new mandatory parameters have been added, former parameters have been dropped, or complete former requests are no longer available.
It is best practice to support the current major version as well as at least one major version back. In case new versions are released frequently (e.g. every few months) more major versions back have to be supported. Otherwise, clients will break (too fast).

Use of Proper HTTP Methods

Manipulation of resources in the REST style is done by create, retrieve, update, and delete operations (so-called CRUD operations), that map to the HTTP methods POST, GET, PUT, PATCH and DELETE.
A request that can be used without producing any side-effect is called a safe request. A request that can be used multiple times and that is always producing the same effect as the first invocation is called idempotent.

Get

GET is in HTTP as well as in the REST style specified as a safe and idempotent request. Thus, an API using the GET method must not produce any side-effects. Retrieving an atomic resource  or a composite resource is done by performing a GET on the URI identifying the resource to be retrieved.

Put

PUT substitutes the resource identified by the URI. Thus, the body of the corresponding PUT message provides the modified but complete representation of a resource that will completely substitute the existing resource: parts of the resource that have not changed must be included in the modified resource of the message body. Especially, a PUT request must not be used for a partial update.

Post

POST is neither safe nor idempotent. The main usages of POST are the creation of new resource, and the initiation of functions, i.e. to interact with processing function resources as well as controller resources.
In order to create a new resource, a POST request is used with the URI of the collection resource to which the new resource should be added.

Delete

A resource is deleted by means of the DELETE request on the URI of the resource. Once a DELETE request returned successfully with a "200 OK" response, following DELETE requests on the same URI will result in a "404 Not Found" response because there is no resource available with the URI of the deleted resource.

Patch

Usually HTTP Patch request will use to do partial update on resources. For example if you do not change entire resource and just changes some of the attributes you can use patch method. Unlike Put method we can use this for partial update for resources.