WSO2 API Manager 1.10 New REST API - How to obtain access token and invoke APIs


From API Manager 1.10 onwards we will provide complete REST API to do all store publisher operations. In this article we will dicuss how we can use it in secured manner.


Sample Request to registration API

URL: http://10.100.1.65:9763/client-registration/v1/register
HTTP Method: POST
Headers >
Content-Type: application/json
Basic Auth Headers


Body Payload:
{
"callbackUrl": "www.google.lk",
"clientName": "rest_api_store",
"tokenScope": "Production",
"owner": "admin",
"grantType": "password refresh_token",
"saasApp": true
}


Sample response

{
"callBackURL": "www.google.lk",
"jsonString": "{\"username\":\"admin\",\"redirect_uris\":\"www.google.lk\",\"tokenScope\":[Ljava.lang.String;@3a73796a,\"client_name\":\"admin_rest_api_store\",\"grant_types\":\"authorization_code password refresh_token iwa:ntlm urn:ietf:params:oauth:grant-type:saml2-bearer client_credentials implicit\"}",
"clientName": null,
"clientId": "HfEl1jJPdg5tbtrxhAwybN05QGoa",
"clientSecret": "l6c0aoLcWR3fwezHhc7XoGOht5Aa"
}


Then get token with scopes
API_PUBLISHER_SCOPE, publisher
API_SUBSCRIBER_SCOPE, "subscriber
API_CREATOR_SCOPE, creator
API_ADMINISTRATIVE_SCOPE, admin



curl -k -d "grant_type=password&username=admin&password=admin&scope=API_CREATOR_SCOPE" -H "Authorization: Basic Y1lIaGxnRU9UcmM5Q05LUGZVaDdaMDBQdUtzYTpzX1VjOVdRWURLdHNBd1lrSl9qcFc5bnVmdVVh" https://127.0.0.1:8243/token


Then invoke API with obtained token.

How to modify tenant API store main page in WSO2 API Manager.

In this article we will discuss how we can modify tenant API Store to support lengthy tenant domain names. With API Manager 1.9.0 we cannot have very lengthy names. In this modification we will show part of tenant name in store and show full name when you place mouse courser on tenant store.

in template.jag(/store/site/themes/fancy/templates/api/tenant-stores-listing/template.jag) paste following before this line

"<li class="thumbnail span3 tenent-thumb">"
<a href="<%= encode.forHtmlAttribute(encode.forUri(jagg.getSiteContext() + 
"?tenant="+tenantDomains[i])) %>" title="<%=tenantDomains[i]%>">

in css(styles-layout,css) do following
#tenent_list{ width:900px }
.tenent-thumb{ margin: 0.79% 0.79% !important; /* re-declare thumb border styles since it is overidden by styles-layout */ border: 1px solid #DDDDDD; border-radius: 4px 4px 4px 4px; box-shadow: 0 1px 3px rgba(0, 0, 0, 0.055); display: block; line-height: 20px; padding: 8px 4px 4px 4px; transition: all 0.2s ease-in-out 0s; }
.tenent-thumb h3{ margin:0px !important; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; width:200px; }

Then you will see it as follows.

How to create custom error message when our message flow hit fault sequence

When we use WSO2 ESB fault sequence may hit due to different reasons. In such cases we normally send error message as defined in fault sequence. In this blog i will explain how we can modify fault sequence to send customized error to client.

To try this i created simple proxy which directly go to fault sequence. In your case it can happen due to some other reason

<proxy name="testp"
transports="https http"
startOnLoad="true"
trace="disable">
<target>
<inSequence>
<sequence key="fault"/>
</inSequence>
<outSequence>
<send/>
</outSequence>
</target>
</proxy>

Inside fault sequence log message and call my custom sequence named convert.

<sequence name="fault">
<log>
<property name="Inside fault sequence============" expression="get-property('test')"/>
</log>
<sequence key="convert"/>
<drop/>
</sequence>

My convert sequence will generate custom error message and send it to client application.

<sequence name="convert">
<payloadFactory media-type="xml">
<format>
<am:fault xmlns:am="http://wso2.org/apimanager">
<am:code>$1</am:code>
<am:type>Status report</am:type>
<am:message>Runtime Error</am:message>
<am:description>$2</am:description>
</am:fault>
</format>
<args>
<arg evaluator="xml" expression="$ctx:ERROR_CODE"/>
<arg evaluator="xml" expression="$ctx:ERROR_MESSAGE"/>
</args>
</payloadFactory>
<property name="RESPONSE" value="true"/>
<header name="To" action="remove"/>
<property name="HTTP_SC" value="555" scope="axis2"/>
<property name="NO_ENTITY_BODY" scope="axis2" action="remove"/>
<property name="ContentType" scope="axis2" action="remove"/>
<property name="Authorization" scope="transport" action="remove"/>
<property name="Access-Control-Allow-Origin" value="*" scope="transport"/>
<property name="Host" scope="transport" action="remove"/>
<property name="Accept" scope="transport" action="remove"/>
<property name="messageType" value="application/json" scope="axis2"/>
<send/>
</sequence>

Client will recieve following error message as we difined in fault sequence.

Status Code: 555
Accept-Encoding: gzip, deflate
Accept-Language: null
Connection: keep-alive
Content-Type: application/json
Date: Tue, 20 Oct 2015 06:19:51 GMT
Server: WSO2-PassThrough-HTTP
Transfer-Encoding: chunked
access-control-allow-origin: *

How WSO2 API Manager gateway to back end secure connection works and troubleshoot SSL gateway issues.

Whenever Java(here in this care WSO2 API Manager act as JVM) attempts to connect to another application over SSL, it will only be able to connect to that application if it can trust it. The way trust is handled in the Java world is that you have a key store (typically $JAVA_HOME/lib/security/cacerts), also known as the trust store. This contains a list of all known Certificate Authority (CA) certificates, and Java will only trust certificates that are signed by one of those CAs or public certificates that exist within that key store.

This problem is therefore caused by a certificate that is self-signed (a CA did not sign it) or a certificate chain that does not exist within the Java key store. Java does not trust the certificate and fails to connect to the application.

We can simply test it with following commands. Here i will use external class to perform SSL poke(class) using our client trust store.

For 1st URL
===========

sanjeewa@sanjeewa-ThinkPad-T530:~/Downloads$ java -Djavax.net.ssl.trustStore=/home/sanjeewa/work/packs/wso2am-1.9.0/repository/resources/security/client-truststore.jks SSLPoke test.com 443
    Successfully connected

For 2nd URL
===========

    sanjeewa@sanjeewa-ThinkPad-T530:~/Downloads$ java -Djavax.net.ssl.trustStore=/home/sanjeewa/work/packs/wso2am-1.9.0/repository/resources/security/client-truststore.jks SSLPoke test-one.com 443
    sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:136)
    at SSLPoke.main(SSLPoke.java:31)
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
    ... 15 more

As you can see first will print "Successfully connected". Then clearly 2nd one is self signed certificate which does not reside in our trust store. And that cause issue here.
Even if we have host name verification or not it doesn't matter here.

Host name verifier will come next.
If we didn't put host name verifier AllowAll in axis2.xml for first end point (https://test.com/posts) then you will see following.

    [2015-09-22 18:36:36,765] ERROR - TargetHandler I/O error: Host name verification failed for host : test.com
    javax.net.ssl.SSLException: Host name verification failed for host : test.com
    at org.apache.synapse.transport.http.conn.ClientSSLSetupHandler.verify(ClientSSLSetupHandler.java:152)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:285)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:380)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:118)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
    at java.lang.Thread.run(Thread.java:745)

During handshaking, if the URL's host name and the server's identification host name mismatch, the verification mechanism can call back to implements of interface(what we have done is override it with property defined in axis2.xml file and return true if property is there) to determine if this connection should be allowed. The policies can be certificate-based or may depend on other authentication schemes. Here second URL failed not because host name issue its failing because its self signed certificate and java does not trusted it. Even first one will fail if we didn't enabled host name verification.

Please refer SSLPoke(class) class for your reference.

How to add 2 step authentication flow to WSO2 APP Manager.

WSO2 App Manager provides a unique one-stop store app management solution where users can pick and choose apps required for them to do their jobs efficiently. Businesses can leverage its single-sign-on (SSO) functionality, which can reduce help desk and administrative costs with no long lists of passwords to memorise. It automatically applies a security layer on top of the web apps published in the store, which eliminates the need to embed security rules at the application layer.

In this article we will discuss how we can add two step authentication when we register service provider.

When you create application in app manager it will call to SSOConfigurator class we defined in app-manager.xml to create service providers.
So i implemented custom class to do it. Class name is TESTIS500SAMLSSOConfigurator and it implemented SSOConfigurator.


Within this class we will create SP and store it. So when we create SP we will add 2 authentication steps.
1. Authentication step to redirect users to Dictao to login system.
2. Then add local authentication step to validate subscriptions and let user to accept terms and conditions if he haven't already done.

We have added few parameters to app-manager.xml to make this configurable. With that new configuration would be something like this. Actually if you need you can add more parameters according to your requirement.

        <Configurators>
            <Configurator>
                <name>wso2is</name>
                <version>5.0.0</version>
        <!--class name of new sso configurator class-->                
        <providerClass>com.test.identity.sso.configurator.customauth.TESTIS500SAMLSSOConfigurator</providerClass>
                <parameters>
                <providerURL>https://is.test.com:9447</providerURL>
                <username>admin</username>
                <password>admin</password>
        <!--IDP name which need to be engage in authentication step 01. You have to create IDP with this name-->    
                <idpName>IDP1</idpName>
        <!--The name of authenticator we should used in step 02. This authenticator is another custom implementation -->    
                <idpStepTwo>TEST-customAuthenticator</idpStepTwo>
            <!--Authentication mechanism of step one--> 
                <authenticationStep>federated</authenticationStep>
                </parameters>
           </Configurator> 




Following code block will be use to add 2 step authentication and engage IDP, Authenticators to authentication flow. You can modify TESTIS500SAMLSSOConfigurator class by adding following.

   if (idpName != null && authenticationStep != null && authenticationStep.equalsIgnoreCase("federated")) {
            if (log.isDebugEnabled()) {
                log.debug("Adding federated authentication step. Added IDP named: " + idpName);
            }
            //Following code will set external IDP as authentication EP
            serviceProvider.getLocalAndOutBoundAuthenticationConfig().setAuthenticationType("flow");
            InboundProvisioningConfig inBoundProConfig = new InboundProvisioningConfig();
            inBoundProConfig.setProvisioningUserStore("");
            serviceProvider.setInboundProvisioningConfig(inBoundProConfig);
            serviceProvider.setOutboundProvisioningConfig(new OutboundProvisioningConfig());
            serviceProvider.setRequestPathAuthenticatorConfigs(null);
            AuthenticationStep[] steps = new AuthenticationStep[2];

            //Add local authenticator
            AuthenticationStep step1 = new AuthenticationStep();
            List<LocalAuthenticatorConfig> localAuthList = new ArrayList<LocalAuthenticatorConfig>();
            LocalAuthenticatorConfig localAuth = new LocalAuthenticatorConfig();
            localAuth.setName(idpStepTwo);
            localAuth.setDisplayName(idpStepTwo);
            localAuth.setEnabled(true);
            localAuthList.add(localAuth);
            step1.setLocalAuthenticatorConfigs(localAuthList.toArray(new LocalAuthenticatorConfig[localAuthList.size()]));
            step1.setStepOrder(2);

            //Add federated authenticator
            AuthenticationStep step = new AuthenticationStep();
            List<IdentityProvider> federatedAuthList = new ArrayList<IdentityProvider>();
            FederatedAuthenticatorConfig federatedAuthenticatorConfig = new FederatedAuthenticatorConfig();
            IdentityProvider identityProvider = new IdentityProvider();
            federatedAuthenticatorConfig.setName("SAMLSSOAuthenticator");
            identityProvider.setIdentityProviderName(idpName);
            identityProvider.setFederatedAuthenticatorConfigs(new FederatedAuthenticatorConfig[]{federatedAuthenticatorConfig});
            identityProvider.setDefaultAuthenticatorConfig(federatedAuthenticatorConfig);
            federatedAuthList.add(identityProvider);
            step.setFederatedIdentityProviders(federatedAuthList.toArray(new IdentityProvider[federatedAuthList.size()]));
            step.setStepOrder(1);
          
             //Here federated authenticator would be added as step 01 and local authenticator would be step 02.
            steps[0] = step;
            steps[1] = step1;
            serviceProvider.setPermissionAndRoleConfig(new PermissionsAndRoleConfig());
            serviceProvider.getLocalAndOutBoundAuthenticationConfig().setAuthenticationSteps(steps);
        } 


       
      
Then once you create application it will automatically register service provider with 2 step authentication.
Once you go to service provider user interface you will see authentication steps as follows.



Complete source code.


https://drive.google.com/file/d/0B3OmQJfm2Ft8LW9TeE5JRW1HU2M/view?usp=sharing

How to set password validation policy in WSO2 Identity Server


If you need to add custom password policy there are multiple layers you can add that. First one is user-mgt.xml file and other configuration file is identity-mgt.properties file.

If identity management listener is enabled(only), user passwords should be satisfied both both regrEx defined in user-mgt.xml and identity-mgt.properties files. Otherwise we will check user-mgt.xml to validate password policy.

/repository/conf/user-mgt.xml
         
 <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>

Following properties will be picked only if we enabled identity listener(Identity.Listener.Enable=true). Otherwise configurations on user management xml will only affect.

/repository/conf/security/identity-mgt.properties
Password.policy.extensions.1.min.length=6
Password.policy.extensions.1.max.length=12
Password.policy.extensions.3.pattern=^((?=.*\\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[!@#$%&*])).{0,100}$

Sample data source configuration for WSO2 Servers to connect jdbc using LDAP

Please find following sample data source configuration to access jdbc using LDAP connection.

       <datasource>
            <name>DATASOURCE_NAME</name>
            <description>The datasource used for BPS</description>
            <jndiConfig>
                <name>jdbc/JNDI_NAME</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:oracle:thin:@ldap://localhost:389/cn=wso2dev2,cn=OracleContext,dc=test,dc=com</url>
                    <username>DB_USER_NAME</username>
                    <password>DB_PASSWORD</password>
                    <driverClassName>oracle.jdbc.OracleDriver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1 FROM DUAL</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>




How to install Redis in ubuntu and send event

Please follow below instructions to install and use Redis
Type following commands in command line.

wget http://download.redis.io/releases/redis-stable.tar.gz

tar xzf redis-stable.tar.gz

cd redis-stable

make

make test

sudo make install

cd utils

sudo ./install_server.sh
As the script runs, you can choose the default options by pressing enter.

Port depends on the port you set during the installation. 6379 is the default port setting.

sudo service redis_6379 start
sudo service redis_6379 stop

Access redis command line tool
redis-cli

You will see following command line.
redis 127.0.0.1:6379>


Then send events with key and value as follows.
127.0.0.1:6379> publish EVENTCHANNEL sanjeewa11111199999999



How to increase time out value in WSO2 API Manager

When we increase timeout value in API Manager we have to set 3 properties.

1) Global timeout defined in synapse.properties (\repository\conf\synapse.properties)

synapse.global_timeout_interval=60000000


2) Socket timeout defined in the passthru-http.properties (ESB_HOME\repository\conf\passthru-http.properties )

http.socket.timeout=60000000

3) Also we need to set timeout in API level per each API.
 <endpoint name="admin--Stream_APIproductionEndpoint_0">
      <address uri="http://localhost:9763/example-v4/example">
 <timeout>
   <duration>12000000</duration>
  <responseAction>fault</responseAction>
  </timeout>
 </address>

How to avoid getting incorrect access tokens due to constraint violation when we have high load on token API(CON_APP_KEY violated).

Sometimes you may see following behavior when we have very high load on token API.
1. Call https://localhost:8243/token
2. Get constraint error.
{org.wso2.carbon.identity.oauth2.dao.TokenPersistenceTask} - Error occurred while persisting access token bsdsadaa209esdsadasdae21a17d {org.wso2.carbon.identity.oauth2.dao.TokenPersistenceTask}
org.wso2.carbon.identity.oauth2.IdentityOAuth2Exception: Access Token for consumer key : H5sadsdasdasddasdsa, user : sanjeewa and scope : default already exists
at org.wso2.carbon.identity.oauth2.dao.TokenMgtDAO.storeAccessToken(TokenMgtDAO.java:194)
at org.wso2.carbon.identity.oauth2.dao.TokenMgtDAO.persistAccessToken(TokenMgtDAO.java:229)
at org.wso2.carbon.identity.oauth2.dao.TokenPersistenceTask.run(TokenPersistenceTask.java:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (WSO2_APIM.CON_APP_KEY) violated

3. Attempt to use the access token when calling an API but get HTTP Status-Code=401 (Unauthorized) Invalid Credentials error

This issue happens because our token storing logic is not blocking call(this was implemented as improvement to token API as persisting can block token generation flow).
So due to that we already returned incorrect token to client(which is not already persisted). This happens only if constraint failed when we try to persist token.
But at that time we may return token to client.

If we made token persisting pool size 0 then this issue will not be there and user will immediately get error (probably internal server error) and token will not be return to client.
See following code block

try {
tokenMgtDAO.storeAccessToken(accessToken, oAuth2AccessTokenReqDTO.getClientId(),
accessTokenDO, userStoreDomain);
} catch (IdentityException e) {
throw new IdentityOAuth2Exception(
"Error occurred while storing new access token : " + accessToken, e);
}

You can set pool size as follows. By default it set to 100.
wso2am-1.9.0/repository/conf/identity.xml

<JDBCPersistenceManager>
    <SessionDataPersist>
        <PoolSize>0</PoolSize>
    </SessionDataPersist>
</JDBCPersistenceManager>
 

This Will resolve your issue
 

How to minimize solr idexing time(registry artifact loading time) in newly spawned instance

In API Management platform sometimes we need to add store and publisher nodes to cluster. But if you have large number of resources in registry solr indexing will take some time.
Solr indexing will be used to index registry data in local file system. In this post we will discuss how we can minimize time take to this loading process. Please note this will apply to all carbon kernel 4.2.0 or below versions. In GREG 5.0.0 we have handled this issue and do not need to do anything to handle this scenario.

You can minimize the time taken to list down existing API list in Store and Publisher by copying an already indexed Solr/data directory to a fresh APIM instance.
However note that you should NOT copy and replace Solr/data directories of different APIM product versions. (For example you can NOT copy and replace Solr/data directory of APIM 1.9 to APIM 1.7)

[1] First create a backup of Solr indexed files using the last currently running API product version.
   [APIM_Home]/solr/data directory
[2] Now copy and replace [Product_Home]/solr/data directory in the new APIM instance/s before puppet initializes it, which will list existing APIs since by the time your new carbon instance starts running we have already copied the Solr indexed file to the instance.

If you are using automated process its recommend to automate this process.
So you can follow these instructions to automate it.
01. Get backup of solr/data directory of running server and push it to some artifact server(you can use Rsynch or svn for this).
02. When new instance spawned before start it copy updated solr content from remote artifact server.
03. Then start new server.


If you need to manually re-index data you can follow approach listed below.

Shutdown the server if it is already started.
Rename the lastAccessTimeLocation in registry.xml ,
Eg:
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime
To
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime_1
Backup the solr directory and delete it.

/solr

Restart the server and keep the server idle few minutes to re-index.

How to handle ditributed counter across cluster when each node contribute to counter - Distributed throttling

Handle throttling in distributed environment is bit tricky task. For this we need to maintain time window and counters per instance and also those counters should be shared across cluster as well. Recently i worked on similar issue and i will share my thoughts about this problem.

Lets say we have 5 nodes. Then each node will serve x number of requests within minutes. So across cluster we can server 5X requests per minutes. And some cases node1 may server 2x while other servers 1x. But still we need to have 5x across cluster. To address this issue we need shared counter across cluster. So each and every node can contribute to that and maintain counters.

To implement something like that we may use following approach.

We can maintain two Hazelcast IAtomicLong data structures or similar distributed counter as follow. This should be handle in cluster level.
And node do not have to do anything about replication.

  • Shared Counter : This will maintain global request count across the cluster
  • Shared Timestamp : This will be used for manage time window across the cluster for particular throttling period

In each and every instance we should maintain following per each counter object.
  • Local global counter which sync up with shared counter in replication task(Local global counter = shared counter + Local counter )
  • Local counter which holds request counts until replication task run.(after replication Local counter = 0)

We may use replication task that will run periodically.
During the replication task following tasks will be happen.
Update the shared counter with node local counter and then update local global counter with the shared counter.
If global counter set to zero, it will reset the global counter.


In addition we need to set the current time into the hazelcast Atomic Long .When other servers get the first request it sets it first access time as the value in hazelcast according to the caller context ID.So all the servers will set into the one first access time. We check the throttle time will become to the time come from hazelcast and unit Time according to tier.
To check time window is elapsed so if this happen We set previous callercontext globalcount to 0.
As assumption we made was all nodes in cluster are having the same timestamp.




See following diagrams.






If you need to use throttle core for your application/component run in WSO2 runtime you can import throttle core to your project and use following code to check access availability.


Here i have listed code to throttle message using handler. So you can write your own handler and call doThrottle method in message flow. First you need to import org.wso2.carbon.throttle.core to your project.


       private boolean doThrottle(MessageContext messageContext) {
            boolean canAccess = true;
            boolean isResponse = messageContext.isResponse();
            org.apache.axis2.context.MessageContext axis2MC = ((Axis2MessageContext) messageContext).
                    getAxis2MessageContext();
            ConfigurationContext cc = axis2MC.getConfigurationContext();
            synchronized (this) {

                if (!isResponse) {
                    initThrottle(messageContext, cc);
                }
            }         // if the access is success through concurrency throttle and if this is a request message
            // then do access rate based throttling
            if (!isResponse && throttle != null) {
                AuthenticationContext authContext = APISecurityUtils.getAuthenticationContext(messageContext);
                String tier;             if (authContext != null) {
                    AccessInformation info = null;
                    try {

                        String ipBasedKey = (String) ((TreeMap) axis2MC.
                                getProperty("TRANSPORT_HEADERS")).get("X-Forwarded-For");
                        if (ipBasedKey == null) {
                            ipBasedKey = (String) axis2MC.getProperty("REMOTE_ADDR");
                        }
                        tier = authContext.getApplicationTier();
                        ThrottleContext apiThrottleContext =
                                ApplicationThrottleController.
                                        getApplicationThrottleContext(messageContext, cc, tier);
                        //    if (isClusteringEnable) {
                        //      applicationThrottleContext.setConfigurationContext(cc);
                        apiThrottleContext.setThrottleId(id);
                        info = applicationRoleBasedAccessController.canAccess(apiThrottleContext,
                                                                              ipBasedKey, tier);
                        canAccess = info.isAccessAllowed();
                    } catch (ThrottleException e) {
                        handleException("Error while trying evaluate IPBased throttling policy", e);
                    }
                }
            }         if (!canAccess) {
                handleThrottleOut(messageContext);
                return false;
            }

            return canAccess;
        }    


    private void initThrottle(MessageContext synCtx, ConfigurationContext cc) {
            if (policyKey == null) {
                throw new SynapseException("Throttle policy unspecified for the API");
            }         Entry entry = synCtx.getConfiguration().getEntryDefinition(policyKey);
            if (entry == null) {
                handleException("Cannot find throttling policy using key: " + policyKey);
                return;
            }
            Object entryValue = null;
            boolean reCreate = false;         if (entry.isDynamic()) {
                if ((!entry.isCached()) || (entry.isExpired()) || throttle == null) {
                    entryValue = synCtx.getEntry(this.policyKey);
                    if (this.version != entry.getVersion()) {
                        reCreate = true;
                    }
                }
            } else if (this.throttle == null) {
                entryValue = synCtx.getEntry(this.policyKey);
            }         if (reCreate || throttle == null) {
                if (entryValue == null || !(entryValue instanceof OMElement)) {
                    handleException("Unable to load throttling policy using key: " + policyKey);
                    return;
                }
                version = entry.getVersion();             try {
                    // Creates the throttle from the policy
                    throttle = ThrottleFactory.createMediatorThrottle(
                            PolicyEngine.getPolicy((OMElement) entryValue));

                } catch (ThrottleException e) {
                    handleException("Error processing the throttling policy", e);
                }
            }
        }



How to get MD5SUM of all files available in conf directory

We can use following command to get MD5SUM of all files available in the system. We can use this approach to check status of configuration file of multiple servers and check those are same or not.
find ./folderName -type f -exec md5sum {} \; > test.xml

How to use SAML2 grant type to generate access tokens in web applications (Generate access tokens programatically using SAML2 grant type). - WSO2 API Manager

Exchanging SAML2 bearer tokens with OAuth2 (SAML extension grant type)

SAML 2.0 is an XML-based protocol. It uses security tokens containing assertions to pass information about an enduser between a SAML authority and a SAML consumer.
A SAML authority is an identity provider (IDP) and a SAML consumer is a service provider (SP).
A lot of enterprise applications use SAML2 to engage a third-party identity provider to grant access to systems that are only authenticated against the enterprise application.
These enterprise applications might need to consume OAuth-protected resources through APIs, after validating them against an OAuth2.0 authentication server.
However, an enterprise application that already has a working SAML2.0 based SSO infrastructure between itself and the IDP prefers to use the existing trust relationship, even if the OAuth authorization server is entirely different from the IDP. The SAML2 Bearer Assertion Profile for OAuth2.0 helps leverage this existing trust relationship by presenting the SAML2.0 token to the authorization server and exchanging it to an OAuth2.0 access token.

You can use SAML grant type for web applications to generate tokens.
https://docs.wso2.com/display/AM160/Token+API#TokenAPI-ExchangingSAML2bearertokenswithOAuth2(SAMLextensiongranttype)


Sample curl command .
curl -k -d "grant_type=urn:ietf:params:oauth:grant-type:saml2-bearer&assertion=&scope=PRODUCTION" -H "Authorization: Basic SVpzSWk2SERiQjVlOFZLZFpBblVpX2ZaM2Y4YTpHbTBiSjZvV1Y4ZkM1T1FMTGxDNmpzbEFDVzhh, Content-Type: application/x-www-form-urlencoded" https://serverurl/token

How to invoke token API from web app and get token programmatically.

To generate user access token using SAML assertion you can add following code block inside your web application.
When you login to your app using SSO there would be access you will get SAML response. You can store that in application session and use it to get token whenever requires.



Please refer following code for Access token issuer.

package com.test.org.oauth2;
import org.apache.amber.oauth2.client.OAuthClient;
import org.apache.amber.oauth2.client.URLConnectionClient;
import org.apache.amber.oauth2.client.request.OAuthClientRequest;
import org.apache.amber.oauth2.common.token.OAuthToken;
import org.apache.catalina.Session;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class AccessTokenIssuer {
    private static Log log = LogFactory.getLog(AccessTokenIssuer.class);
    private Session session;
    private static OAuthClient oAuthClient;

    public static void init() {
        if (oAuthClient == null) {
            oAuthClient = new OAuthClient(new URLConnectionClient());
        }
    }

    public AccessTokenIssuer(Session session) {
        init();
        this.session = session;
    }

    public String getAccessToken(String consumerKey, String consumerSecret, GrantType grantType)
            throws Exception {
        OAuthToken oAuthToken = null;

        if (session == null) {
            throw new Exception("Session object is null");
        }
// You need to implement logic for this operation according to your system design. some url
        String oAuthTokenEndPoint = "token end point url"

        if (oAuthTokenEndPoint == null) {
            throw new Exception("OAuthTokenEndPoint is not set properly in digital_airline.xml");
        }


        String assertion = "";
        if (grantType == GrantType.SAML20_BEARER_ASSERTION) {
    // You need to implement logic for this operation according to your system design
            String samlResponse = "get SAML response from session";
    // You need to implement logic for this operation according to your system design
            assertion = "get assertion from SAML response";
        }
        OAuthClientRequest accessRequest = OAuthClientRequest.
                tokenLocation(oAuthTokenEndPoint)
                .setGrantType(getAmberGrantType(grantType))
                .setClientId(consumerKey)
                .setClientSecret(consumerSecret)
                .setAssertion(assertion)
                .buildBodyMessage();
        oAuthToken = oAuthClient.accessToken(accessRequest).getOAuthToken();

        session.getSession().setAttribute("OAUTH_TOKEN" , oAuthToken);
        session.getSession().setAttribute("LAST_ACCESSED_TIME" , System.currentTimeMillis());

        return oAuthToken.getAccessToken();
    }

    private static org.apache.amber.oauth2.common.message.types.GrantType getAmberGrantType(
            GrantType grantType) {
        if (grantType == GrantType.SAML20_BEARER_ASSERTION) {
            return org.apache.amber.oauth2.common.message.types.GrantType.SAML20_BEARER_ASSERTION;
        } else if (grantType == GrantType.CLIENT_CREDENTIALS) {
            return org.apache.amber.oauth2.common.message.types.GrantType.CLIENT_CREDENTIALS;
        } else if (grantType == GrantType.REFRESH_TOKEN) {
            return org.apache.amber.oauth2.common.message.types.GrantType.REFRESH_TOKEN;
        } else {
            return org.apache.amber.oauth2.common.message.types.GrantType.PASSWORD;
        }
    }
}


After you login to system get session object and initiate access token issuer as follows.
AccessTokenIssuer accessTokenIssuer = new AccessTokenIssuer(session);

Then keep reference for that object during session.
Then when you need access token request token as follows. You need to pass consumer key and secret key.

tokenResponse = accessTokenIssuer.getAccessToken(key,secret, GrantType.SAML20_BEARER_ASSERTION);

Then you will get access token and you can use it as required.

How to change endpoit configurations, timeouts of already created large number of APIs - WSO2 API Manager

How to add additional properties for already create APIs. Sometimes in deployments we may need to change endpoint configurations and some other parameters after we created them.
For this we can go to management console, published and change them. But if you have large number of APIs that may be extremely hard. In this post lets see how we can do it for batch of API.

Please note that test this end to end before you push this change to production deployment. And also please note that some properties will be stored in registry, database and synapse configurations. So we need to change all 3 places. In this example we will consider endpoint configurations only(which available on registry and synapse).

Changing velocity template will work for new APIs. But when it comes to already published APIs, you have to do following process if you are not modifying it manually.

Write simple application to change synapse configuration and add new properties(as example we can consider timeout value).
 Use a checkin/checkout client to edit the registry files with the new timeout value.
   you can follow below mentioned steps to use the checkin/checkout client,
 Download Governance Registry binary from http://wso2.com/products/governance-registry/ and extract the zip file.
 Copy the content of Governance Registry in to APIM home.
 Go into the bin directory of the Governance Registry directory.
 Run the following command to checkout registry files to your local repository.
         ./checkin-client.sh co https://localhost:9443/registry/path -u admin -p admin  (linux environment)
           checkin-client.bat co https://localhost:9443/registry/path -u admin -p admin (windows environment)
        
Here the path is where your registry files are located. Normally API meta data will be listed under each provider '_system/governance/apimgt/applicationdata/provider'.

Once you run this command, registry files will be downloaded to your Governance Registry/bin directory. You can find the directories with user names who created the API.
Inside those directories there are files with same name 'api' in the location of '{directory with name of the api}/{directory with version of the api}/_system/governance
/apimgt/applicationdata/provider/{directory with name of the user}\directory with name of the api}/{directory with version of the api}' and you can edit the timeout value by
using a batch operation(shell script or any other way).

Then you have to checkin what you have changed by using the following command.
     ./checkin-client.sh ci https://localhost:9443/registry/path -u admin -p admin  (linux)
      checkin-client.bat ci https://localhost:9443/registry/path -u admin -p admin (windows)
   

Open APIM console and click on browse under resources. Provide the loaction as '/_system/governance/apimgt/applicationdata/provider'. Inside the {user name} directory
there are some directories with your API names. Open the 'api' files inside those directories and make sure the value has been updated.

Its recommend to change both registry and synapse configuration. This change will not be applicable to all properties available in API Manager.
This solution specifically designed for endpoint configurations such as time outs etc.

How to add secondry user store domain name to SAML response from shibboleth side. WSO2 Identity server SSO with secondary user store.

When we configure shibboleth as identity provider in WSO2 Identity server as described in this article(http://xacmlinfo.org/2014/12/04/federatation-shibboleth/) deployment would be something like below.

http://i0.wp.com/xacmlinfo.org/wp-content/uploads/2014/12/sidp0.png



In this case shibboleth will act as identity provider for WSO2 IS and will provide SAML assertion to WSO2 IS. But actual permission check will happen from IS side and we may need complete user name for that. If we configured user store as secondary user store then user store domain should be part of name. But shibboleth do not know about secondary user store. So in IS side you will username instead of DomainName/UserName. Then it will be an issue if we try to validate permissions per user.

To over come this we can configure shibboleth to send domain aware user name from their end. Let say domain name is LDAP-Domain then we can set it from shibboleth side with following configuration. Then it will send user name like this LDAP-Domain/userName.

 (attribute-resolver.xml)

    <!-- This is the NameID value we send to the WS02 Identity Server. -->
    <resolver:AttributeDefinition xsi:type="ad:Script" id="eduPersonPrincipalNameWSO2">
        <resolver:Dependency ref="eduPersonPrincipalName" />

        <resolver:AttributeEncoder xsi:type="enc:SAML2StringNameID" nameFormat="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" />

        <ad:Script>
            <![CDATA[
                importPackage(Packages.edu.internet2.middleware.shibboleth.common.attribute.provider);

                eduPersonPrincipalNameWSO2 = new BasicAttribute("eduPersonPrincipalNameWSO2");
                eduPersonPrincipalNameWSO2.getValues().add("LDAP-Domain/" + eduPersonPrincipalName.getValues().get(0));
            ]]>
        </ad:Script>
    </resolver:AttributeDefinition>

How to write custom throttle handler to throttle requests based on IP address - WSO2 API Manager

Please find the sample source code for custom throttle handler to throttle requests based on IP address. Based on your requirements you can change the logic here.

package org.wso2.carbon.apimgt.gateway.handlers.throttling; import org.apache.axiom.om.OMAbstractFactory;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.OMFactory;
import org.apache.axiom.om.OMNamespace;
import org.apache.axis2.context.ConfigurationContext;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpStatus;
import org.apache.neethi.PolicyEngine;
import org.apache.synapse.Mediator;
import org.apache.synapse.MessageContext;
import org.apache.synapse.SynapseConstants;
import org.apache.synapse.SynapseException;
import org.apache.synapse.config.Entry;
import org.apache.synapse.core.axis2.Axis2MessageContext;
import org.apache.synapse.rest.AbstractHandler;
import org.wso2.carbon.apimgt.gateway.handlers.Utils;
import org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityUtils;
import org.wso2.carbon.apimgt.gateway.handlers.security.AuthenticationContext;
import org.wso2.carbon.apimgt.impl.APIConstants;
import org.wso2.carbon.throttle.core.AccessInformation;
import org.wso2.carbon.throttle.core.RoleBasedAccessRateController;
import org.wso2.carbon.throttle.core.Throttle;
import org.wso2.carbon.throttle.core.ThrottleContext;
import org.wso2.carbon.throttle.core.ThrottleException;
import org.wso2.carbon.throttle.core.ThrottleFactory;

import java.util.Map;
import java.util.TreeMap;


public class IPBasedThrottleHandler extends AbstractHandler {

    private static final Log log = LogFactory.getLog(IPBasedThrottleHandler.class);

    /** The Throttle object - holds all runtime and configuration data */
    private volatile Throttle throttle;

    private RoleBasedAccessRateController applicationRoleBasedAccessController;

    /** The key for getting the throttling policy - key refers to a/an [registry] entry    */
    private String policyKey = null;
    /** The concurrent access control group id */
    private String id;
    /** Version number of the throttle policy */
    private long version;

    public IPBasedThrottleHandler() {
        this.applicationRoleBasedAccessController = new RoleBasedAccessRateController();
    }

    public boolean handleRequest(MessageContext messageContext) {
        return doThrottle(messageContext);
    }

    public boolean handleResponse(MessageContext messageContext) {
        return doThrottle(messageContext);
    }

    private boolean doThrottle(MessageContext messageContext) {
        boolean canAccess = true;
        boolean isResponse = messageContext.isResponse();
        org.apache.axis2.context.MessageContext axis2MC = ((Axis2MessageContext) messageContext).
                getAxis2MessageContext();
        ConfigurationContext cc = axis2MC.getConfigurationContext();
        synchronized (this) {

            if (!isResponse) {
                initThrottle(messageContext, cc);
            }
        }         // if the access is success through concurrency throttle and if this is a request message
        // then do access rate based throttling
        if (!isResponse && throttle != null) {
            AuthenticationContext authContext = APISecurityUtils.getAuthenticationContext(messageContext);
            String tier;             if (authContext != null) {
                AccessInformation info = null;
                try {

                    String ipBasedKey = (String) ((TreeMap) axis2MC.
                            getProperty("TRANSPORT_HEADERS")).get("X-Forwarded-For");
                    if (ipBasedKey == null) {
                        ipBasedKey = (String) axis2MC.getProperty("REMOTE_ADDR");
                    }
                    tier = authContext.getApplicationTier();
                    ThrottleContext apiThrottleContext =
                            ApplicationThrottleController.
                                    getApplicationThrottleContext(messageContext, cc, tier);
                    //    if (isClusteringEnable) {
                    //      applicationThrottleContext.setConfigurationContext(cc);
                    apiThrottleContext.setThrottleId(id);
                    info = applicationRoleBasedAccessController.canAccess(apiThrottleContext,
                                                                          ipBasedKey, tier);
                    canAccess = info.isAccessAllowed();
                } catch (ThrottleException e) {
                    handleException("Error while trying evaluate IPBased throttling policy", e);
                }
            }
        }         if (!canAccess) {
            handleThrottleOut(messageContext);
            return false;
        }

        return canAccess;
    }     private void initThrottle(MessageContext synCtx, ConfigurationContext cc) {
        if (policyKey == null) {
            throw new SynapseException("Throttle policy unspecified for the API");
        }         Entry entry = synCtx.getConfiguration().getEntryDefinition(policyKey);
        if (entry == null) {
            handleException("Cannot find throttling policy using key: " + policyKey);
            return;
        }
        Object entryValue = null;
        boolean reCreate = false;         if (entry.isDynamic()) {
            if ((!entry.isCached()) || (entry.isExpired()) || throttle == null) {
                entryValue = synCtx.getEntry(this.policyKey);
                if (this.version != entry.getVersion()) {
                    reCreate = true;
                }
            }
        } else if (this.throttle == null) {
            entryValue = synCtx.getEntry(this.policyKey);
        }         if (reCreate || throttle == null) {
            if (entryValue == null || !(entryValue instanceof OMElement)) {
                handleException("Unable to load throttling policy using key: " + policyKey);
                return;
            }
            version = entry.getVersion();             try {
                // Creates the throttle from the policy
                throttle = ThrottleFactory.createMediatorThrottle(
                        PolicyEngine.getPolicy((OMElement) entryValue));

            } catch (ThrottleException e) {
                handleException("Error processing the throttling policy", e);
            }
        }
    }     public void setId(String id) {
        this.id = id;
    }     public String getId(){
        return id;
    }     public void setPolicyKey(String policyKey){
        this.policyKey = policyKey;
    }     public String gePolicyKey(){
        return policyKey;
    }     private void handleException(String msg, Exception e) {
        log.error(msg, e);
        throw new SynapseException(msg, e);
    }     private void handleException(String msg) {
        log.error(msg);
        throw new SynapseException(msg);
    }     private OMElement getFaultPayload() {
        OMFactory fac = OMAbstractFactory.getOMFactory();
        OMNamespace ns = fac.createOMNamespace(APIThrottleConstants.API_THROTTLE_NS,
                                               APIThrottleConstants.API_THROTTLE_NS_PREFIX);
        OMElement payload = fac.createOMElement("fault", ns);         OMElement errorCode = fac.createOMElement("code", ns);
     errorCode.setText(String.valueOf(APIThrottleConstants.THROTTLE_OUT_ERROR_CODE));
        OMElement errorMessage = fac.createOMElement("message", ns);
        errorMessage.setText("Message Throttled Out");
        OMElement errorDetail = fac.createOMElement("description", ns);
        errorDetail.setText("You have exceeded your quota");

        payload.addChild(errorCode);
        payload.addChild(errorMessage);
        payload.addChild(errorDetail);
        return payload;
    }     private void handleThrottleOut(MessageContext messageContext) {
        messageContext.setProperty(SynapseConstants.ERROR_CODE, 900800);
        messageContext.setProperty(SynapseConstants.ERROR_MESSAGE, "Message throttled out");

        Mediator sequence = messageContext.getSequence(APIThrottleConstants.API_THROTTLE_OUT_HANDLER);
        // Invoke the custom error handler specified by the user
        if (sequence != null && !sequence.mediate(messageContext)) {
            // If needed user should be able to prevent the rest of the fault handling
            // logic from getting executed
            return;
        }         // By default we send a 503 response back
        if (messageContext.isDoingPOX() || messageContext.isDoingGET()) {
            Utils.setFaultPayload(messageContext, getFaultPayload());
        } else {
            Utils.setSOAPFault(messageContext, "Server", "Message Throttled Out",
                               "You have exceeded your quota");
        }
        org.apache.axis2.context.MessageContext axis2MC = ((Axis2MessageContext) messageContext).
                getAxis2MessageContext();

        if (Utils.isCORSEnabled()) {
            /* For CORS support adding required headers to the fault response */
            Map headers = (Map) axis2MC.getProperty(org.apache.axis2.context.MessageContext.TRANSPORT_HEADERS);
            headers.put(APIConstants.CORSHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, Utils.getAllowedOrigin((String)headers.get("Origin")));
            headers.put(APIConstants.CORSHeaders.ACCESS_CONTROL_ALLOW_METHODS, Utils.getAllowedMethods());
            headers.put(APIConstants.CORSHeaders.ACCESS_CONTROL_ALLOW_HEADERS, Utils.getAllowedHeaders());
            axis2MC.setProperty(org.apache.axis2.context.MessageContext.TRANSPORT_HEADERS, headers);
        }
        Utils.sendFault(messageContext, HttpStatus.SC_SERVICE_UNAVAILABLE);
    }
}

As listed above your custom handler class is : "org.wso2.carbon.apimgt.gateway.handlers.throttling.IPBasedThrottleHandler", the following will be the handler definition for your API.


<handler class="org.wso2.carbon.apimgt.gateway.handlers.throttling.IPBasedThrottleHandler">
<property name="id" value="A"/>
<property name="policyKey" value="gov:/apimgt/applicationdata/tiers.xml"/>
</handler>

Then try to invoke API and see how throttling works.

Empowering the Future of API Management: Unveiling the Journey of WSO2 API Platform for Kubernetes (APK) Project and the Anticipated Alpha Release

  Introduction In the ever-evolving realm of API management, our journey embarked on the APK project eight months ago, and now, with great a...