How to write test case using wiremock and test Mutual SSL enabled Backend Service invocation

Add following dependencies to the pom file so we can use wiremock for testing. I specifically added below slf4j versions to dependency as they required. Also I excluded some of the components as they started sending errors. Same way if you get any errors just type mvn dependency:tree and get all dependencies. Then you can exclude problematic components.

<dependency>

             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-log4j12</artifactId>
             <version>1.7.7</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-api</artifactId>
             <version>1.7.7</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.hamcrest</groupId>
             <artifactId>hamcrest-all</artifactId>
             <version>1.3</version>
             <scope>test</scope>
        </dependency>
         <dependency>
                     <groupId>com.github.tomakehurst</groupId>
                     <artifactId>wiremock</artifactId>
                     <version>2.5.0</version>
                     <exclusions>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>slf4j-jdk14</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>jcl-over-slf4j</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>org.slf4j</groupId>
                             <artifactId>log4j-over-slf4j</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>com.fasterxml.jackson.core</groupId>
                             <artifactId>jackson-annotations</artifactId>
                         </exclusion>
                         <exclusion>
                             <groupId>com.fasterxml.jackson.core</groupId>
                             <artifactId>jackson-core</artifactId>
                         </exclusion>
                     </exclusions>
        </dependency>

Following is my test class.

package org.test.testpkg;
import jdk.nashorn.internal.objects.NativeObject;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import com.github.tomakehurst.wiremock.junit.WireMockRule;
import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
import java.io.File;
public class MutualSSLTest {
    private static final Log log = LogFactory.getLog(MutualSSLTest.class);
    //Key store and client trustore paths and passwords
    private static final String KEYSTORE_FILE_PATH =
            "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "server" + File.separator + "wso2carbon.jks";
    private static final String TRUSTSTORE_FILE_PATH =
            "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "server" + File.separator + "client-truststore.jks";
    private static final String KEYSTORE_FILE_PATH_CLIENT =
             "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "client" + File.separator + "wso2carbon.jks";
    private static final String TRUSTSTORE_FILE_PATH_CLIENT =
             "src" + File.separator + "test" + File.separator + "resources" + File.separator + "security"
                    + File.separator + "client" + File.separator + "client-truststore.jks";
    public void testAPIProvider() {
    }
    @Rule
    public WireMockRule wireMockRule;
    @Test
    public void testMutualSSLEnabledBackend() {
//Create wiremock rule by providing SSL configuratios. Here we need to pass keystore/trustore, port and other required information.
        wireMockRule = new WireMockRule(wireMockConfig()
                .httpsPort(8081)
                .needClientAuth(true)
                .trustStoreType("JKS")
                .keystoreType("JKS")
                .keystorePath(KEYSTORE_FILE_PATH)
                .trustStorePath(TRUSTSTORE_FILE_PATH)
                .trustStorePassword("wso2carbon")
                .keystorePassword("wso2carbon"));
        wireMockRule.start();
        // Mock service for test endpoint. This will return 200 for http head method.
        wireMockRule.stubFor(head(urlEqualTo("/test"))
                .willReturn(aResponse()
                        .withStatus(200)
                        .withBody("{success}")
                        .withHeader("Content-Type", "application/json")
                ));
        try {
   //Then i will set keystore and client trustore to system properties.
            System.setProperty("javax.net.ssl.keyStoreType", "JKS");
            System.setProperty("javax.net.ssl.keyStore", KEYSTORE_FILE_PATH_CLIENT);
            System.setProperty("javax.net.ssl.keyStorePassword", "wso2carbon");
            System.setProperty("javax.net.ssl.trustStore", TRUSTSTORE_FILE_PATH_CLIENT);
            System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon");
   //Now i will invoke my utility method and call created service
            org.mozilla.javascript.NativeObject obj =
                    HostObjectUtils.sendHttpHEADRequest("https://localhost:8081/test",
                            "404");
            //Then i will assert response.
   Assert.assertEquals("success", obj.get("response"));
        } catch (Exception e) {
            e.printStackTrace();
        }
        wireMockRule.resetAll();
        wireMockRule.stop();
    }
}

In my utility class i have following method to call HTTP service and get response.

    /**
     * Validate the backend by sending HTTP HEAD
     *
     * @param urlVal - backend URL
     * @param invalidStatusCodesRegex - Regex for the invalid status code
     * @return - status of HTTP HEAD Request to backend
     */
    public static NativeObject sendHttpHEADRequest(String urlVal, String invalidStatusCodesRegex) {
        boolean isConnectionError = true;
        String response = null;
        NativeObject data = new NativeObject();
        //HttpClient client = new DefaultHttpClient();
        HttpHead head = new HttpHead(urlVal);
        //Change implementation to use http client as default http client do not work properly with mutual SSL.
        org.apache.commons.httpclient.HttpClient clientnew = new org.apache.commons.httpclient.HttpClient();
        // extract the host name and add the Host http header for sanity
        head.addHeader("Host", urlVal.replaceAll("https?://", "").replaceAll("(/.*)?", ""));
        clientnew.getParams().setParameter("http.socket.timeout", 4000);
        clientnew.getParams().setParameter("http.connection.timeout", 4000);
        HttpMethod method = new HeadMethod(urlVal);
        if (System.getProperty(APIConstants.HTTP_PROXY_HOST) != null &&
                System.getProperty(APIConstants.HTTP_PROXY_PORT) != null) {
            if (log.isDebugEnabled()) {
                log.debug("Proxy configured, hence routing through configured proxy");
            }
            String proxyHost = System.getProperty(APIConstants.HTTP_PROXY_HOST);
            String proxyPort = System.getProperty(APIConstants.HTTP_PROXY_PORT);
            clientnew.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY,
                    new HttpHost(proxyHost, Integer.parseInt(proxyPort)));
        }
        try {
            int statusCodeNew = clientnew.executeMethod(method);
            //Previous implementation
            // HttpResponse httpResponse = client.execute(head);
            String statusCode = String.valueOf(statusCodeNew);//String.valueOf(httpResponse.getStatusLine().getStatusCode());
            String reasonPhrase = String.valueOf(statusCodeNew);//String.valueOf(httpResponse.getStatusLine().getReasonPhrase());
            //If the endpoint doesn't match the regex which specify the invalid status code, it will return success.
            if (!statusCode.matches(invalidStatusCodesRegex)) {
                if (log.isDebugEnabled() && statusCode.equals(String.valueOf(HttpStatus.SC_METHOD_NOT_ALLOWED))) {
                    log.debug("Endpoint doesn't support HTTP HEAD");
                }
                response = "success";
                isConnectionError = false;
            } else {
                //This forms the real backend response to be sent to the client
                data.put("statusCode", data, statusCode);
                data.put("reasonPhrase", data, reasonPhrase);
                response = "";
                isConnectionError = false;
            }
        } catch (IOException e) {
            // sending a default error message.
            log.error("Error occurred while connecting to backend : " + urlVal + ", reason : " + e.getMessage(), e);
            String[] errorMsg = e.getMessage().split(": ");
            if (errorMsg.length > 1) {
                response = errorMsg[errorMsg.length - 1]; //This is to get final readable part of the error message in the exception and send to the client
                isConnectionError = false;
            }
        } finally {
            method.releaseConnection();
        }
        data.put("response", data, response);
        data.put("isConnectionError", data, isConnectionError);
        return data;
    }
}

Now we have successfully implemented mutual ssl test case. You can run test and verify this behavior. If you need to test negative impact then comment keystore password in client.
Then you will see errors in logs.

How to do zero downtime migration - WSO2 API Manager

Here in this post i will explain how we can do zero downtime migration in deployment.

Test Migration

First we can take backup of running system and create setup locally. Then we can perform migration and identify if there are any issues. Usually dump restore and setup take sometime based on deployment.

Then after that we can test APIs, resources etc. Also we can test extension points, integrations etc. 

Then after we completed above we can come up with exact steps on migrating. So we can run same in deployment backup(or staging deployment). Ideally it should work without issues. Then we can repeat this to verify exact steps.

Following steps need to follow(if need they can automate that)

• Create dump of current running setup
• Restore it to lower environment.
• Perform migration.
• Do sanity check

Then we need to analyze traffic pattern of the deployment for about one week or more to identify time slot for migration. After some analysis we can identify time window with minimum user interaction.

Then we need to perform all above mentioned steps in production server. But at this time we should test all use cases end to end.

Also we need to create one way synchup script to synchronize tokens to new deployment(tokens created within maintain window).

Real migration

Then comes real migration,

• Make API Manager store and publisher read only for maintain window(say 3 hours).
• Perform creating dump and restore to new environment.
• Do migration in new envirionment.
• Run synchup script to update new tokens created in original deployment.
• Then test functionalities using automated tests and manual tests.
• Once we are 100% confident we can do DNS switch.
• If everything works fine open traffic for store/publisher as well.

If you wish to do gradual traffic movement to new deployment it may bit different from above above mention approach. In that case we may need 2 way synch up scripts as tokens can created in both deployments.