Thursday, March 19, 2015

How to fine tune API Manager 1.8.0 to get maximum TPS and minimum response time


Here in this post i will discuss about API Manager 1.8.0 performance tuning. I have tested this is below mentioned deployment. Please note that this results can vary depend on you hardware server load and network. And this is not even fully optimized environment and probably you may go beyond this with better hardware+ network and configuration combination according to your use case.

Server specifications


System Information
Manufacturer: Fedora Project
Product Name: OpenStack Nova
Version: 2014.2.1-1.el7.centos

4 X CPU cores
Processor Information
Socket Designation: CPU 1
Type: Central Processor
Family: Other
Manufacturer: Bochs
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled

Memory Device
Total Width: 64 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Type: RAM



Deployment Details


Deployment 01.
2 gateways (each run on dedicated machine)
2 key managers(each run on dedicated machine)
MySQL database server
1 dedicated machine to run Jmeter

Deployment 02.
1 gateways
1 key managers
MySQL database server
1 dedicated machine to run Jmeter  

 

Configuration changes.

Gateway changes.
Enable WS key validation for key management.
Edit /home/sanjeewa/work/wso2am-1.8.0/repository/conf/api-manager.xml with following configurations.
[Default value is ThriftClient]
<KeyValidatorClientType>WSClient</KeyValidatorClientType> 

[Default value is true]
<EnableThriftServer>false</EnableThriftServer 
Other than this all configurations will be default configuration.
However please note that each gateway should configure to communicate key manager.

Key Manager changes.
Edit /home/sanjeewa/work/wso2am-1.8.0/repository/conf/api-manager.xml with following configurations.
[Default value is true]
<EnableThriftServer>false</EnableThriftServer> 
No need to run thrift server there as we use WS client to key validation calls.
Both gateway and key managers configured with mysql servers. For this i configured usr manager, api manager and registry database with mysql servers.

Tuning parameters applied.

Gateway nodes.

01. Change synapse configurationsAdd following entries to /home/sanjeewa/work/wso2am-1.8.0/repository/conf/synapse.properties file.
synapse.threads.core=100
synapse.threads.max=250
synapse.threads.keepalive=5
synapse.threads.qlen=1000



02. Disable HTTP access logs
Since we are testing gateway functionality here we should not much worry about http access logs. However we may need to enable this to track access. But for this deployment we assume key managers are running in DMZ and no need track http access. For gateways most of the time this does not require as we do not expose servlet ports to outside(normally we only open 8243 and 8280).
Add following entry to /home/sanjeewa/work/wso2am-1.8.0/repository/conf/log4j.properties file.

log4j.logger.org.apache.synapse.transport.http.access=OFF

Remove following entry from /wso2am-1.8.0/repository/conf/tomcat/catalina-server.xml to disable http access logs.


          <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="combined" />





03. Tune parameters in axis2client.xml file. We will be using axis2 client to communicate from gateway to key manager for key validation. For this edit wso2am-1.8.0/repository/conf/axis2/axis2_client.xml and update following entries.

    <parameter name="defaultMaxConnPerHost">1000</parameter>
    <parameter name="maxTotalConnections">30000</parameter>



Key manager nodes.

01. Disable HTTP access logs
Since we are testing gateway functionality here we should not much worry about http access logs. However we may need to enable this to track access. But for this deployment we assume key managers are running in DMZ and no need track http access.

Add following entry to /home/sanjeewa/work/wso2am-1.8.0/repository/conf/log4j.properties file.

log4j.logger.org.apache.synapse.transport.http.access=OFF


02. Change DBCP connection parameters / Datasource configurations.
There can be argument on these parameters. Specially disable validation query. But when we have high concurrency and well performing database servers we may disable this as created connections are heavily used. And on the other hand connection may work when we validate it but when we really use it connection may not work. So as per my understanding there is no issue with disabling in high concurrency scenario.

Also i added following additional parameters to optimize database connection pool.
<maxWait>60000</maxWait>
<initialSize>20</initialSize>
<maxActive>150</maxActive>
<maxIdle>60</maxIdle>
<minIdle>40</minIdle>
<testOnBorrow>false</testOnBorrow>

If you dont want to disable validation query you may use following configuration(here i increased validation interval to avoid frequent query validation).

<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>120000</validationInterval>



03. Tuning Tomcat parameters in key manager node.
This is important because we call key validation web service service from gateway.
change following properties in this(/home/sanjeewa/work/wso2am-1.8.0/repository/conf/tomcat/catalina-server.xml) file.


Here is the brief description about changed parameters. Also i added description for each field copied from this(http://tomcat.apache.org/tomcat-7.0-doc/config/http.html) document for your reference.

I updated acceptorThreadCount to 4(default it was 2) because in my machine i have 4 cores.
However after adding this change i noticed considerable reduction of CPU usage of each core.

Increased maxThreads to 750(default value was 250)
The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200. If an executor is associated with this connector, this attribute is ignored as the connector will execute tasks using the executor rather than an internal thread pool.

Increased minSpareThreads to 250 (default value was 50)
The minimum number of threads always kept running. If not specified, the default of 10 is used.

Increased maxKeepAliveRequests to 400 (default value was 200)
The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100.

Increased acceptCount to 400 (default value was 200)
The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.

compression="off"
Disabled compression. However this might not effective as we do not use compressed data format.



<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="9443"
               bindOnInit="false"
               sslProtocol="TLS"
               maxHttpHeaderSize="8192"
               acceptorThreadCount="2"
               maxThreads="750"
               minSpareThreads="250"
               disableUploadTimeout="false"
               enableLookups="false"
               connectionUploadTimeout="120000"
               maxKeepAliveRequests="400"
               acceptCount="400"
               server="WSO2 Carbon Server"
               clientAuth="false"
               compression="off"
               scheme="https"
               secure="true"
               SSLEnabled="true"
               compressionMinSize="2048"
               noCompressionUserAgents="gozilla, traviata"                
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text
/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
               URIEncoding="UTF-8"/>





Testing

Test 01 - Clustered gateway/key manager test(2 nodes)

For this test we used 10000 tokens and 150 concurrency per single gateway server. Test carried out 20 minutes to avoid caching effect on performance figures.


-->
Round 01 with default configuration parameters
sampler_labelreq_countaveragemedian90%_lineminmaxerror%ratebandwidth
GW1310498650235014967602587.4236482276.629596
GW2315110949205302301302625.8847782310.470884
TOTAL625609550215104967605213.212664587.016218










Round 02 with http access logs disabled on key manager/ DB tuned/ with test on barrow/ synapse tuned
sampler_labelreq_countaveragemedian90%_lineminmaxerror%ratebandwidth
GW151257042220351994104271.9077093758.77817
GW253617252118330169104468.0334233931.345814
TOTAL104874292219340994108739.0580847689.347005










Round 03 with http access logs disabled/ DB tuned/ with test on barrow/ synapse tuned
sampler_labelreq_countaveragemedian90%_lineminmaxerror%ratebandwidth
GW1551192222193511425704592.3154284040.699415
GW2580467420173401190904836.3819664255.449367
TOTAL1131659621183401425709428.4774018295.955213










Round 04 with http access logs disabled/ DB tuned and removed test on barrow/ synapse tuned
sampler_labelreq_countaveragemedian90%_lineminmaxerror%ratebandwidth
GW1563604923203511496104697.2790024133.05506
GW2585450522183401033204879.4698224293.361631
TOTAL1149055422193501496109576.6028798426.288275
 


tps.png

  Results table

dsdsadsad.png

Response Time graph

Screenshot from 2015-03-18 13:07:09.png

Transactions per second graph


For 150 concurrency cluster(2 nodes) node
Average TPS - 4687.85
Average response time - 21ms
Average delay added by gateway - 16ms

Test 02 - Single gateway/key manager test


For this test we used 10000 tokens and 300 concurrency per single gateway server. Test carried out 20 minutes to avoid caching effect on performance figures.







Results table.

























Response Time Graph


For 300 concurrency 1 node
Average TPS -4956
Average response time - 55ms
Average delay added by gateway - 49ms

No comments:

Post a Comment