Here in this post i will discuss about API Manager 1.8.0 performance tuning. I have tested this is below mentioned deployment. Please note that this results can vary depend on you hardware server load and network. And this is not even fully optimized environment and probably you may go beyond this with better hardware+ network and configuration combination according to your use case.
Server specifications
System Information
Manufacturer: Fedora Project
Product Name: OpenStack Nova
Version: 2014.2.1-1.el7.centos
4 X CPU cores
Processor Information
Socket Designation: CPU 1
Type: Central Processor
Family: Other
Manufacturer: Bochs
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Memory Device
Total Width: 64 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Type: RAM
Deployment Details
Deployment 01.
2 gateways (each run on dedicated machine)
2 key managers(each run on dedicated machine)
MySQL database server
1 dedicated machine to run Jmeter
Deployment 02.
1 gateways
1 key managers
MySQL database server
1 dedicated machine to run Jmeter
Configuration changes.
Gateway changes.Edit /home/sanjeewa/work/wso2am-1.8.0/repository/conf/api-manager.xml with following configurations.
[Default value is ThriftClient]
<KeyValidatorClientType>WSClient</KeyValidatorClientType>
[Default value is true]
Other than this all configurations will be default configuration. <EnableThriftServer>false</EnableThriftServer>
However please note that each gateway should configure to communicate key manager.
Key Manager changes.
Edit /home/sanjeewa/work/wso2am-1.8.0/repository/conf/api-manager.xml with following configurations.
[Default value is true]
<EnableThriftServer>false</EnableThriftServer>
No need to run thrift server there as we use WS client to key validation calls. Both gateway and key managers configured with mysql servers. For this i configured usr manager, api manager and registry database with mysql servers.
Tuning parameters applied.
Gateway nodes.
synapse.threads.core=100
synapse.threads.max=250
synapse.threads.keepalive=5
synapse.threads.qlen=1000
02. Disable HTTP access logs
Since we are testing gateway functionality here we should not much worry about http access logs. However we may need to enable this to track access. But for this deployment we assume key managers are running in DMZ and no need track http access. For gateways most of the time this does not require as we do not expose servlet ports to outside(normally we only open 8243 and 8280).
Add following entry to /home/sanjeewa/work/wso2am-1.8.0/repository/conf/log4j.properties file.
log4j.logger.org.apache.synapse.transport.http.access=OFF
Remove following entry from /wso2am-1.8.0/repository/conf/tomcat/catalina-server.xml to disable http access logs.
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs" prefix="http_access_" suffix=".log" pattern="combined" />
03. Tune parameters in axis2client.xml file. We will be using axis2 client to communicate from gateway to key manager for key validation. For this edit wso2am-1.8.0/repository/conf/axis2/axis2_client.xml and update following entries.
<parameter name="defaultMaxConnPerHost">1000</parameter> <parameter name="maxTotalConnections">30000</parameter>
Key manager nodes.
Since we are testing gateway functionality here we should not much worry about http access logs. However we may need to enable this to track access. But for this deployment we assume key managers are running in DMZ and no need track http access.
Add following entry to /home/sanjeewa/work/wso2am-1.8.0/repository/conf/log4j.properties file.
log4j.logger.org.apache.synapse.transport.http.access=OFF
02. Change DBCP connection parameters / Datasource configurations.
There can be argument on these parameters. Specially disable validation query. But when we have high concurrency and well performing database servers we may disable this as created connections are heavily used. And on the other hand connection may work when we validate it but when we really use it connection may not work. So as per my understanding there is no issue with disabling in high concurrency scenario.
Also i added following additional parameters to optimize database connection pool.
<maxWait>60000</maxWait> <initialSize>20</initialSize> <maxActive>150</maxActive> <maxIdle>60</maxIdle> <minIdle>40</minIdle> <testOnBorrow>false</testOnBorrow>
If you dont want to disable validation query you may use following configuration(here i increased validation interval to avoid frequent query validation).
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>120000</validationInterval>
03. Tuning Tomcat parameters in key manager node.
This is important because we call key validation web service service from gateway.
change following properties in this(/home/sanjeewa/work/wso2am-1.8.0/repository/conf/tomcat/catalina-server.xml) file.
Here is the brief description about changed parameters. Also i added description for each field copied from this(http://tomcat.apache.org/tomcat-7.0-doc/config/http.html) document for your reference.
I updated acceptorThreadCount to 4(default it was 2) because in my machine i have 4 cores.
However after adding this change i noticed considerable reduction of CPU usage of each core.
Increased maxThreads to 750(default value was 250)
The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200. If an executor is associated with this connector, this attribute is ignored as the connector will execute tasks using the executor rather than an internal thread pool.
Increased minSpareThreads to 250 (default value was 50)
The minimum number of threads always kept running. If not specified, the default of 10 is used.
Increased maxKeepAliveRequests to 400 (default value was 200)
The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100.
Increased acceptCount to 400 (default value was 200)
The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.
compression="off"
Disabled compression. However this might not effective as we do not use compressed data format.
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
bindOnInit="false"
sslProtocol="TLS"
maxHttpHeaderSize="8192"
acceptorThreadCount="2"
maxThreads="750"
minSpareThreads="250"
disableUploadTimeout="false"
enableLookups="false"
connectionUploadTimeout="120000"
maxKeepAliveRequests="400"
acceptCount="400"
server="WSO2 Carbon Server"
clientAuth="false"
compression="off"
scheme="https"
secure="true"
SSLEnabled="true"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text
/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
URIEncoding="UTF-8"/>
Testing
Test 01 - Clustered gateway/key manager test(2 nodes)-->
| Round 01 with default configuration parameters | |||||||||
| sampler_label | req_count | average | median | 90%_line | min | max | error% | rate | bandwidth |
| GW1 | 3104986 | 50 | 23 | 50 | 1 | 49676 | 0 | 2587.423648 | 2276.629596 |
| GW2 | 3151109 | 49 | 20 | 53 | 0 | 23013 | 0 | 2625.884778 | 2310.470884 |
| TOTAL | 6256095 | 50 | 21 | 51 | 0 | 49676 | 0 | 5213.21266 | 4587.016218 |
| Round 02 with http access logs disabled on key manager/ DB tuned/ with test on barrow/ synapse tuned | |||||||||
| sampler_label | req_count | average | median | 90%_line | min | max | error% | rate | bandwidth |
| GW1 | 5125704 | 22 | 20 | 35 | 1 | 9941 | 0 | 4271.907709 | 3758.77817 |
| GW2 | 5361725 | 21 | 18 | 33 | 0 | 1691 | 0 | 4468.033423 | 3931.345814 |
| TOTAL | 10487429 | 22 | 19 | 34 | 0 | 9941 | 0 | 8739.058084 | 7689.347005 |
| Round 03 with http access logs disabled/ DB tuned/ with test on barrow/ synapse tuned | |||||||||
| sampler_label | req_count | average | median | 90%_line | min | max | error% | rate | bandwidth |
| GW1 | 5511922 | 22 | 19 | 35 | 1 | 14257 | 0 | 4592.315428 | 4040.699415 |
| GW2 | 5804674 | 20 | 17 | 34 | 0 | 11909 | 0 | 4836.381966 | 4255.449367 |
| TOTAL | 11316596 | 21 | 18 | 34 | 0 | 14257 | 0 | 9428.477401 | 8295.955213 |
| Round 04 with http access logs disabled/ DB tuned and removed test on barrow/ synapse tuned | |||||||||
| sampler_label | req_count | average | median | 90%_line | min | max | error% | rate | bandwidth |
| GW1 | 5636049 | 23 | 20 | 35 | 1 | 14961 | 0 | 4697.279002 | 4133.05506 |
| GW2 | 5854505 | 22 | 18 | 34 | 0 | 10332 | 0 | 4879.469822 | 4293.361631 |
| TOTAL | 11490554 | 22 | 19 | 35 | 0 | 14961 | 0 | 9576.602879 | 8426.288275 |
Results table
Response Time graph
Transactions per second graph
For 150 concurrency cluster(2 nodes) node
Average TPS - 4687.85
Average response time - 21ms
Average delay added by gateway - 16ms
Average TPS - 4687.85
Average response time - 21ms
Average delay added by gateway - 16ms
Test 02 - Single gateway/key manager test
Results table.
Response Time Graph
For 300 concurrency 1 node
Average TPS -4956
Average response time - 55ms
Average delay added by gateway - 49ms
No comments:
Post a Comment