UTF-8 string support in API manager - create APIs, Applications with different languages

Here are the steps i followed in order to work UTF8 functionality in API Manager 1.4.0.
01. Create API manager database using scripts found in API Manager distribution.
02. Change master-datasources.xml to point new API manager database.
Configuration is shown below

<url>jdbc:mysql://localhost:3306/database_name?relaxAutoCommit=true&amp;characterEncoding=UTF-8&lt;/url> 
                    <username>root</username> 
                    <password>root</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval>

03. Run alter database and table commands as follows.
   ALTER DATABASE database_name CHARACTER SET utf8 COLLATE utf8_general_ci;
   ALTER TABLE database_name_table_names CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

04. Run following command to check changes. All field values should contain utf8_unicode_ci, utf8_general_ci or desired type.
    show variables like 'collation%';

05. add following to /etc/mysql/my.cnf and restart MySQL server.
   collation-server = utf8_unicode_ci
   init-connect='SET NAMES utf8'
   character-set-server = utf8

Also followed below instructions.
 validated that there was a character set filter in /repository/conf/tomcat/carbon/WEB-INF/web.xml
 updated /repository/conf/tomcat/catalina-server.xml and added URIEncoding="UTF-8" in both Connectors
 included -Dfile.encoding=UTF-8 in /bin/wso2server.sh java startup command

06. Start API manager and perform following actions.
    create API with Japanese string. API stored in API manager database and registry properly.
    create Application with Japanese name(application name persisted to database without issue).
    create subscription and generate keys. Invoke APIs.

Here i have attached screen-shots for store and publisher and how data stored in database. After applying those configurations i was able to perform API manager operations as expected.

How to avoid jmap error - Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

When you need to attach process to jmap you might get following error here i have described how we can avoid that.

Run fllowing to get process ID
 :jps
3252 Bootstrap
3513 Jps

Then run following to execute jmap
:jmap -heap:format=b 3252
Attaching to process ID 3252, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

Then to avoid error run following
:echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Then run again above command
:jmap -heap:format=b 3252
Attaching to process ID 3252, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.0-b16

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1073741824 (1024.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 268435456 (256.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 263847936 (251.625MB)
   used     = 226730648 (216.2271957397461MB)
   free     = 37117288 (35.397804260253906MB)
   85.93231822742021% used
From Space:
   capacity = 36372480 (34.6875MB)
   used     = 16924568 (16.140525817871094MB)
   free     = 19447912 (18.546974182128906MB)
   46.53124560106982% used
To Space:
   capacity = 40763392 (38.875MB)
   used     = 0 (0.0MB)
   free     = 40763392 (38.875MB)
   0.0% used
PS Old Generation
   capacity = 333053952 (317.625MB)
   used     = 179019752 (170.7265396118164MB)
   free     = 154034200 (146.8984603881836MB)
   53.75097665858053% used
PS Perm Generation
   capacity = 151519232 (144.5MB)
   used     = 93595192 (89.25933074951172MB)
   free     = 57924040 (55.24066925048828MB)
   61.771163148451016% used

You will see correct output

WSO2 Carbon 4.2.0 based products general fine tuning parameters.

Application server and API store and publisher

 

catalina-server.xml

maxThreads="750"

minSpareThreads="150"

disableUploadTimeout="false"

enableLookups="false"

connectionUploadTimeout="120000"

maxKeepAliveRequests="600"

acceptCount="600"

API Key Manager and Identity server

catalina-server.xml

maxThreads="750"

minSpareThreads="150"

disableUploadTimeout="false"

enableLookups="false"

connectionUploadTimeout="120000"

maxKeepAliveRequests="600"

acceptCount="600"

api-manager.xml

<EnableGatewayKeyCache>true</EnableGatewayKeyCache>

API gateway if nhhtp transport enabled.

/repository/conf/nhttp.properties

http.socket.timeout=60000

snd_t_core=200

snd_t_max=250

snd_io_threads=16

lst_t_core=200

lst_t_max=250

lst_io_threads=16

api-manager.xml

<EnableGatewayKeyCache>true</EnableGatewayKeyCache>

STORE/PUBLISHER

Axis2Client.xml

<parameter name="defaultMaxConnPerHost">1000</parameter>
<parameter name="maxTotalConnections">30000</parameter>

Elastic Load Balancer

passthru-http.properties

http.socket.timeout=120000

worker_pool_size_core=400

worker_pool_size_max=500

io_buffer_size=16384

All servers

masterdatasource.properties
<maxActive>250</maxActive>
<testOnBorrow>false</testOnBorrow>
<validationInterval>120000</validationInterval>

wso2server.sh

-Xms2048m -Xmx2048m -XX:MaxPermSize=1024m

Open Files Limit

Set open files limit to 200000 by editing

/etc/sysctl.conf

sudo sysctl -p

Test gzip encoded response in WSO2 ESB and App server

For this example i used WSO2 ESB 4.8.0 and AS 5.2.1. Here we will deploy echo service in application service and create API in ESB and invoke it.

01. Enable gzip encoding in Application server
      Go to /wso2as-5.2.1/repository/conf/tomcat directory and edit catalina-server.xml file to enable gzip encoding as follows. Here you will see compression is on and compression minimum size is set as 112 then any response larger than that we encode as gzip. Also we need to specify content types list.

compression="on"
compressionMinSize="112"
compressableMimeType="text/xml;charset=UTF-8,text/plain,text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"

Then deploy echo service in Application Server.


02. Enable wire logs for ESB
add or uncomment following line in log4j.properties file available in /repository/conf directory. 
   log4j.logger.org.apache.synapse.transport.passthru.wire=DEBUG 


Create proxy by pointing to echo service. Then invoke it using some client. You will see following output in ESB console.

[2014-01-08 11:53:16,908] DEBUG - wire >> "POST /addcredits HTTP/1.1[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Host: 10.100.1.65:8280[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Accept-Language: null[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Accept-Encoding: gzip, deflate[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Content-Length: 303[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
[2014-01-08 11:53:16,909] DEBUG - wire >> "Cookie: menuPanel=visible; menuPanelType=main; region2_humantask_menu=visible; region2_bpel_instances_menu=visible; region3_registry_menu=none; requestedURI="../../carbon/configadmin/index.jsp?status=updated&rand=1389152702315"; Modernizr=; current-breadcrumb=manage_menu%2Cservices_menu%2Cservices_list_menu%23configadmin+index.jsp*[\r][\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "Connection: keep-alive[\r][\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "Pragma: no-cache[\r][\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "Cache-Control: no-cache[\r][\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "[\r][\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "[\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "   [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "   [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "      [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "         [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "         444444[\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "      [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> "   [\n]"
[2014-01-08 11:53:16,910] DEBUG - wire >> ""
[2014-01-08 11:53:16,917] DEBUG - wire << "POST /services/echo HTTP/1.1[\r][\n]"
[2014-01-08 11:53:16,923] DEBUG - wire << "Accept-Language: null[\r][\n]"
[2014-01-08 11:53:16,923] DEBUG - wire << "Cookie: menuPanel=visible; menuPanelType=main; region2_humantask_menu=visible; region2_bpel_instances_menu=visible; region3_registry_menu=none; requestedURI="../../carbon/configadmin/index.jsp?status=updated&rand=1389152702315"; Modernizr=; current-breadcrumb=manage_menu%2Cservices_menu%2Cservices_list_menu%23configadmin+index.jsp*[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Accept-Encoding: gzip, deflate[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Pragma: no-cache[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Cache-Control: no-cache[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Transfer-Encoding: chunked[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Host: 10.100.1.65:9764[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "Connection: Keep-Alive[\r][\n]"
[2014-01-08 11:53:16,924] DEBUG - wire << "[\r][\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "12f[\r][\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "[\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "   [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "   [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "      [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "         [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "         444444[\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "      [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "   [\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "[\r][\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "0[\r][\n]"
[2014-01-08 11:53:16,925] DEBUG - wire << "[\r][\n]"
[2014-01-08 11:53:16,928] DEBUG - wire >> "HTTP/1.1 200 OK[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Content-Type: text/xml;charset=UTF-8[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Transfer-Encoding: chunked[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Content-Encoding: gzip[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Vary: Accept-Encoding[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Date: Wed, 08 Jan 2014 06:23:16 GMT[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "Server: WSO2 Carbon Server[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "a[\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "[0x1f][0xffffff8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0][\r][\n]"
[2014-01-08 11:53:16,929] DEBUG - wire >> "aa[\r][\n]"
[2014-01-08 11:53:16,930] DEBUG - wire >> "[0xffffff80][0x0][0x0][0x0][0xffffffff][0xffffffff]t[0xffffff81][0xffffff9b][0x17][0x1][0x0][0x0][\r][\n]"
[2014-01-08 11:53:16,930] DEBUG - wire >> "0[\r][\n]"
[2014-01-08 11:53:16,930] DEBUG - wire >> "[\r][\n]"
[2014-01-08 11:53:16,933]  WARN - SequenceMediator onError handler : ON_ERROR for sequence : OUT_SEQ_ADD_CREDITS cannot be found
[2014-01-08 11:53:16,941] DEBUG - wire << "HTTP/1.1 200 OK[\r][\n]"
[2014-01-08 11:53:16,943] DEBUG - wire << "Vary: Accept-Encoding[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Content-Encoding: gzip[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Content-Type: application/json[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Server: WSO2 Carbon Server[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Date: Wed, 08 Jan 2014 06:23:16 GMT[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Transfer-Encoding: chunked[\r][\n]"
[2014-01-08 11:53:16,944] DEBUG - wire << "Connection: keep-alive[\r][\n]"
[2014-01-08 11:53:16,945] DEBUG - wire << "[\r][\n]"
[2014-01-08 11:53:16,945] DEBUG - wire << "58[\r][\n]"