Sanjeewa Malalgoda's Blog

How To install HAProxy Load Balancer on redhat

In my previous post i have described how to configure HAProxy load balancer on ubuntu. Here in this post i will describe how to do it on red-hat. Please make sure you are doing all this with necessary permissions for the your account.

01. Install HAProxy using following commands

wget http://haproxy.1wt.eu/download/1.2/src/haproxy-1.2.17.tar.gz
tar -zxf haproxy-1.2.17.tar.gz
cd haproxy-1.2.17
make
cp haproxy /usr/sbin/haproxy

02.Copy and edit config files

wget http://layer1.rack911.com/haproxy/haproxy-standard.cfg -O /etc/haproxy.cfg
wget http://layer1.rack911.com/haproxy/haproxy.init -O /etc/init.d/haproxy
chmod +x /etc/init.d/haproxy

03.Starting server

sudo /etc/init.d/haproxy start

04.restarting server after any changes

sudo /etc/init.d/haproxy restart

Use the same config file mentioned in earlier post of mine. After installation everything is

same as ubuntu.

Setup HAProxy Load Balancer on Ubuntu

01. installing haproxy on ubuntu machine

sudo apt-get install haproxy

02. Edit configurations file

sudo vi /etc/haproxy/haproxy.cfg

Add following text to file and save it

global
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout
5000
clitimeout
50000
srvtimeout
50000
listen webcluster *:80
mode http
stats enable
stats auth us3r:passw0rd
balance roundrobin
option httpchk HEAD / HTTP/1.0
option forwardfor
cookie LSW_WEB insert
option httpclose
server web01 10.100.2.85:8280 cookie LSW_WEB01 check
server web02 10.100.2.85:8281 cookie LSW_WEB02 check

03. Setting startup parameter for HAProxy set enabled as 1 to start HAproxy
sudo vi /etc/default/haproxy

# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1

04. Start HAProxy form command line

sudo /etc/init.d/haproxy start

05. Restart HAProxy after any configuration change form command line

sudo /etc/init.d/haproxy restart

06. Start 2 ESB servers (Set HTTP port as 8280 and 8281) and with default configurations

07. Access Web user interface from browser
Type following url in browser and provide stat credentials you provided in configurations file(haproxy.cfg)

http://10.100.2.85:80/haproxy?stats

You will see following User interface.

08. Access ESB from browser or send requests to following url
http://10.100.2.85:80/services/echo?wsdl

Following Diagram shows you how this laod balancer works.

Tenant Aware Load balancer

Tenant Aware Load Balancer is the upcoming Load Balancer from WSO2. Here in this article i will briefly describe it's architecture and how it works.

Introduction

LoadBalancer
Load balancing is a key concern in a Platform-as-a-Service (PaaS) or a middleware platform deployed on the Cloud.
The primary functionality of a load balancer is to distribute the load of the incoming traffic amongst a set of back end worker nodes. This set of worker nodes can be either statically configured or can be dynamically discovered. In WSO2 carbon based products we use cluster messages which based on axis2 clustering to identify the node join to cluster or leave cluster.

Multi-tenancy - Tenant-aware Load Balancing
If a Cloud deployment has to be able to scale to thousands, hundreds of thousands or millions of tenants, we need tenant-partitioning. This means, for a single Cloud service, there will be multiple clusters, and each of these Service clusters will handle a subset of the tenants in the system. Creating dynamic tenant clusters & tenant partitioning strategies are some of the ongoing research areas. In such a tenant-partitioned deployment, the load balancers themselves need to be tenant-aware, in order to be able to route the requests to the proper tenant clusters. This means the load balancer has to be tenant-aware as well as Service-aware, since it is the Service clusters that are partitioned according to the tenants. WSO2 Load Balancer is now capable of routing messages in both tenant aware and service aware way.

How WSO2 Tenant Aware Load Balancer Works
Load Balancer's task is to distribute the load of the incoming traffic amongst a set of back end worker nodes.
We will see how this works now.
when the request comes in to Load Balancer we have to identify host header and determine cluster domain
Eg: Host header = https://appserver.cloud-test.wso2.com/t/ttt.ttt/carbon/admin/login.jsp
Service domain is appserver domain
Then determine tenant Id from the url. Here tenant domain is ttt.ttt so we can determine tenant domain.
These parameters can extract from host header.
Now we have both service domain and tenant domain. so with both of them we can direct this request to correct cluster as we defined in loadbalancer.conf. There we can configure services with their cluster domains and tenant ranges. When load balancer loads it picks those ranges and cluster domains.

See the following sample configuration . You can find those configuration under the “repository/conf” folder. There you will find loadbalancer.conf file.

appserver {
# multiple hosts should be separated by a comma.
hosts appserver.cloud-test.wso2.com;
domains {
wso2.as1.domain {
tenant_range 1-100;
}
wso2.as.domain {
tenant_range *;
}
}
}

Let’s examine this configuration.
Above configuration is for Application service with two service clusters. Let’s examine it line by line.
First line says appserver is the main node of this configurations.
In next line you will find hosts entries. There we have to add host address that maps to application server service. If you need to add multiple service domains add it with separated by commas.
Next you will find domains there you will find two domains with pre defined tenant ranges.
wso2.as1.domain named cluster is responsible for load tenants with tenant id 1 to 100
wso2.as.domain named cluster is responsible for load tenants with all tenants except the tenants belong to above range.
Then it can forward request to correct cluster based on service domain and tenant domain.

How Load Balance End Point works
For load balancing we are using synapse end point as a end point to determine how message should forward. So here we have defined external end point which is named as Tenant Aware Load Balance End point. It has logic to forward messages to correct back end node. We can define loadbalance end point in synapse main sequence.

In order to do this you have to change main sequence with correct load balance end point. If you need to implement your own custom load balancing implementation you can write your own end point and configure load balancer to use it.
As an example we can load balance based on any other parameter which is available in request header (something similar to tenant domain). It's simple and straightforward.

See the following diagram to get general idea on how it works.

Advantages and other features of WSO2 Tenant Aware Load Balancer

Tenants are loaded in demand to the pre defined cluster based on the configurations in loadbalancer.conf.
Unload unused tenants after paticular predefined time.
Tenant Partitioning
Tenant may have multiple service clusters
Tenants will define the partitions.
When a new instance added, notify the LB instances.
Auto scaling - starting and terminating of instances based on the load. This supports any type of infrastructure as a service. What we have to do is write simple adapter to auto-scaler service based on the service vendor's API. It’s straight forward and simple. External components Autoscaler service can add, assign nodes into LB. So load Balancer is independent from the infrastructure that we are using.
Group communication between clusters using cluster messages.
Notifying all the load balancers when a new node is added to cluster.
Data partitioning(tenants data) - we can use this design to data partitioning based on the Geography or any other factors. As an example if we decide tenant 1 to 100 should loaded and store their data on United state we can deploy new application cluster at united states and inform it to loadbalancer. So we have to add configuration to loadbalancer.conf file saying tenant 1 to 100 are loaded at particular cluster(named as wso2.as.usa.domain). Then all requests come to the tenant 1 to 100 will forward to that cluster. so all data related to those tenants (Application files, databases) will stored at united state servers.
Private jet mode - Also we can assign some particular application cluster to specific tenant. Its like private jet. If you are willing to pay for entire cluster we can assign entire cluster for your tenant. This is again we can do using configurations at loadbalancer.conf file. So all request comes to that tenant will forward to that particular cluster.

Always running an additional cluster(Default cluster) to quickly load new tenant. This cluster is responsible for all newly created tenants in the system and super admin tenant. We can define this cluster in load balancer configuration file with * mark as upper tenant ID.

After next release of WSO2 Load Balancer you will be able to use all above mentioned functionalities. Please visit http://wso2.org/projects/load-balancer

How to avoid loading error in soapui on ubuntu 11.04 (ERROR [SoapUI] An error occured [no protocol: POECommonEPrescribeService-wsdl-soapui-project.xml])

When you start soapui in ubuntu 11.04 you may see following error.

sanjeewa@sanjeewa-TECRA-M11:/media/9558-83FD/software/soapui-4.0.0/bin$ sh soapui.sh
================================
=
= SOAPUI_HOME = /media/9558-83FD/software/soapui-4.0.0
=
================================
Configuring log4j from [/media/9558-83FD/software/soapui-4.0.0/bin/soapui-log4j.xml]
10:55:27,301 INFO [DefaultSoapUICore] initialized soapui-settings from [/media/9558-83FD/software/soapui-4.0.0/soapui-settings.xml]
10:55:29,001 INFO [WorkspaceImpl] Loading workspace from [/media/9558-83FD/software/soapui-4.0.0/bin/../../TestPBE-workspace.xml]
10:55:29,285 INFO [SoapUI] File [/media/9558-83FD/software/soapui-4.0.0/bin/POECommonEPrescribeService-wsdl-soapui-project.xml] does not exist, trying URL instead
10:55:29,285 ERROR [SoapUI] An error occured [no protocol: POECommonEPrescribeService-wsdl-soapui-project.xml], see error log for details
10:55:30,144 INFO [SoapUI] Used java version: 1.6.0_21
Exception in thread "XpcMessageLoop" java.lang.RuntimeException: java.io.FileNotFoundException: Library libgthread-2.0 not found at any known locations !
    at com.jniwrapper.linux.utils.LinuxLibraryLoader.(LinuxLibraryLoader.java:90)
    at com.jniwrapper.gdk.GdkMultithreading.(GdkMultithreading.java:48)
    at com.jniwrapper.gdk.GdkMultithreading.getInstance(GdkMultithreading.java:35)
    at com.jniwrapper.gtk.GTK.initialize(GTK.java:97)
    at com.teamdev.xpcom.a.b.a.a.(Unknown Source)
    at com.teamdev.xpcom.XpcMessageLoop.(Unknown Source)
    at com.teamdev.xpcom.XpcMessageLoop.getInstance(Unknown Source)
    at com.teamdev.xpcom.b$a.run(Unknown Source)
Caused by: java.io.FileNotFoundException: Library libgthread-2.0 not found at any known locations !
    ... 8 more

You can fix this issue by disabling browser component. To do that open soapui.sh file locates inside bin folder of the soapui distribution and edit it. you have to uncomment following line.

JAVA_OPTS="$JAVA_OPTS -Dsoapui.jxbrowser.disable=true"

Then save it and start soapui again.

Tenant Aware Load Balancer Concept

Tenant Aware Load Balancer is the upcoming Load Balancer from WSO2. If a Cloud deployment has to be able to scale to thousands, hundreds of thousands or millions of tenants, we need tenant-partitioning. This means, for a single Cloud service, there will be multiple clusters, and each of these Service clusters will handle a subset of the tenants in the system.Here i will briefly describe about it. I will add complete article on this later.

Tenants are loaded in demand and assign to a cluster
Unloading the unused tenants.
Notifying the load balancers when a tenant is loaded.
by adding the Loadbalancers in to a group communication group and publishing a message to the group when a tenant is loaded.
Can define statically tenants to each clusters
Single service may have multiple clusters and each cluster is assigned in tp particular tenant range
Always running an additional cluster(Default cluster) to quickly load new tenants.

Following diagram describes the overall design of tenant aware load balancer

Carbon Context and it's usages in WSO2 Products

A context can be defined as a construct that strictly bounds the execution environment. Because Carbon Context is a runtime container for your app(s) you gain the benefit of leveraging the Carbon Context runtime API allows to obtain contextual information about various actors utilizing your web apps and web services. For example, you may want to use registry,cache for that particular user or tenant. The carbon context allows you to access 5 APIs and has several utility methods. In each time we create user specific thing we create carbon context for that user to store specific things. This is valid for all WSO2 Carbon 3.2.2 based servers.

Here is the link to post written by regarding carbon context and its usages.
http://wso2.org/library/articles/2012/03/carbon-context-its-usages-wso2-products

How to trace inside the clone mediator and enable statistics and tracing for dynamic sequences

Trace inside the clone mediator and enable statistics and tracing for dynamic sequences in WSO2 ESB

Here in this article i will try to brefly describe how to use WSO2 ESB samples and how to trace inside clone mediator.

You can download WSO2 ESB from here
http://wso2.com/products/enterprise-service-bus/

Starting the WSO2 ESB
Goto /bin folder run following command for ubuntu
> sh wso2server.sh
Run with sample here 0 is sample number
>sh wso2esb-samples.sh -sn 0

Log in to ESB as admin and
Goto Home> Configure > Logging > Service Bus Configuration by Clicking on source view. Then you will see synapse configuration source view.
Add following sequence to esb Configuration.

<sequence name="main" trace="enable" statistics="enable">
        <in>
            <clone>
                <target>
                    <sequence>
                        <log/>
                        <property name="a" value="a" scope="default"/>
                        <drop/>
                    </sequence>
                </target>
                <target>
                    <sequence>
                        <log/>
                        <property name="b" value="B" scope="default"/>
                        <drop/>
                    </sequence>
                </target>
            </clone>
        </in>
        <out>
            <send/>
        </out>
    </sequence>

As you can see here this will enable logs inside clone mediator. This will clone message and then add property a and b then drop the message.

Starting the Sample Apache Axis2 Server
Here we are using a standalone Apache Axis2 Web services engine(bundled with your ESB distribution) as the server

user@host:~/wso2esb/samples/axis2Server/src/SimpleStockQuoteService$ ant

now We have to start axis2server. Goto directory wso2esb-3.0.1/samples/axis2Server and type following command

wso2esb-3.0.1/samples/axis2Server$ sh axis2server.sh

To run sample client and send request to server goto folder wso2esb-3.0.1/samples/axis2Client and type following command

ant stockquote -Daddurl=http://localhost:9000/services/SimpleStockQuoteService -Dtrpurl=http://localhost:8280/

When you goto logs you will find logs related to 2 sequences inside clone mediator.See following logs
2012-02-29 14:43:12,029 [-] [HttpServerWorker-17] DEBUG Axis2SynapseEnvironment Injecting MessageContext for asynchronous mediation using the : Anonymous Sequence
2012-02-29 14:43:12,029 [-] [SynapseWorker-9] DEBUG PropertyMediator Setting property : a at scope : default to : a (i.e. constant : a)
2012-02-29 14:43:12,029 [-] [SynapseWorker-9] DEBUG PropertyMediator End : Property mediator
2012-02-29 14:43:12,030 [-] [SynapseWorker-10] DEBUG SequenceMediator Start : Sequence
2012-02-29 14:43:12,030 [-] [SynapseWorker-9] DEBUG DropMediator Start : Drop mediator
2012-02-29 14:43:12,030 [-] [SynapseWorker-9] DEBUG DropMediator End : Drop mediator
2012-02-29 14:43:12,030 [-] [HttpServerWorker-17] DEBUG CloneMediator End : Clone mediator
2012-02-29 14:43:12,030 [-] [SynapseWorker-9] DEBUG SequenceMediator End : Sequence
2012-02-29 14:43:12,030 [-] [SynapseWorker-10] DEBUG SequenceMediator Sequence :: mediate()