New API Manager Throttling implementation for API Manager 2.0.0

As you know at the moment we are working on completely new throttling implementation for API Manager next release. So in this article i will briefly summarize what we are going to do with next release. Please note that these facts are depend on the discussions happen at architecture@wso2.com and developers mailing lists. Also these content may subject to change before release.

Existing API Manager Throttling DesignBased on Hazelcast IAtomicLong distributed counter.
High performance with accuracy.
Bit difficult to design complex policies.
Cannot define policies specific to given API.
Can throttle based on requests count only.

Advantages of New DesignBased on Central Policy Server.
Extensible and Flexible to define advanced rules based on API properties such as headers, users and etc.
Efficient siddhi(https://github.com/wso2/siddhi) based implementation for throttle core.
Efficient DB lookups with Bloom Filter based implementation of Siddhi.
Can design throttling policies based on request count and bandwidth both.

New architecture and message flow.

Message Flow and how it worksAPI Gateway will be associated with new throttle handler.
Throttle handler will extract all the relevant properties from message context and generate throttle keys.
To check API level throttling API level key would be context:version combination.
For resources context:version:resource_path:http_method.
Throttle handler will do map lookup for throttled events when new request comes to API gateway.
Then once throttling process completed handler will set message context to agent.
Throttle data process and publisher agent will asynchronously process message to push events to Central Policy Server.
Central Policy Server will evaluate complex rules based on events and update topic accordingly.
All gateway workers will fetch throttled events from database time to time in asynchronous manner.
Two layers of cache will be used to store throttling decisions.
Local decisions will be based map lookup.

So far we have identified few throttle/ rate limit conditions.
Number of requests per unit time(what we have now). This can be associated with the tier.
Data amount transferred through gateway per unit time. This also can be associated with the tier.
Dynamic rules(such as blocking some IP or API). This should be applied globally.
Rate limiting(this should be applied in node level as replicating counters will cause performance issues). Ex: requests on fly at given time is 500 for API.

Content Based Throttling

We have identified some parameters available in message context which we can use as throttling parameters.
We may use one or more of them to design policy.

• IP Address.
• IP Address Range.
• Query Parameters.
• Transport Headers.
• Http Verb.
• Resource path.

Policy Design Scenarios

You may design new policies based on request count per unit time interval or bandwidth for given time period. 

Policies can be designed per API. Therefore this will facilitate  the current resource level throttling implementation.  

Example: For API named “testAPI” resource “people” HTTP GET allows 5 requests per minute and POST allows 3 requests per minute. 

If our API support only mobile devices then we can add policy in API level to check user agent and throttle.

System administrators can define set of policies which apply across all APIs.

Example: If user bob is identified as fraud user then admin can set policy saying block bob.

Same Way we can block given IP address, user agent, token etc. 

Policies can be applied at multiple levels such as:

• API Level
• Application Level
• Global Level(custom policy and blocking conditions)
• Subscription Level

We can create new policy using admin dashboard user interface.

Then it will create policy file and send it to central policy server.

Central policy server will deploy it.

Here i have attached some of the images in admin dashboard related to throttling policy design.

How to create API/Resource level policy with multiple conditions

How to block certain requests based on API, Application, IP address and User.

How to add and use custom policy to handle custom throttling scenarios based on requirements.

Key Advantages

• Ability to design complex throttle policies.
• Advanced policy designer user interface.
• Users can design policies with multiple attributes present in request.
• Ex: transport headers, body content, HTTP verb etc. 
• Can design tier by combining multiple policies
• Ex: For given IP range, given HTTP verb, given header limit access.
• If client is mobile device throttle them based on user agent header.
• Can design API specific policies.