Cory Taylor bio photo

Cory Taylor

Email Twitter Google+ LinkedIn Instagram Github Stackoverflow

Today I’d like to show a little plugin I wrote to enable throttling on any of my api endpoints. ServiceStack is my framework of choice and we use it at Kobo to serve up our client APIs. If you aren’t already familiar with ServiceStack, head on over and check it out1. I couldn’t be happier with ServiceStack, it’s an incredibly fast and flexible framework and currently meets all of our needs. So let’s jump right in.

So Why Throttle Clients?

There may be many business-driven reasons why one would want to add throttling to an api. In a lot of cases, throttling is intended to prevent abuse of an api whether that abuse is intentional or not, it’s always a good idea to ensure your apis are protected and meet your defined SLAs. Other cases might be to control costs, to limit an attack vector (e.g., an attacker brute forcing an authentication endpoint) and so on.

What can be Throttled?

ProgrammableWeb2 had a good summary of the many ways companies are limiting their apis and it’s not just request per ip over a specific time period. A few examples are:

  • Time based limits: 1 call per second
  • Call volume by IP: 5,000 queries per IP per day
  • Call volume per-application: 10,000 queries per application per day
  • Return results volume: 10 results per query
  • Data transmission volume: 120 packets of 1.6KB per minute

So with that, our goal for this post is to easily enable specific ServiceStack endpoints to monitor and throttle requests by clients at per minute, per hour or per day intervals.

Annotating our API Endpoints

In order to let our plugin know which endpoints we want to throttle, we need a way to specifically mark each ServiceStack Operation. We’ll do that with the ThrottleInfoAttribute3 class and example usage below.

1 public class ThrottleInfoAttribute : Attribute
2 {
3     public int PerMinute { get; set; }
4     public int PerHour { get; set; }
5     public int PerDay { get; set; }
6 }
1 [ThrottleInfo(PerMinute = 10, PerHour = 0, PerDay = 0)]
2 [Route("/perminute")]
3 public class GetPerMinuteThrottle : IReturn<string>
4 {
5 
6 }

The ServiceStack Plugin

Don’t worry if you aren’t familiar with writing a plugin for ServiceStack, their wiki4 has a pretty thorough walkthrough and you can also view the full code for the plugin5.

The Register function is called when ServiceStack loads all plugins, at which point we want our plugin to lookup all of the Operations with a ThrottleInfoAttribute so that we can cache the results and we don’t have to do it on each request.

1 public void Register(IAppHost appHost)
2 {
3     RegisterThrottleInfoForAllOperations(appHost);
4 
5     _redisClient.RemoveAllLuaScripts();
6     _scriptSha = _redisClient.LoadLuaScript(ReadLuaScriptResource("rate_limit.lua"));
7 
8     appHost.GlobalRequestFilters.Add(CheckIfRequestShouldBeThrottled);
9 }
 1 public void RegisterThrottleInfoForAllOperations(IAppHost appHost)
 2 {
 3     foreach (var operation in appHost.Metadata.Operations)
 4     {
 5         var throttleAttribute = operation.RequestType.GetCustomAttributes(typeof(ThrottleInfoAttribute)).First() as ThrottleInfoAttribute;
 6         if (throttleAttribute == null)
 7             continue;
 8 
 9         _throttleInfoMap.Add(operation.RequestType, throttleAttribute);
10     }
11 }

We also want to optimize our calls to Redis so we should pre-load our Lua script into Redis so as to avoid sending this on each request.

1 _redisClient.RemoveAllLuaScripts();
2 _scriptSha = _redisClient.LoadLuaScript(ReadLuaScriptResource("rate_limit.lua"));

Next is our function CheckIfRequestShouldBeThrottled which is called on every request. We can do a quick lookup to see if our request type has throttling info which, if it does, build up a key of RemoteIp and OperationName. This will be used to create buckets based on the time interval configured for this Operation. We’ll touch more on this shortly. Then we actually need to call into redis, using the sha we received earlier to the cached lua script, the key and the throttling information for the given Operation being called. Our script returns 1 if we should throttle the request or null if we shouldn’t, so check the result, and if we are supposed to throttle this request, set the status to 429 with a description that the client has been throttled.

 1 private void CheckIfRequestShouldBeThrottled(IRequest request, IResponse response, object requestDto)
 2 {
 3     if (!_throttleInfoMap.ContainsKey(requestDto.GetType()))
 4         return;
 5 
 6     var throttleInfo = _throttleInfoMap[requestDto.GetType()];
 7     var key = string.Format("{0}:{1}", request.RemoteIp, request.OperationName);
 8 
 9     try
10     {
11         var result = _redisClient.ExecLuaShaAsString(_scriptSha, new[] {key}, new[]
12         {
13             throttleInfo.PerMinute.ToString(), 
14             throttleInfo.PerHour.ToString(), 
15             throttleInfo.PerDay.ToString(),
16             SecondsFromUnixTime().ToString()
17         });
18         if (result != null)
19         {
20             response.StatusCode = 429;
21             response.StatusDescription = "Too many Requests. Back-off and try again later.";
22             response.Close();
23         }
24     }
25     catch ()
26     {
27         //got an error calling redis so log something and let the request through
28     }
29 }

Determining if we should throttle

The benefit of using a script directly on the redis server is that we can reduce the number of roundtrips we need to make. Everything can be done in one call to redis. Above you’ll note we are passing into the script the key of ip:operation followed by four arguments (3 limits and a timestamp). The last argument is the timestamp, which we remove from the table and assign to ts. Next we loop over the lesser of the number of arguments or the number of durations. We build up our bucket key which we call INCR on and set it’s expiry to the duration window of 60sec, 3600sec or 86400sec. This ensures the bucket will expire at the end of this period or we’ll keep incrementing the counter until we hit the limit.

 1 local durations = {60, 3600, 86400}
 2 local prefix    = {'m', 'h', 'd'}
 3 local ts        = tonumber(table.remove(ARGV))
 4 local request   = KEYS[1]
 5 
 6 --iterate over all of the limits provided
 7 for i = 1, math.min(#ARGV, #durations) do
 8 	local limit = tonumber(ARGV[i])
 9 
10 	--only check limits that have been set
11     if limit > 0 then
12 	    local bucket = ':' .. prefix[i] .. ':' .. durations[i] .. ':' .. math.floor(ts / durations[i])
13 	    local key = request .. bucket
14 
15 	    local total = tonumber(redis.call('INCR', key) or '0')
16 	    redis.call('EXPIRE', key, durations[i])
17 	    if total > limit then
18 	        return true
19 	    end
20     end
21 end
22 return false

The main goal of this post was to show how we could write a plugin for ServiceStack to enable throttling per operation so I’ve kept the implementation simple enough to follow even if you aren’t familiar with lua. You can check out the post on stack exchange6 which lead me to explore using lua in the first place or read antirez blog7 for some background on redis and lua scripting.