Thursday 9 May 2013

Performance series - Announcing SuperBenchmarker sb.exe for generating load on your API/Site

[Level T1] Some of you might be doing performance analysis for living, and if so, you have access to expensive performance tools. But for those who are not and get drawn into investigating performance of their own application or their peers, there is really not a great free tool that you can just point to a URL and hammer.

Well, actually there is: Apache benchmark or ab.exe which has served the community since almost 20 years ago. It is free and very performant, i.e. does not require a lot of resources to run so does not affect the performance even if you benchmark your localhost server. But it has limitations (only GET, no HTTPS, no parameterisation) and I could not find a tool so I ended up writing one for my own use. And now I am sharing it.

SuperBenchmarker or sb.exe is a single-exe (ilmerged with dependencies inside) application that can be used to call various endpoints. It can do GET as well as other verbs, can do request templating and you can provide values for placeholders in URL or your template. It also can output a lot of tracing info in case you need it.

It requires .NET 4.5 but you can target any site or API written in Ruby, PHP, Java, ect. You can download it from here if you do not want to use NuGet. [Update: It has been reported that some people have had issue running on x64 windows. If so please comment on this GitHub issue]

So version 0.1 is out and it works although project needs some love with tests and a few features. There are also some shortcuts taken which needs to be sorted out - but it works.You can clone the code and build it (running build.ps1) or simply use NuGet to download it. [Update: preferred method is using Chocolatey - see below]

PM> Install-Package SuperBenchmarker

For usage, see the wiki on the GitHub. I value feedbacks, also if interested to get involved with the project let me know.


Courtesy of Mike Chaliy, this has now a chocolatey package. So if you have chocolatey installed (and if not here are the instructions), simply run this command in command line:

cinst SuperBenchmarker


sb.exe -u url [-c concurrency] [-n numberOfRequests] [-m method] [-t template] [-p plugin] [-f file] [-d] [-v] [-k] [-x] [-q] [-h] [-?]
-u Required. Target URL to call. Can include placeholders.
-c Optional. Number of concurrent requests (default=1)
-n Optional. Total number of requests (default=100)
-m Optional. HTTP Method to use (default=GET)
-p Optional. Name of the plugin (DLL) to replace placeholders. Should contain one class which implements IValueProvider. Must reside in the same folder.
-f Optional. Path to CSV file providing replacement values for the test
-d Optional. Runs a single dry run request to make sure all is good (boolean switch)
-v Optional. Provides verbose tracing information (boolean switch)
-k Optional. Outputs cookies (boolean switch)
-x Optional. Whether to use default browser proxy. Useful for seeing request/response in Fiddler. (boolean switch)
-q Optional. In a dry-run (debug) mode shows only the request. (boolean switch)
-h Optional. Displays headers for request and response. (boolean switch)
-? Optional. Displays this help. (boolean switch)


-u -n 1000 -c 10
-u -n 1000 -c 10 -d (runs only once)
-u http://localhost/api/myApi/ -t template text (file contains headers to be sent for GET. format is same as HTTP request)
-u http://localhost/api/myApi/ -m POST -t template.txt (file contains headers to be sent for POST. format is same as HTTP request with double CRLF separating headers and payload)
-u http://localhost/api/myApi/{{{ID}}} -f values.txt (values file is CSV and has a column for ID)-u http://localhost/api/myApi/{{{ID}}} -p myplugin.dll (has a public class implementing IValueProvider defined in this exe)
-u -h (shows headers)
-u -h -q (shows cookies)
-u -v (shows some verbose information including URL to target - especially useful if parameterised)