SuperBenchmarker v0.4 released

Level [T2]

This is a quick shoutout on the release of version 0.4 of SuperBenchmarker, a Web and/or Web API performance benchmarking command line tool for Windows.

You might have heard about and used Apache Benchmark (ab.exe) in the past which is a very useful tool but on Windows it is very limited (e.g cannot make POST, PUT, etc requests and only supports GET). SuperBenchmarker (sb.exe) supports PUT, DELETE, POST or any arbitrary method and allows you to parameterise the URL and headers using a data file, a .NET DLL plugin and the new feature is the randomisation feature which removes the need for any setup when all needed is random data.

Getting started

The best way to get SuperBenchmarker is to use awesome Chocolatey which is Windows' equivalent of apt-get tool on Linux.

To get Chocolatey, just run this command in your Powershell console (in Administrative mode):

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))

And then install SuperBenchmarker in the command line shell:

c:\> cinst SuperBenchmarker

And now you are ready to load test:

c:\> sb -u http://google.com

Note: if you are using Visual Studio's command line shell, you cannot use ampersand character (&) and you have to escape it using hat (^).

Using SuperBenchmarker

Normally you would define total number of requests and concurrency:

c:\> sb -u http://google.com -c 10 -n 2000

Statement above runs 2000 requests with concurrency of 10. At the end, you are shown important metrics of the test:

Status 503:    1768
Status 200:    232

TPS: 98 (requests/second)
Max: 11271.1890515802ms
Min: 3.15724613377097ms
Avg: 497.181240820346ms

50% below 34.0499543844287ms
60% below 41.8178295863705ms
70% below 48.7612961478952ms
80% below 87.4385213898198ms
90% below 490.947293319644ms

So the breakdown of the statuses returned, TPS (transaction per second), minimum, maximum and average of the time taken. But more importantly, your percentiles that really should be driving your performance SLAs (90% or 99%). [Never use the average for anything].

In case you need to dig deeper, a log file gets created in the current directory with the name run.log which you can change using -l parameter:

c:\> sb -u http://google.com -c 10 -n 2000 -l c:\temp\mylog.txt

log file is a tab separated file which contains these columns: order number (based on the time started not the time ended), status code, time taken in ms and then any custom parameters that you might have had - see below.

Sometimes when running a test for the first time, something might not have been quite right in which case you can make a dry run/debug using -d parameter that makes a single request and the body of the response will be shown at the end. If you want to see the headers as well, use -h parameter.

c:\> sb -u http://google.com -c 10 -n 2000 -d -h

Supplying request headers or a payload for POST, PUT and DELETE

In order to pass your tailored request headers, a template file needs to be defined which is basically the HTTP request template (minus the first line defining verb and URL and version):

c:\> sb -u http://google.com -t template.txt

And the template.txt contains our custom headers (from the second line of the HTTP request):

User-Agent: SuperBenchmarker
MyCustomHeader: foo-bar;baz=biz

Please note that you don't have to provide headers such as Host and Content-Length - in fact it will raise errors. These headers will be automatically added by the underlying framework.

For using POST, PUT and DELETE we need to supply the verb parameter:

c:\> sb -u http://google.com -v POST

But this request would require a payload as well which we need to supply. We use the template file to supply HTTP payload as well as any headers. Similar to an HTTP request, there must be an empty line between headers and body:

User-Agent: WhateverValueIWant
Content-Type: x-www-formurlencoded

name=value&age=25

Parameterising your requests

Basically you can parameterise your requests using a CSV file containing values, your plugin DLL or by specifying randomisation.

You would define parameters in URL and headers (payload not yet supported but coming soon in 0.5) using SuperBenchmarker's syntax:

{{{MyParameter}}}

As you can see, we use three curly brackets to denote a parameter. For example the statement below defines a customerId parameter:

c:\> sb -u "http://myserver.com/api/customer?customerid={{{customerId}}}^&ignore=false"

Please note quoting the URL and use of ^ to escape & character - if you are using Visual Studio command prompt. To run the test successfully, you need to provide a CSV file containing customerId:

customerId
123,
245,

and use -f option to run the test:

c:\> sb -u "http://myserver.com/api/customer?customerid={{{customerId}}}&ignore=false" -f c:\mypath\values.csv

Alternatively, you can use a plugin DLL to provide values:

c:\> sb -u "http://myserver.com/api/customer?customerid={{{customerId}}}&ignore=false" -p c:\mypath\myplugin.dll

This DLL must have a single public class implementing IValueProvider interface which has a single method:

public interface IValueProvider
{
    IDictionary<string, object> GetValues(int index);
}

For every request implementation of the interface is called and the index of the request is passed to and in return a dictionary of field names with their respective values is passed back.

Now we have a new feature that in most cases alleviates the need for CSV file or plugin and that is the ability to setup random value provider in the definition of the parameter itself:

c:\> sb -u "http://myserver.com/api/customer?customerid={{{customerId:RAND_INTEGER:[1000:2000]}}}&ignore=false"

The parameter above is set up to be filled by a random integer between 1000 and 2000.
Possible value types are:

• String: using RAND_STRING. Will output random words
• Date: using RAND_DATE (accepts range)
• DateTime: using RAND_DATETIME (accepts range)
• DateTimeOffset: using RAND_DATETIMEOFFSET which outputs ISO dates (accepts range)
• Double: using RAND_DOUBLE (accepts range)
• Name: using RAND_NAME. Will output random names

Feedback

Don't forget to feedback with issues and feature requests in the GitHub page. Happy load testing!