Distributed Load Testing with Jmeter

Many of you have probably used apache Jmeter for load testing before. Still, it is easy to run into the limits imposed by running it on just one machine when trying to make sure that our API will be able to serve hundreds of thousands or even millions of users.

We can get around this issue by deploying and running our tests to multiple machines in the cloud.

In this article, we will take a look at one way to distribute and run Jmeter tests along multiple droplets on DigitalOcean using Terraform, AnsibleAnsible is an open-source software provisioning, configuration management, and application-deployment tool. It enables Infrastructure-as-Code (IaC), meaning that it can handle the state of infrastructure through idempotent changes, defined with an easily readable, domain-specific language instead of relying on Bash scripts., and a little bit of bash scripting to automate the process as much as possible.

Background: During the COVID19 outbreak induced lockdowns, we’ve been tasked by a company (who builds an e-learning platform primarily for schools) to build out an infrastructure that is:

• geo redundant,
• supports both single and multi tenant deployments ,
• can be easily scaled to serve at least 1.5 million users in huge bursts,
• and runs on-premises.

To make sure the application is able to handle these requirements, we needed to set up the infrastructure, and model a reasonably high burst in requests to get an idea about the load the application and its underlying infrastructure is able to serve.

In this article, we’ll share practical advice and some of the scripts we used to automate the load-testing process using Jmeter, Terraform and Ansible.

Let’s Start!

Have these tools installed before you begin!

brew install ansible
brew install terraform
brew install jmeter

You can just run them from your own machine. The full codebase is available on Github at RisingStack/distributed-loadtests-jmeter for your convenience.

Why do we use Jmeter for distributed load testing?

Jmeter is not my favorite tool for load testing owing mostly to the fact that scripting it is just awkward. But looking at the other tools that support distribution, it seems to be the best free one for now. K6 looks good, but right now it does not support distribution outside the paid, hosted version. Locust is another interesting one, but it’s focusing too much on random test picking, and if that’s not what I’m looking for, it is quite awkward to use as well – just not flexible enough right now.

So, back to Jmeter!

Terraform is infrastructure as code, which allows us to describe the resources we want to use in our deployment and configure the droplets so we have them ready for running some tests. This will, in turn, be deployed by Ansible to our cloud service provider of choice, DigitalOcean – though with some changes, you can make this work with any other provider, as well as your on-premise machines if you wish so.

Deploying the infrastructure

There will be two kinds of instances we’ll use:

• primary, of which we’ll have one coordinating the testing,
• and runners, that we can have any number of.

In the example, we’re going to go with two, but we’ll see that it is easy to change this when needed.

You can check the variables.tf file to see what we’ll use. You can use these to customise most aspects of the deployment to fit your needs. This file holds the vars that will be plugged into the other template files – main.tf and provider.tf.

The one variable you’ll need to provide to Terraform for the example setup to work is your DigitalOcean api token, that you can export like this from the terminal:

export TF_VAR_do_token=DO_TOKEN

Should you wish to change the number of test runner instances, you can do so by exporting this other environment variable:

export TF_VAR_instance_count=2

You will need to generate two ssh key pairs, one for the root user, and one for a non-privileged user. These will be used by Ansible, which uses ssh to deploy the testing infrastructure as it is agent-less. We will also use the non-privileged user when starting the tests for copying over files and executing commands on the primary node. The keys should be set up with correct permissions, otherwise, you’ll just get an error.

Set the permissions to 600 or 700 like this:

chmod 600 /path/to/folder/with/keys/*

To begin, we should open a terminal in the terraform folder, and call terraform init which will prepare the working directory. Thisl needs to be called again if the configuration changes.

You can use terraform plan that will output a summary of what the current changes will look like to the console to double-check if everything is right. At the first run, it will be what the deployment will look like.

Next, we call terraform apply which will actually apply the changes according to our configuration, meaning we’ll have our deployment ready when it finishes! It also generates a .tfstate file with all the information about said deployment.

If you wish to dismantle the deployment after the tests are done, you can use terraform destroy. You’ll need the .tfstate file for this to work though! Without the state file, you need to delete the created droplets by hand, and also remove the ssh key that has been added to DigitalOcean.

Running the Jmeter tests

The shell script we are going to use for running the tests is for convenience – it consists of copying the test file to our primary node, cleaning up files from previous runs, running the tests, and then fetching the results.

#!/bin/bash

set -e

# Argument parsing, with options for long and short names
for i in "$@"
do
case $i in
    -o=*|--out-file=*)
    # i#*= This removes the shortest substring ending with
    # '=' from the value of variable i - leaving us with just the
    # value of the argument (i is argument=value)
    OUTDIR="${i#*=}"
    shift
    ;;
    -f=*|--test-file=*)
    TESTFILE="${i#*=}"
    shift
    ;;
    -i=*|--identity-file=*)
    IDENTITYFILE="${i#*=}"
    shift
    ;;
    -p=*|--primary-ip=*)
    PRIMARY="${i#*=}"
    shift
    ;;
esac
done

# Check if we got all the arguments we'll need
if [ -z "$TESTFILE" ] || [ ! -f "$TESTFILE" ]; then
    echo "Please provide a test file"
    exit 1
fi

if [ -z "$OUTDIR" ]; then
    echo "Please provide a result destination directory"
    exit 1
fi

if [ -z "$IDENTITYFILE" ]; then
    echo "Please provide an identity file for ssh access"
    exit 1
fi

if [ -z "$PRIMARY" ]; then
  PRIMARY=$(terraform output primary_address)
fi

# Copy the test file to the primary node
scp -i "$IDENTITYFILE" -o IdentitiesOnly=yes -oStrictHostKeyChecking=no "$TESTFILE" "runner@$PRIMARY:/home/runner/jmeter/test.jmx"
# Remove files from previous runs if any, then run the current test
ssh -i "$IDENTITYFILE" -o IdentitiesOnly=yes -oStrictHostKeyChecking=no "runner@$PRIMARY" << "EOF"
 rm -rf /home/runner/jmeter/result
 rm -f /home/runner/jmeter/result.log
 cd jmeter/bin ; ./jmeter -n -r -t ../test.jmx -l ../result.log -e -o ../result -Djava.rmi.server.hostname=$(hostname -I | awk ' {print $1}')
EOF
# Get the results
scp -r -i "$IDENTITYFILE" -o IdentitiesOnly=yes -oStrictHostKeyChecking=no "runner@$PRIMARY":/home/runner/jmeter/result "$OUTDIR"

Running the script will require the path to the non-root ssh key. The call will look something like this:

bash run.sh -i=/path/to/non-root/ssh/key  -f=/path/to/test/file -o=/path/to/results/dir

You can also supply the IP of the primary node using -p= or --primary-ip= in case you don’t have access to the .tfstate file. Otherwise, the script will ask terraform for the IP.

Jmeter will then take care of distributing the tests across the runner nodes, and it will aggregate the data when they finish. The only thing we need to keep in mind is that the number of users we set for our test to use will not be split but will be multiplied. As an example, if you set the user count to 100, each runner node will then run the tests with 100 users.

And that’s how you can use Terraform and Ansible to run your distributed Jmeter tests on DigitalOcean!

Check this page for more on string manipulation in bash.

Looking for DevOps & Infra Experts?

In case you’re looking for expertise in infrastructure related matters, I’d recommend to read our articles and ebooks on the topic, and to check out our various service pages:

• DevOps, SRE & Cloud Consulting Services
• Infrastructure Assessment and Code Review Services
• Kubernetes Consulting & Training Services

An early draft of this article was written by Mate Boer, and then subsequently rewritten by Janos Kubisch – both engineers at RisingStack.