Redis performance tuning – Top 10 mistakes

Redis is the most popular Key-value data store and one of the most popular database systems overall.

 

According to Db-engine ranking:

https://db-engines.com/en/ranking/key-value+store

(You can check the picture below), Redis is on the top position by large margin among Key-values data stores.

Amazon DynamoDb on the second place, lags behind Redis more than a twice.

 

Even if you take a look overall database popularity index which includes RDBMS databases such as Oracle, Postgres, MySQL and others, Redis is on the 6^th position, and its popularity is still growing.

https://db-engines.com/en/ranking

 

 

The reason behind Redis popularity is in its simplicity and high performance.

Additionally, Redis is always good fit for all types of architecture (SOA, Microservice architecture, Web…).

The most common use case for Redis is to speed up data retrieval when latency and fast response are critical.

In this article I’ll describe top 10 mistakes I’ve found when inspecting systems that includes Redis as part of architecture.

 

1. Running production with outdated version of Redis

The main reason why on so many places I can see old version of Redis in production is probably due to the fact its part of official Red Hat/CentOS repository.

In RHEL/CentOS 7 its Redis 3.2 while the current release as of time of writing is 6.2.6.

You can check the version by executing the following command:

redis-server --version
Redis server v=3.2.10 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=c8b45a0ec7dc67c6

 

It is difficult to list all improvements made between 3.2 and the current release (you can check in the Release notes).

Solution: Instead to rely on official repo, install newest release manually.

 

2. Incorrectly set Linux memory related parameters

There are two Linux related parameters you need to check:

• vm.overcommit_memory

• vm.swappiness

 

vm.overcommit_memory

Redis extensively uses copy-on-write when starting background slave if persistence is enabled.

For that reason, there is no need to make free amount of RAM equal to size of dataset in Redis.

To avoid a risk of getting Redis killed by Linux OOM (Out of memory), you should set overcommit parameter to 1.

With overcommit parameter set to 1, each call to malloc() function (Redis is written in pure C, which explains why it is so fast) will succeed.

 

swappiness

The main purpose of Redis is to speed up critical part of your system, you want to get values out from memory, not from disk.

This is where swappiness parameter jumps in as it defines how much and how often Linux kernel will copy RAM content to swap.

There is another memory related parameter – Transparent Huge pages, but I won’t describe it here because if it is enabled in Linux kernel, you will get warning when starting Redis.

Because of that, THP feature of Linux kernel are usually disabled.

 

3. Identify Redis clients by using CLIENT SETNAME

Setting the client who connects to Redis is one of the most important, although often overlooked feature.

It’s essential if you want to identify incoming connections by its name in cases when more than one application connects to Redis server.

By executing INFO CLIENTS you can get related metrics with human names for each client.

 

4. Network related kernel parameters not set correctly

There are two network parameters that are often leave at their default values:

• net.core.somaxconn

• net.ipv4.tcp_max_syn_backlog

with default values 512 and 4096 respectively.

Proposed new values for those two parameters are: 65535 for both, much higher than default values.

 

First parameter (somaxconn) sets upper limit of backlog parameter which are passed to listen function.

Second parameter (tcp_max_syn_backlog) sets the maximum queue length of all pending connections.

This is related to tcp-backlog parameter (which I set to 511), which you need to increase.

It sets the size of the pending socket requests.

 

5. Number of files that process can open

Here we have two important parameters:

• redis soft nofile

• redis hard nofile

 

It is important to keep those values higher than maxclient.

I usually set both values to 288000, and set maxclient to 10000.

 

6. Setting timeout for incoming connections

Parameter timeout means that Redis server will close connection after X seconds (client idle time).

I usually set this parameter to 0.

The other connected parameter is tcp-keepalive which I set to 300.

If you set it to nonzero value, server will send TCP ACK signal to inform the network that connection is still alive.

 

7. Client output buffer parameters

• client-output-buffer-limit normal 0 0 0

• client-output-buffer-limit slave 512mb 256mb 60

• client-output-buffer-limit pubsub 32mb 8mb 60

 

Main purpose of the last three parameters is to allow Redis to first populate the result into an output buffer for each connection, and to send the content in one batch.

 

8. Choosing correct data type

Although it seems to be obvious, often I can see incorrect data type has been used.

For that reason, choosing correct data type with related APIs is essential since it will have huge impact on performance and memory footprint.

One such example is string versus JSON or hash Redis data types.

 

9. Select the correct API

There is usually more than one choice of Redis API that can do the same task.

To achieve good performance, you need to carefully chose a method based on your internal knowledge of Redis.

Here it is important to recognize that Redis is single threaded key-value store.

It means if you use HGETALL to fetch data from the Redis hash, significant latency can be expected.

During time of processing HGETALL command, Redis will be unavailable to serve other requests.

For that reason, in case of hash data type, it is always better to perform HSCAN command in a loop.

There are some other commands such as DEL, HDEL etc. that could block the Redis server which is something you don’t want to see.

 

10. Select the correct method to do your job

Second goal when developing application that will use Redis is elimination of the round-trip time.

For example, you can use HSET to set hash values one by one, or you can use HMSET to do the same in one shot.

In a later case, dramatic performance boost is expected.

 

Summary:

I this article I’ve listed the most common pitfall found when I need to tune systems which includes Redis.

The list is far from complete since many important things are missing, starting from benchmarking, monitoring, troubleshooting and persistency.

For that reason, you can use my “Top 10 mistakes” list as a starting point to tune your own Redis cluster.

---