How to clean up Redis memory fragments

What is Redis memory fragmentation?

The total amount of remaining space in the operating system is sufficient, but when applying for a space with a continuous address of N bytes, there is no continuous space of N bytes in the remaining memory space. Then, among the remaining memory spaces, less than N The contiguous memory space of bytes is memory fragmentation.

How is Redis memory fragmentation formed?

There are internal and external reasons for the formation of memory fragmentation:

• Internal reason: The allocation strategy of the memory allocator determines that the operating system cannot achieve "allocation on demand".

• Redis uses libc, jemalloc, and tcmalloc to allocate memory. By default, jemalloc is used.

• The memory allocator allocates memory space according to a fixed size, not entirely according to the memory size requested by the application.

• Taking jemalloc as an example, it divides the memory space according to a series of fixed sizes, such as 8 bytes, 16 bytes, 32 bytes,..., 2KB, 4KB, etc. Jemalloc will allocate a fixed size space closest to the memory requested by the program.

• External reason: The key-value pairs are of different sizes, and the key-value pairs can be modified and deleted.

• When Redis applies for memory space allocation, for memory space requirements of different sizes, the memory allocator allocates memory space according to a fixed size. The allocated memory space is generally larger than the requested memory space. The memory space is larger, which will produce certain memory fragmentation.

• Key-value pairs will be modified and deleted, which will lead to space expansion and release.

How to determine whether Redis has memory fragmentation?

DAS queries the detailed information of memory usage through the INFO command provided by Redis. The command is as follows:

INFO memory
# Memory
used_memory:350458970752
used_memory_human:326.39G
used_memory_rss:349066919936
used_memory_rss_human:325.09G
…
mem_fragmentation_ratio:1.00

• used_memory: Indicates that Redis is used to save data. The actual memory space requested.

• used_memory_rss: Indicates the physical memory space actually allocated to Redis by the operating system, which includes memory space fragments.

• mem_fragmentation_ratio refers to the current memory fragmentation rate of Redis. Calculation formula: mem_fragmentation_ratio=used_memory_rss/used_memory

• mem_fragmentation_ratio is greater than or equal to 1 but less than or equal to 1.5. This situation is reasonable.

• mem_fragmentation_ratio is greater than 1.5, indicating that the memory fragmentation rate has exceeded 50%.

How to clean up memory fragments?

A "simple and crude" method is to restart the Redis instance. However, this method will bring two consequences:

• If the data in Redis is not persisted, the data will be lost;

• Regardless of the Redis data Whether it is persisted or not, the AOF or RDB method needs to be used to restore the data, and the recovery time depends on the size of the AOF or RDB file. And if there is only one Redis instance, services cannot be provided during the recovery phase.

Is there a better way? Yes, from version 4.0-RC3 onwards, Redis itself provides a method to automatically clean up memory fragments.

Automatic cleaning of memory fragments

Cleaning of memory fragments, simply put, means "moving to make way and merging space".

When there is data that divides a continuous memory space into several discontinuous spaces, the operating system will copy the data to another place, and the original discontinuous memory space becomes a continuous memory space. .

But fragmentation cleanup comes at a cost. Moving multiple copies of data to a new location and freeing up the original space is something the operating system must do, but this process takes time. In addition, when data is copied, Redis will be blocked and performance will be reduced.

How to alleviate this problem?

Redis specifically provides parameter settings for the automatic memory fragmentation cleaning mechanism. You can set parameters to control the start and end timing of fragmentation cleaning, as well as the proportion of CPU occupied, thereby reducing the performance impact of fragmentation cleaning on Redis request processing.

First, turn on automatic memory fragmentation cleanup:

config set activedefrag yes
Then, set the conditions that trigger memory cleanup:

• active- defrag-ignore-bytes 100mb: Indicates that when the number of bytes of memory fragmentation reaches 100MB, cleanup will begin;

• active-defrag-threshold-lower 10: Indicates that the memory fragmentation space occupies the operating system allocation When the total space allocated to Redis reaches 10%, cleanup begins.

Finally, control the upper and lower limits of the proportion of CPU time occupied by cleaning operations:

• active-defrag-cycle-min 25: Indicates automatic cleaning The proportion of CPU time used in the process is not less than 25% to ensure that the cleaning can proceed normally;

• active-defrag-cycle-max 75: Indicates that the proportion of CPU time used in the automatic cleaning process is not high Above 75%, once it exceeds, the cleaning will be stopped to avoid a large number of memory copies blocking Redis during cleaning, resulting in increased response latency.

The above is the detailed content of How to clean up Redis memory fragments. For more information, please follow other related articles on the PHP Chinese website!