Redis的内存分析(set/get命令集)
为了知道Redis是如何使用内存的,我分析了源码。这篇文章针对的是set/get命令集,也就是key、value都是简单的字符串。除此之外,Redis还支持像hset/hget命令,它们的value是一个字典,留到以后分析。 首先来看看Redis存储数据的关键数据结构: 1. zmalloc (
为了知道Redis是如何使用内存的,我分析了源码。这篇文章针对的是set/get命令集,也就是key、value都是简单的字符串。除此之外,Redis还支持像hset/hget命令,它们的value是一个字典,留到以后分析。
首先来看看Redis存储数据的关键数据结构:
1. zmalloc (见zmalloc.h)
Redis是用zmalloc管理内存的,其底层可以是glibc malloc、jemalloc、tcmalloc,具体由用户选择,目前的版本默认为jemalloc。本文不分析jemalloc,而是从zmalloc这一层开始往上分析。
zmalloc封装了底层内存管理,并为每一段申请的内存增加了一个头部,大小为PREFIX_SIZE = 8字节,这个头部记录了申请内存的大小。因此每一块申请的内存有额外的8字节内存开销。
2. sds (见sds.h)
sds是Redis使用的字符串工具,取代string.h。sds的结构体如下,包含4=4=8字节的头部,buf后面就是字符串的空间。因此每个字符串需要8字节的额外空间。
struct sdshdr {
unsigned int len;
unsigned int free;
char buf[];
};
3. 哈希表(见dict.h)
这是核心数据结构,所有的key value都存在哈希表内。dict是哈希表的元数据,ht[2]数组是实际的哈希表,ht[0]是当前使用的哈希表;当数据太多(超过了ht[0]的数组大小),就分配2倍大的ht[1],然后逐渐将数据从ht[0]迁移到ht[1]。由于dict结构体是唯一的,它的内存开销可以忽略不计。
dictht是实际的哈希表,包含一个指针数组,数组的每一项是指向dictEntry的指针,被哈希到同一项的数据以链表的方式解决冲突。
typedef struct dict {
dictType *type;
void *privdata;
dictht ht[2];
long rehashidx; /* rehashing not in progress if rehashidx == -1 */
int iterators; /* number of iterators currently running */
} dict;
/* This is our hash table structure. Every dictionary has two of this as we
* implement incremental rehashing, for the old to the new table. */
typedef struct dictht {
dictEntry **table;
unsigned long size;
unsigned long sizemask;
unsigned long used;
} dictht;
typedef struct dictEntry {
void *key;
union {
void *val;
uint64_t u64;
int64_t s64;
double d;
} v;
struct dictEntry *next;
} dictEntry;
在redis中,key是sds字符串,而value则是robj数据结构。当插入一对key-value,会产生一个dictEntry,key指向一个sds字符串,而v.val指向一个robj。dictEntry的大小为24字节。
4. 对象robj(见redis.h)
robj结构体如下。ptr指向数据;type指明数据的类型(比如可以是sds字符串);encoding表示是否进行了编码(即压缩字符串,见下文);lru用于实现lru替换策略,每次查询都会更新lru的值;refcount是引用次数,因为Redis允许共享对象,节约内存。robj结构体的大小为16字节。
#define REDIS_LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:REDIS_LRU_BITS; /* lru time (relative to server.lruclock) */
int refcount;
void *ptr;
} robj;
内存分析
每一对key-value,需要一个sds字符串存储key,这需要额外8字节zmalloc头部+8字节sds头部,一共16字节;需要一个robj存储value,robj->ptr实际指向一个sds字符串(假设没有压缩),这需要额外16字节的robj + 8字节zmalloc头部,以及robj->ptr指向的字符串的8字节zmalloc头部和8字节sds头部,一共40字节。
插入哈希表后,需要一个24字节的dictEntry和8字节的zmalloc头部,以及一个指针数组中的8字节指针,一共40字节。
最终,每个key-value对,除去数据本身,一共需要额外 16 +40 +40 = 96字节的元数据。并且这96个字节分属于四个不同的内存区块(这里忽略哈希表指针数组):
- 存储key的sds字符串(8+8+keylen);
- 存储robj(8+16);
- 存储robj->ptr对应的value(8+8+vallen);
- 存储dictEntry(8+24)。
其中2、4是定长的小内存,1、3是变长内存。
对于key-value较小的负载,这些元数据开销并不低;以我上篇实验为例,采用了20字节的key和100字节的value,元数据开销占96/(120+96)=44.4%。
以上为set/get命令的分析,hset/hget命令的内存开销应该会更高,留待以后分析。
Redis如何节约内存
Redis通过对字符串压缩、共享对象等方式节约内存:对于每一个字符串value,Redis都尝试一下方法:
1. 如果字符串全部由数字组成,则尝试将其表示为长整形。最多可以将21字节压缩成8字节。因为指针ptr是8字节,数值可以直接存在ptr里面而无需额外内存,又节约了8字节。
2. 如果整型编码成功,且数值在1-10000之间,则使用共享对象,因为1-10000这些对象已经实现创建好了。
3. 如果整型编码不行,但是字符串又
原文地址:Redis的内存分析(set/get命令集), 感谢原作者分享。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Redis cluster mode deploys Redis instances to multiple servers through sharding, improving scalability and availability. The construction steps are as follows: Create odd Redis instances with different ports; Create 3 sentinel instances, monitor Redis instances and failover; configure sentinel configuration files, add monitoring Redis instance information and failover settings; configure Redis instance configuration files, enable cluster mode and specify the cluster information file path; create nodes.conf file, containing information of each Redis instance; start the cluster, execute the create command to create a cluster and specify the number of replicas; log in to the cluster to execute the CLUSTER INFO command to verify the cluster status; make

How to clear Redis data: Use the FLUSHALL command to clear all key values. Use the FLUSHDB command to clear the key value of the currently selected database. Use SELECT to switch databases, and then use FLUSHDB to clear multiple databases. Use the DEL command to delete a specific key. Use the redis-cli tool to clear the data.

To read a queue from Redis, you need to get the queue name, read the elements using the LPOP command, and process the empty queue. The specific steps are as follows: Get the queue name: name it with the prefix of "queue:" such as "queue:my-queue". Use the LPOP command: Eject the element from the head of the queue and return its value, such as LPOP queue:my-queue. Processing empty queues: If the queue is empty, LPOP returns nil, and you can check whether the queue exists before reading the element.

Using the Redis directive requires the following steps: Open the Redis client. Enter the command (verb key value). Provides the required parameters (varies from instruction to instruction). Press Enter to execute the command. Redis returns a response indicating the result of the operation (usually OK or -ERR).

Using Redis to lock operations requires obtaining the lock through the SETNX command, and then using the EXPIRE command to set the expiration time. The specific steps are: (1) Use the SETNX command to try to set a key-value pair; (2) Use the EXPIRE command to set the expiration time for the lock; (3) Use the DEL command to delete the lock when the lock is no longer needed.

On CentOS systems, you can limit the execution time of Lua scripts by modifying Redis configuration files or using Redis commands to prevent malicious scripts from consuming too much resources. Method 1: Modify the Redis configuration file and locate the Redis configuration file: The Redis configuration file is usually located in /etc/redis/redis.conf. Edit configuration file: Open the configuration file using a text editor (such as vi or nano): sudovi/etc/redis/redis.conf Set the Lua script execution time limit: Add or modify the following lines in the configuration file to set the maximum execution time of the Lua script (unit: milliseconds)

Use the Redis command line tool (redis-cli) to manage and operate Redis through the following steps: Connect to the server, specify the address and port. Send commands to the server using the command name and parameters. Use the HELP command to view help information for a specific command. Use the QUIT command to exit the command line tool.

In Debian systems, readdir system calls are used to read directory contents. If its performance is not good, try the following optimization strategy: Simplify the number of directory files: Split large directories into multiple small directories as much as possible, reducing the number of items processed per readdir call. Enable directory content caching: build a cache mechanism, update the cache regularly or when directory content changes, and reduce frequent calls to readdir. Memory caches (such as Memcached or Redis) or local caches (such as files or databases) can be considered. Adopt efficient data structure: If you implement directory traversal by yourself, select more efficient data structures (such as hash tables instead of linear search) to store and access directory information
