[转]缓存大量小文件？Redis是首选！
现在有3KW的数据，单条数据都很小的，如果按key-value来看的话，key就是32位的MD5字符串，value按平均算大概是100字节左右。 现在需要将这些数据做缓存以在高并发的时候依然可以快速响应。 因为这些数据基本没有冷热数据之分，所以需要将全部数据都放到缓存
现在有3KW的数据，单条数据都很小的，如果按key-value来看的话，key就是32位的MD5字符串，value按平均算大概是100字节左右。
现在需要将这些数据做缓存以在高并发的时候依然可以快速响应。
因为这些数据基本没有冷热数据之分，所以需要将全部数据都放到缓存中。
1、直接生成静态文件，利用nginx对静态文件的高效做静态缓存。
- 当时服务器硬件资源有限，所以就采用这种方式，一直源用至今。
- 服务器间通过NFS来共享
- 太多小文件，不方便管理
- NFS不方便运维与扩展
-
文件内容很小(100字节左右)，3KW大概就是2.5G大小左右
- 不过文件存储的时候和硬盘分区的族大小有关，在这里磁盘分区的族大小为8K，所以尽管文件内容只有100字节，但是实际存储到磁盘上的时候单个文件其实是8K
- 所以3KW的文件世界占的磁盘空间大约为：200G左右（ 严重浪费空间啊）
2、Redis(V2.2.11)【KV数据库】
- 听同事说开启VM会使性能急剧下降，所以基本无视VM，数据全放内存。
- key为32位MD5字符串
- 测试数据：10W数据大概占内存20M
- 测试数据：500W数据大概占内存1G，持久化的rdb数据文件大概350M
- 推算3KW数据：内存6G，持久化的rdb数据文件大概为2G(压缩了？)
- 因为Redis在持久化的时候内存会加倍，和考虑到数据的增长，所以需要1台20G内存的机器基本就没问题了（容灾啥的另算）。
- Redis非常快，如果硬件条件没问题，基本选这个最好了。
3、Moosefs(MFS)【分布式文件存储系统】
- mfs支持NFS的方式mount到本地直接操作（如使用mfs，则现在的架构基本不用改）
- 最基本的需要一台主控服务器(Master Server)、一台数据服务器(Chunk Server)
- 文件和目录的索引需要全部加载到主控服务器的内存中，所以对主控服务器的内存有一定的要求
- 写入30W文件，到20W的时候写入就开始下降得厉害了（我的5400转的笔记本硬盘）
- 30W文件，4W目录，主控服务器占用大概120M内存
- 文件存储方式貌似和普遍的文件一样单个存储的（不确定），30W文件大概占了2.4G的磁盘空间（同样是8K一个文件）。
- 小文件一样很多，不方便维护与迁移(不知是否我设置不对？)。
- 内存、硬盘都占用得比较多，而且性能相对来说不是很出众。
- 所以这个基本不考虑了。
作者：QLeelulu??出处： http://QLeelulu.cnblogs.com/
作者：zeo112140 发表于2013-7-11 10:28:48 原文链接
阅读：7 评论：0 查看评论
![[转]缓存大量小文件？Redis是首选！](http://www.68idc.cn/help/uploads/allimg/150219/10132B364-0.gif "IT 资讯")
原文地址：[转]缓存大量小文件？Redis是首选！, 感谢原作者分享。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How to build the redis cluster mode
Apr 10, 2025 pm 10:15 PM
Redis cluster mode deploys Redis instances to multiple servers through sharding, improving scalability and availability. The construction steps are as follows: Create odd Redis instances with different ports; Create 3 sentinel instances, monitor Redis instances and failover; configure sentinel configuration files, add monitoring Redis instance information and failover settings; configure Redis instance configuration files, enable cluster mode and specify the cluster information file path; create nodes.conf file, containing information of each Redis instance; start the cluster, execute the create command to create a cluster and specify the number of replicas; log in to the cluster to execute the CLUSTER INFO command to verify the cluster status; make
How to clear redis data
Apr 10, 2025 pm 10:06 PM
How to clear Redis data: Use the FLUSHALL command to clear all key values. Use the FLUSHDB command to clear the key value of the currently selected database. Use SELECT to switch databases, and then use FLUSHDB to clear multiple databases. Use the DEL command to delete a specific key. Use the redis-cli tool to clear the data.
How to read redis queue
Apr 10, 2025 pm 10:12 PM
To read a queue from Redis, you need to get the queue name, read the elements using the LPOP command, and process the empty queue. The specific steps are as follows: Get the queue name: name it with the prefix of "queue:" such as "queue:my-queue". Use the LPOP command: Eject the element from the head of the queue and return its value, such as LPOP queue:my-queue. Processing empty queues: If the queue is empty, LPOP returns nil, and you can check whether the queue exists before reading the element.
How to use the redis command
Apr 10, 2025 pm 08:45 PM
Using the Redis directive requires the following steps: Open the Redis client. Enter the command (verb key value). Provides the required parameters (varies from instruction to instruction). Press Enter to execute the command. Redis returns a response indicating the result of the operation (usually OK or -ERR).
How to configure Lua script execution time in centos redis
Apr 14, 2025 pm 02:12 PM
On CentOS systems, you can limit the execution time of Lua scripts by modifying Redis configuration files or using Redis commands to prevent malicious scripts from consuming too much resources. Method 1: Modify the Redis configuration file and locate the Redis configuration file: The Redis configuration file is usually located in /etc/redis/redis.conf. Edit configuration file: Open the configuration file using a text editor (such as vi or nano): sudovi/etc/redis/redis.conf Set the Lua script execution time limit: Add or modify the following lines in the configuration file to set the maximum execution time of the Lua script (unit: milliseconds)
How to use redis lock
Apr 10, 2025 pm 08:39 PM
Using Redis to lock operations requires obtaining the lock through the SETNX command, and then using the EXPIRE command to set the expiration time. The specific steps are: (1) Use the SETNX command to try to set a key-value pair; (2) Use the EXPIRE command to set the expiration time for the lock; (3) Use the DEL command to delete the lock when the lock is no longer needed.
How to use the redis command line
Apr 10, 2025 pm 10:18 PM
Use the Redis command line tool (redis-cli) to manage and operate Redis through the following steps: Connect to the server, specify the address and port. Send commands to the server using the command name and parameters. Use the HELP command to view help information for a specific command. Use the QUIT command to exit the command line tool.
How to set the redis expiration policy
Apr 10, 2025 pm 10:03 PM
There are two types of Redis data expiration strategies: periodic deletion: periodic scan to delete the expired key, which can be set through expired-time-cap-remove-count and expired-time-cap-remove-delay parameters. Lazy Deletion: Check for deletion expired keys only when keys are read or written. They can be set through lazyfree-lazy-eviction, lazyfree-lazy-expire, lazyfree-lazy-user-del parameters.
