Table of Contents
回复内容:
Home Backend Development PHP Tutorial 约 1亿条记录, 每条1k左右,key =>value形式,用于前台查询,选择什么作为存储方案比较合适呢,要求效率比较高并且相对稳定可靠?

约 1亿条记录, 每条1k左右,key =>value形式,用于前台查询,选择什么作为存储方案比较合适呢,要求效率比较高并且相对稳定可靠?

Jun 17, 2016 am 08:32 AM
mysql

每天需要从文件导入一次,目前想到使用mysql分表存储,不知道有没有更好的办法。

回复内容:

对于这个小问题, 如果 key 比较小, 可以将所有 key-pos 放内存, pos 为文件中 value 的偏移, 每次访问只需要一次磁盘seek.

用 protocol-buf 或 thrift 架成网络服务, 还可以加一层proxy, 按 key hash 分发到不同的服务器来减小压力.

嗯, 好像这些, beansdb 都已经做好了. 数据之间没有强关联,这种情况下用mongo+redis是比较合适的,热点数据保留在redis里,其他数据md5分散到若干台mongodb. 然后请求分散到不同的mongodb上.
扩展也方便 1.总元数据量是100G不到,加上主键索引的话,预计100G-120G

2.每天需要从文件导入一次,目前想到使用mysql分表存储,不知道有没有更好的办法 ?这个不太清楚,是当下已经有1亿条记录,还是每天累加在一起之后,可能为1亿条记录

这个非常关键,若是每天累计在一起是1亿条记录,且是按PK查询查询的话,没有必要分表



3.非常关键的一点,只有查询 还是可能也提供其他的操作,比如UPDATE,INSERT,一定要确认清楚

4.查询的话是按PK单条的方式,还是大批量的读出,关键点是否需要GROUP BY 统计,ORDER BY 分页


5.主键是否简单比如1个或2个整型字段的模式

6.采用MySQL的话,若是非常简单的查询,且按PK,可以考虑采用Handlersocket的模式,对于复杂的建议走SQL协议...


推荐handlersocket的技术文章资料:
HandlerSocket的原理等系列篇章
mysqlops.com/2011/10/19

1. mongodb+redis
更酷,且对于你的场景来说,没有风险(即使丢失1天数据,也可以重新导入),成熟可靠.
mongodb存储数据(须使用主从or replication set),redis做cache.
2. mysql+memcached.
毫无疑问可以实现你的需求. 倒入之后只读的?用 java 随便写一个就可以了吧。index 放在内存,数据放在硬盘。如果访问有冗余,加个 cache 就好了。 使用Redis+Mysql吧,Mysql作磁带,Redis做前台的查询,比较合适。稳定,快速。 推荐用redis,它把k-v数据缓存在内存里面,性能很高,技术成熟稳定好,支持主从同步 thuir.org/thuirdb/
刚在微博上看到的,非常符合你的需求。

当然,不推荐使用不成熟的技术。 mark mysql用来存数据,redis用来查询,纯内存操作,速度绝对快!

缺点是硬件投入较大,1亿条数据全得存到redis服务器的内存里,redis里用不上的功能,全部不用。
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1662
14
PHP Tutorial
1262
29
C# Tutorial
1235
24
MySQL's Role: Databases in Web Applications MySQL's Role: Databases in Web Applications Apr 17, 2025 am 12:23 AM

The main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.

Laravel Introduction Example Laravel Introduction Example Apr 18, 2025 pm 12:45 PM

Laravel is a PHP framework for easy building of web applications. It provides a range of powerful features including: Installation: Install the Laravel CLI globally with Composer and create applications in the project directory. Routing: Define the relationship between the URL and the handler in routes/web.php. View: Create a view in resources/views to render the application's interface. Database Integration: Provides out-of-the-box integration with databases such as MySQL and uses migration to create and modify tables. Model and Controller: The model represents the database entity and the controller processes HTTP requests.

How to start mysql by docker How to start mysql by docker Apr 15, 2025 pm 12:09 PM

The process of starting MySQL in Docker consists of the following steps: Pull the MySQL image to create and start the container, set the root user password, and map the port verification connection Create the database and the user grants all permissions to the database

MySQL and phpMyAdmin: Core Features and Functions MySQL and phpMyAdmin: Core Features and Functions Apr 22, 2025 am 12:12 AM

MySQL and phpMyAdmin are powerful database management tools. 1) MySQL is used to create databases and tables, and to execute DML and SQL queries. 2) phpMyAdmin provides an intuitive interface for database management, table structure management, data operations and user permission management.

Solve database connection problem: a practical case of using minii/db library Solve database connection problem: a practical case of using minii/db library Apr 18, 2025 am 07:09 AM

I encountered a tricky problem when developing a small application: the need to quickly integrate a lightweight database operation library. After trying multiple libraries, I found that they either have too much functionality or are not very compatible. Eventually, I found minii/db, a simplified version based on Yii2 that solved my problem perfectly.

Laravel framework installation method Laravel framework installation method Apr 18, 2025 pm 12:54 PM

Article summary: This article provides detailed step-by-step instructions to guide readers on how to easily install the Laravel framework. Laravel is a powerful PHP framework that speeds up the development process of web applications. This tutorial covers the installation process from system requirements to configuring databases and setting up routing. By following these steps, readers can quickly and efficiently lay a solid foundation for their Laravel project.

MySQL vs. Other Programming Languages: A Comparison MySQL vs. Other Programming Languages: A Comparison Apr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages ​​such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages ​​have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

MySQL for Beginners: Getting Started with Database Management MySQL for Beginners: Getting Started with Database Management Apr 18, 2025 am 12:10 AM

The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA

See all articles