Table of Contents
MySQL 大数据操作注意事项
http://netkiller.github.io/journal/mysql.parallel.html
Mr. Neo Chen (netkiller), 陈景峰(BG7NYT)
1. 关于 delete
2. 关于 update
3. 关于创建索引
4. 关于 OPTIMIZE
5. 确保SELECT不被受阻
Home Database Mysql Tutorial MySQL 大数据操作注意事项_MySQL

MySQL 大数据操作注意事项_MySQL

Jun 01, 2016 pm 01:11 PM

MySQL 大数据操作注意事项

http://netkiller.github.io/journal/mysql.parallel.html

Mr. Neo Chen (netkiller), 陈景峰(BG7NYT)


中国广东省深圳市龙华新区民治街道溪山美地
518131
+86 13113668890
+86 755 29812080

版权 © 2011, 2012, 2013, 2014 http://netkiller.github.io

版权声明

转载请与作者联系,转载时请务必标明文章原始出处和作者信息及本声明。

MySQL 大数据操作注意事项_MySQL
文档出处:
http://netkiller.github.io
http://netkiller.sourceforge.net

2014-05-16

摘要 我的系列文档
Netkiller Architect 手札 Netkiller Developer 手札 Netkiller PHP 手札 Netkiller Python 手札 Netkiller Testing 手札 Netkiller Cryptography 手札
Netkiller Linux 手札 Netkiller Debian 手札 Netkiller CentOS 手札 Netkiller FreeBSD 手札 Netkiller Shell 手札 Netkiller Security 手札
Netkiller Web 手札 Netkiller Monitoring 手札 Netkiller Storage 手札 Netkiller Mail 手札 Netkiller Docbook 手札 Netkiller Version 手札
Netkiller Database 手札 Netkiller PostgreSQL 手札 Netkiller MySQL 手札 Netkiller NoSQL 手札 Netkiller LDAP 手札 Netkiller Network 手札
Netkiller Cisco IOS 手札 Netkiller H3C 手札 Netkiller Multimedia 手札 Netkiller Perl 手札 Netkiller Amateur Radio 手札  

目录
  • 1. 关于 delete
  • 2. 关于 update
  • 3. 关于创建索引
  • 4. 关于 OPTIMIZE
  • 5. 确保SELECT不被受阻

1. 关于 delete

delete from mytable 必死无疑,你需要分批删除,尽量缩小每个批次删除的记录数,delete 是可以并行执行的,你可以同时运行多个删除操作

mysql> show processlist;+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+| Id     | User            | Host                | db        | Command | Time  | State                       | Info                                                   |+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+|      1 | event_scheduler | localhost           | NULL      | Daemon  |    52 | Waiting for next activation | NULL                                                   || 115986 | dba             | localhost           | example   | Query   |     0 | NULL                        | show processlist                                       || 117446 | dba             | localhost           | example   | Query   |    20 | updating                    | delete from mytable where OPEN_TIME like '2011.11.28%' || 117525 | dba             | localhost           | example   | Query   |     2 | updating                    | delete from mytable where OPEN_TIME like '2011.12.02%' || 117526 | dba             | localhost           | example   | Query   |    49 | updating                    | delete from mytable where OPEN_TIME like '2011.12.12%' || 117527 | dba             | localhost           | example   | Query   |     6 | updating                    | delete from mytable where OPEN_TIME like '2011.12.21%' || 117528 | dba             | localhost           | example   | Query   |    64 | updating                    | delete from mytable where OPEN_TIME like '2011.12.30%' || 117546 | dba             | localhost           | example   | Query   |    33 | updating                    | delete from mytable where OPEN_TIME like '2011.11.10%' |+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+23 rows in set (0.00 sec)
Copy after login

2. 关于 update

在电商领域常常遇到一个问题“调价”,经常需要调整一批商品的价格, 程序猿一条预警搞定有没有?

update goods set price=price+10 where category_id = xxx
Copy after login

在开发,测试环境是可以通过测试的,一旦部署到生产环境,必死无疑

3. 关于创建索引

大表创建索引需要很久的时间,通常要经历 manage keys 与 copy to tmp table 的过程

mysql> show processlist;+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+| Id     | User            | Host                | db       | Command | Time  | State                       | Info                                                             |+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+|      1 | event_scheduler | localhost           | NULL     | Daemon  |    47 | Waiting for next activation | NULL                                                             || 115986 | dba             | localhost           | example  | Query   |     0 | NULL                        | show processlist                                                 || 118814 | dba             | 192.168.6.20:50459  | example  | Query   |     8 | copy to tmp table           | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`)   |+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+17 rows in set (0.00 sec)
Copy after login

删除索引,也需要经理 copy to tmp table 过程,漫长的等待

mysql> show processlist;+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+| Id     | User            | Host                | db           | Command | Time  | State                       | Info                                            |+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+|      1 | event_scheduler | localhost           | NULL         | Daemon  |    11 | Waiting for next activation | NULL                                            || 115986 | dba             | localhost           | example      | Query   |     0 | NULL                        | show processlist                                || 118814 | dba             | 192.168.6.20:50459  | example      | Query   |     4 | copy to tmp table           | ALTER TABLE `mytable`	DROP INDEX `modifiy_time` |+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+17 rows in set (0.00 sec)
Copy after login

所以数据设计要深思熟虑,做到提前未雨绸缪,不要亡羊补牢

4. 关于 OPTIMIZE

OPTIMIZE 的操作是将当前表复制到临时表操作后再删除当前表,最后将临时表改名

mysql> show processlist;+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+| Id     | User            | Host                | db                        | Command | Time  | State                       | Info                     |+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+|      1 | event_scheduler | localhost           | NULL                      | Daemon  |    14 | Waiting for next activation | NULL                     || 115835 | dba             | 192.168.6.20:49664  | example                   | Query   |     9 | copy to tmp table           | OPTIMIZE TABLE `mytable` || 115986 | dba             | localhost           | example                   | Query   |     0 | NULL                        | show processlist         |+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+17 rows in set (0.00 sec)
Copy after login

5. 确保SELECT不被受阻

使用各种手段保证select操作不被受阻,只要select一直可以查询网站前端就能提供80%的功能,一旦select受阻一切都是浮云。

保证 select 操作优先于其他操作

UPDATE [LOW_PRIORITY] [IGNORE] tbl_name  SET col_name1=expr1 [, col_name2=expr2 ...]  [WHERE where_definition]  [ORDER BY ...]  [LIMIT row_count]
Copy after login

update的时候增加 LOW_PRIORITY 参数,可以降低更新语句的优先级。

my.cnf

[mysqld]		low_priority_updates=1
Copy after login

或者启动是添加--low-priority-updates参数

全局开启

SET @@global.low_priority_updates = 1;
Copy after login

适用于本次会话连接

SET @@session.low_priority_updates = 1;
Copy after login
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1268
29
C# Tutorial
1247
24
MySQL: Simple Concepts for Easy Learning MySQL: Simple Concepts for Easy Learning Apr 10, 2025 am 09:29 AM

MySQL is an open source relational database management system. 1) Create database and tables: Use the CREATEDATABASE and CREATETABLE commands. 2) Basic operations: INSERT, UPDATE, DELETE and SELECT. 3) Advanced operations: JOIN, subquery and transaction processing. 4) Debugging skills: Check syntax, data type and permissions. 5) Optimization suggestions: Use indexes, avoid SELECT* and use transactions.

MySQL's Role: Databases in Web Applications MySQL's Role: Databases in Web Applications Apr 17, 2025 am 12:23 AM

The main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.

Explain the role of InnoDB redo logs and undo logs. Explain the role of InnoDB redo logs and undo logs. Apr 15, 2025 am 12:16 AM

InnoDB uses redologs and undologs to ensure data consistency and reliability. 1.redologs record data page modification to ensure crash recovery and transaction persistence. 2.undologs records the original data value and supports transaction rollback and MVCC.

MySQL: An Introduction to the World's Most Popular Database MySQL: An Introduction to the World's Most Popular Database Apr 12, 2025 am 12:18 AM

MySQL is an open source relational database management system, mainly used to store and retrieve data quickly and reliably. Its working principle includes client requests, query resolution, execution of queries and return results. Examples of usage include creating tables, inserting and querying data, and advanced features such as JOIN operations. Common errors involve SQL syntax, data types, and permissions, and optimization suggestions include the use of indexes, optimized queries, and partitioning of tables.

MySQL's Place: Databases and Programming MySQL's Place: Databases and Programming Apr 13, 2025 am 12:18 AM

MySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen

Why Use MySQL? Benefits and Advantages Why Use MySQL? Benefits and Advantages Apr 12, 2025 am 12:17 AM

MySQL is chosen for its performance, reliability, ease of use, and community support. 1.MySQL provides efficient data storage and retrieval functions, supporting multiple data types and advanced query operations. 2. Adopt client-server architecture and multiple storage engines to support transaction and query optimization. 3. Easy to use, supports a variety of operating systems and programming languages. 4. Have strong community support and provide rich resources and solutions.

MySQL vs. Other Programming Languages: A Comparison MySQL vs. Other Programming Languages: A Comparison Apr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages ​​such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages ​​have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

How does MySQL index cardinality affect query performance? How does MySQL index cardinality affect query performance? Apr 14, 2025 am 12:18 AM

MySQL index cardinality has a significant impact on query performance: 1. High cardinality index can more effectively narrow the data range and improve query efficiency; 2. Low cardinality index may lead to full table scanning and reduce query performance; 3. In joint index, high cardinality sequences should be placed in front to optimize query.

See all articles