Home Database Mysql Tutorial 缓解MySQL写入压力和主从延迟的尝试

缓解MySQL写入压力和主从延迟的尝试

Jun 07, 2016 pm 03:21 PM

最近单位需要用MySQL存放大量的日志数据,写入压力很大,并且有很大的主从延迟。

最近单位需要用MySQL存放大量的日志数据,写入压力很大,并且有很大的主从延迟。

具体环境如下
MySQL 5.6.14
服务器(单CPU,6核心,12线程 32G内存)
服务器硬盘(共33T,Raid5)

第一个尝试,分散IO
一般我们使用/dbdata挂载点存放数据文件
/data挂载点存放日志文件(redo log file,binlog,relay log等)
这样的好处是将随机IO和顺序IO分开,不形成争用.

缓解MySQL写入压力和主从延迟的尝试

缺点是/data挂载点的IO使用率一般较低.

缓解MySQL写入压力和主从延迟的尝试

当然,这种情况在一般用途的数据库并无大碍.
但是在Insert密集的数据库,数据文件所在的物理设备使用率会一直保持在100%.而日志文件所在的物理设备使用率一般也就是5%左右,甚至更低.

这时,也许可以考虑将一部分表的数据文件移动到日志文件所在的物理设备,以平衡IO资源使用.

缓解MySQL写入压力和主从延迟的尝试

MySQL的datadir在/dbdata挂载点,
police_im_user_mac数据库通过软链接,实际指向/data的挂载点.
这样,会将一部分的随机IO分散到/data挂载点,减轻/dbdata挂载点的压力,并且减小了CPU的等待

缓解MySQL写入压力和主从延迟的尝试


 第二个尝试,一个库一个表
 因为MySQL 5.6的多线程复制是基于数据库的.
而我们每个表的写入压力都很大,所以修改了结构,每个数据库只有一个表.然后启动多线程复制(实际上就是多个SQL线程),并且设置传输压缩

slave_compressed_protocol=on (Master,Slave都需要配置)
slave_parallel_workers=10

第三个尝试,修改参数
innodb_flush_log_at_trx_commit = 0
 innodb_support_xa=0
 sync_binlog=0
增加innodb_buffer_pool_size
将innodb_max_dirty_pages_pct设置的更大
 将redolog file size设置的更大

经过这些修改,写入压力和主从延迟有了很大的缓解,,但是启动slave_parallel_workers,一旦出现SQL线程异常导致的复制中断,恢复之后,可能会出现1062号错误

这是因为多SQL线程的线程同步机制可能有问题,一旦复制中断,现象上看,等同于Slave异常关机.

当然,处理方法也是一样的

本文永久更新链接地址

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

When might a full table scan be faster than using an index in MySQL? When might a full table scan be faster than using an index in MySQL? Apr 09, 2025 am 12:05 AM

Full table scanning may be faster in MySQL than using indexes. Specific cases include: 1) the data volume is small; 2) when the query returns a large amount of data; 3) when the index column is not highly selective; 4) when the complex query. By analyzing query plans, optimizing indexes, avoiding over-index and regularly maintaining tables, you can make the best choices in practical applications.

Explain InnoDB Full-Text Search capabilities. Explain InnoDB Full-Text Search capabilities. Apr 02, 2025 pm 06:09 PM

InnoDB's full-text search capabilities are very powerful, which can significantly improve database query efficiency and ability to process large amounts of text data. 1) InnoDB implements full-text search through inverted indexing, supporting basic and advanced search queries. 2) Use MATCH and AGAINST keywords to search, support Boolean mode and phrase search. 3) Optimization methods include using word segmentation technology, periodic rebuilding of indexes and adjusting cache size to improve performance and accuracy.

Can I install mysql on Windows 7 Can I install mysql on Windows 7 Apr 08, 2025 pm 03:21 PM

Yes, MySQL can be installed on Windows 7, and although Microsoft has stopped supporting Windows 7, MySQL is still compatible with it. However, the following points should be noted during the installation process: Download the MySQL installer for Windows. Select the appropriate version of MySQL (community or enterprise). Select the appropriate installation directory and character set during the installation process. Set the root user password and keep it properly. Connect to the database for testing. Note the compatibility and security issues on Windows 7, and it is recommended to upgrade to a supported operating system.

Difference between clustered index and non-clustered index (secondary index) in InnoDB. Difference between clustered index and non-clustered index (secondary index) in InnoDB. Apr 02, 2025 pm 06:25 PM

The difference between clustered index and non-clustered index is: 1. Clustered index stores data rows in the index structure, which is suitable for querying by primary key and range. 2. The non-clustered index stores index key values ​​and pointers to data rows, and is suitable for non-primary key column queries.

MySQL: Simple Concepts for Easy Learning MySQL: Simple Concepts for Easy Learning Apr 10, 2025 am 09:29 AM

MySQL is an open source relational database management system. 1) Create database and tables: Use the CREATEDATABASE and CREATETABLE commands. 2) Basic operations: INSERT, UPDATE, DELETE and SELECT. 3) Advanced operations: JOIN, subquery and transaction processing. 4) Debugging skills: Check syntax, data type and permissions. 5) Optimization suggestions: Use indexes, avoid SELECT* and use transactions.

Explain different types of MySQL indexes (B-Tree, Hash, Full-text, Spatial). Explain different types of MySQL indexes (B-Tree, Hash, Full-text, Spatial). Apr 02, 2025 pm 07:05 PM

MySQL supports four index types: B-Tree, Hash, Full-text, and Spatial. 1.B-Tree index is suitable for equal value search, range query and sorting. 2. Hash index is suitable for equal value searches, but does not support range query and sorting. 3. Full-text index is used for full-text search and is suitable for processing large amounts of text data. 4. Spatial index is used for geospatial data query and is suitable for GIS applications.

The relationship between mysql user and database The relationship between mysql user and database Apr 08, 2025 pm 07:15 PM

In MySQL database, the relationship between the user and the database is defined by permissions and tables. The user has a username and password to access the database. Permissions are granted through the GRANT command, while the table is created by the CREATE TABLE command. To establish a relationship between a user and a database, you need to create a database, create a user, and then grant permissions.

Can mysql and mariadb coexist Can mysql and mariadb coexist Apr 08, 2025 pm 02:27 PM

MySQL and MariaDB can coexist, but need to be configured with caution. The key is to allocate different port numbers and data directories to each database, and adjust parameters such as memory allocation and cache size. Connection pooling, application configuration, and version differences also need to be considered and need to be carefully tested and planned to avoid pitfalls. Running two databases simultaneously can cause performance problems in situations where resources are limited.

See all articles