MYSQL批量插入数据库实现语句性能分析
假定我们的表结构如下 代码如下 CREATE TABLE example ( example_id INT NOT NULL, name VARCHAR( 50 ) NOT NULL, value VARCHAR( 50 ) NOT NULL, other_value VARCHAR( 50 ) NOT NULL ) 通常情况下单条插入的sql语句我们会这么写: 代码如下 INSERT INTO ex
假定我们的表结构如下
代码如下
CREATE TABLE example (
example_id INT NOT NULL,
name VARCHAR( 50 ) NOT NULL,
value VARCHAR( 50 ) NOT NULL,
other_value VARCHAR( 50 ) NOT NULL
)
通常情况下单条插入的sql语句我们会这么写:
代码如下
INSERT INTO example
(example_id, name, value, other_value)
VALUES
(100, 'Name 1', 'Value 1', 'Other 1');
mysql允许我们在一条sql语句中批量插入数据,如下sql语句:
代码如下
INSERT INTO example
(example_id, name, value, other_value)
VALUES
(100, 'Name 1', 'Value 1', 'Other 1'),
(101, 'Name 2', 'Value 2', 'Other 2'),
(102, 'Name 3', 'Value 3', 'Other 3'),
(103, 'Name 4', 'Value 4', 'Other 4');
如果我们插入列的顺序和表中列的顺序一致的话,还可以省去列名的定义,如下sql
代码如下
INSERT INTO example
VALUES
(100, 'Name 1', 'Value 1', 'Other 1'),
(101, 'Name 2', 'Value 2', 'Other 2'),
(102, 'Name 3', 'Value 3', 'Other 3'),
(103, 'Name 4', 'Value 4', 'Other 4');
上面看上去没什么问题,下面我来使用sql语句优化的小技巧,下面会分别进行测试,目标是插入一个空的数据表200W条数据
第一种方法:使用insert into 插入,代码如下:
代码如下
$params = array('value'=>'50');
set_time_limit(0);
echo date("H:i:s");
for($i=0;$i
$connect_mysql->insert($params);
};
echo date("H:i:s");
最后显示为:23:25:05 01:32:05 也就是花了2个小时多!
第二种方法:使用事务提交,批量插入数据库(每隔10W条提交下)最后显示消耗的时间为:22:56:13 23:04:00 ,一共8分13秒 ,代码如下:
代码如下
echo date("H:i:s");
$connect_mysql->query('BEGIN');
$params = array('value'=>'50');
for($i=0;$i
$connect_mysql->insert($params);
if($i%100000==0){
$connect_mysql->query('COMMIT');
$connect_mysql->query('BEGIN');
}
}
$connect_mysql->query('COMMIT');
echo date("H:i:s");
第三种方法:使用优化SQL语句:将SQL语句进行拼接,使用 insert into table () values (),(),(),()然后再一次性插入,如果字符串太长,
则需要配置下MYSQL,在mysql 命令行中运行 :set global max_allowed_packet = 2*1024*1024*10;消耗时间为:11:24:06 11:25:06;
插入200W条测试数据仅仅用了1分钟!代码如下:
代码如下
$sql= "insert into twenty_million (value) values";
for($i=0;$i
$sql.="('50'),";
};
$sql = substr($sql,0,strlen($sql)-1);
$connect_mysql->query($sql);
最后总结下,,在插入大批量数据时,第一种方法无疑是最差劲的,而第二种方法在实际应用中就比较广泛,第三种方法在插入测试数据或者其他低要求时比较合适,速度确实快。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Apache connects to a database requires the following steps: Install the database driver. Configure the web.xml file to create a connection pool. Create a JDBC data source and specify the connection settings. Use the JDBC API to access the database from Java code, including getting connections, creating statements, binding parameters, executing queries or updates, and processing results.

MySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen

The main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.

The process of starting MySQL in Docker consists of the following steps: Pull the MySQL image to create and start the container, set the root user password, and map the port verification connection Create the database and the user grants all permissions to the database

Laravel is a PHP framework for easy building of web applications. It provides a range of powerful features including: Installation: Install the Laravel CLI globally with Composer and create applications in the project directory. Routing: Define the relationship between the URL and the handler in routes/web.php. View: Create a view in resources/views to render the application's interface. Database Integration: Provides out-of-the-box integration with databases such as MySQL and uses migration to create and modify tables. Model and Controller: The model represents the database entity and the controller processes HTTP requests.

I encountered a tricky problem when developing a small application: the need to quickly integrate a lightweight database operation library. After trying multiple libraries, I found that they either have too much functionality or are not very compatible. Eventually, I found minii/db, a simplified version based on Yii2 that solved my problem perfectly.

The key to installing MySQL elegantly is to add the official MySQL repository. The specific steps are as follows: Download the MySQL official GPG key to prevent phishing attacks. Add MySQL repository file: rpm -Uvh https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm Update yum repository cache: yum update installation MySQL: yum install mysql-server startup MySQL service: systemctl start mysqld set up booting

Installing MySQL on CentOS involves the following steps: Adding the appropriate MySQL yum source. Execute the yum install mysql-server command to install the MySQL server. Use the mysql_secure_installation command to make security settings, such as setting the root user password. Customize the MySQL configuration file as needed. Tune MySQL parameters and optimize databases for performance.
