Data table migration data consistency verification
It is still very useful to do some necessary verification when migrating the database. For example, whether the number of data items before and after migration is consistent, and whether the data is consistent. What should I do at this time? It is easy to verify the number of items, but what if I verify whether the data is consistent? For important data, of course, every piece of data must be error-free. Random sampling verification is definitely not possible. If it is missed, it will be troublesome, and the two tables are no longer on the same server. What to do at this time? There is a way:
- Select a few important fields from the table, such as A, B, and C, and use these fields as a comparison ruler.
- Export these three fields of each piece of data from the original table to a file f1.
- Transfer these three fields of each piece of data from the destination table to file f2.
- Compare each piece of data in files f1 and f2 to see if they are the same.
- Draw a conclusion
The above method was thought up at the same time, and it is not bad, but I think there is room for improvement:
- First of all, not all fields may still appear different in non-main fields.
- The overall efficiency is relatively low
My thoughts are this:
- Splice every n pieces of data in the table (connect directly, the value of n depends on the size of each piece of data).
- Calculate the md5 values of these n pieces of data and add them to file f1 until all data values are obtained.
- The same goes for the destination table, which is recorded in file f2.
- Compare the md5 values of files f1 and f2. If they are consistent, ok, success.
- If they are inconsistent, compare each md5 value from top to bottom and find the mth inconsistent value.
- Concluded that the inconsistent data is between m*(n-1)+1 ~ m*n, and you can choose positioning again.
The advantage of the second method is that the output file will be reduced within a certain range, making it easy to compare. However, it also has disadvantages. It cannot directly locate the location of different data through key fields like the first method.
The following is the specific code implementation of the second method’s effect sum:
<?php
/**
* 使用方法：
* php -f mysql_diff.php yes dir 10
* 是否计算条数 是否计算输出d5并保存到文件 合并数据的级别
*
*/
if(php_sapi_name() != 'cli')
{
die("请在CLI模式下运行");
}
array_shift($argv);
if(empty($argv))
{
die("at letase contain one info");
}
$is_count = array_shift($argv);
$is_md5 = empty($argv) ? false : array_shift($argv);
$conbine_num = empty($argv) ? 1 : intval(array_shift($argv));
if($is_md5 && !is_dir($is_md5) && !mkdir($is_md5, 777, true))
{
die("error info : md5 info must be input to a file");
}
$dbinfos = array(
'host' => 'localhost',
'port' => '3306',
'user' => 'root',
'pswd' => '123456',
'charset' => 'utf8',
'tables' => array(
'lagou.pos',
'lagou.pos_innodb',
),
);
//验证格式
if(!$link = mysql_connect($dbinfos['host'].":".$dbinfos['port'],$dbinfos['user'], $dbinfos['pswd']))
{
die("connect to [{$host}@{$port}] failed!!");
}
if(!mysql_query("set names {$dbinfos['charset']}"))
{
die("set charset error : ".mysql_error());
}
foreach ($dbinfos['tables'] as $table)
{
if($is_count)
{
$sql = "select count(*) as nums from {$table}";
$ret = mysql_query($sql);
if(!$ret)
{
die("error : ".mysql_error());
}
$ret = mysql_fetch_array($ret, MYSQL_ASSOC);
echo "{$table} : {$ret['nums']}\n";
}
if($is_md5)
{
$path = $is_md5.DIRECTORY_SEPARATOR.$table;
$sql = "select * from {$table}";
$ret = mysql_query($sql);
$flag = 0;
$fields = '';
while ($_ret = mysql_fetch_array($ret, MYSQL_NUM)) {
$flag++;
while($_ret)
{
$fields .= array_pop($_ret);
}
if($flag % $conbine_num == 0)
{
file_put_contents($path, md5($fields)."\n", FILE_APPEND);
$fields = '';
}
}
if($flag % $conbine_num != 0 && $flag > 0)
{
file_put_contents($path, md5($fields)."\n", FILE_APPEND);
}
echo "save to file info : ".realpath($path)."\n";
}
}
The above introduces the data consistency verification of data table migration, including aspects of the content. I hope it will be helpful to friends who are interested in PHP tutorials.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
MySQL: An Introduction to the World's Most Popular Database
Apr 12, 2025 am 12:18 AM
MySQL is an open source relational database management system, mainly used to store and retrieve data quickly and reliably. Its working principle includes client requests, query resolution, execution of queries and return results. Examples of usage include creating tables, inserting and querying data, and advanced features such as JOIN operations. Common errors involve SQL syntax, data types, and permissions, and optimization suggestions include the use of indexes, optimized queries, and partitioning of tables.
MySQL's Place: Databases and Programming
Apr 13, 2025 am 12:18 AM
MySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen
Why Use MySQL? Benefits and Advantages
Apr 12, 2025 am 12:17 AM
MySQL is chosen for its performance, reliability, ease of use, and community support. 1.MySQL provides efficient data storage and retrieval functions, supporting multiple data types and advanced query operations. 2. Adopt client-server architecture and multiple storage engines to support transaction and query optimization. 3. Easy to use, supports a variety of operating systems and programming languages. 4. Have strong community support and provide rich resources and solutions.
How to connect to the database of apache
Apr 13, 2025 pm 01:03 PM
Apache connects to a database requires the following steps: Install the database driver. Configure the web.xml file to create a connection pool. Create a JDBC data source and specify the connection settings. Use the JDBC API to access the database from Java code, including getting connections, creating statements, binding parameters, executing queries or updates, and processing results.
How to start mysql by docker
Apr 15, 2025 pm 12:09 PM
The process of starting MySQL in Docker consists of the following steps: Pull the MySQL image to create and start the container, set the root user password, and map the port verification connection Create the database and the user grants all permissions to the database
MySQL's Role: Databases in Web Applications
Apr 17, 2025 am 12:23 AM
The main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.
Laravel Introduction Example
Apr 18, 2025 pm 12:45 PM
Laravel is a PHP framework for easy building of web applications. It provides a range of powerful features including: Installation: Install the Laravel CLI globally with Composer and create applications in the project directory. Routing: Define the relationship between the URL and the handler in routes/web.php. View: Create a view in resources/views to render the application's interface. Database Integration: Provides out-of-the-box integration with databases such as MySQL and uses migration to create and modify tables. Model and Controller: The model represents the database entity and the controller processes HTTP requests.
How to install mysql in centos7
Apr 14, 2025 pm 08:30 PM
The key to installing MySQL elegantly is to add the official MySQL repository. The specific steps are as follows: Download the MySQL official GPG key to prevent phishing attacks. Add MySQL repository file: rpm -Uvh https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm Update yum repository cache: yum update installation MySQL: yum install mysql-server startup MySQL service: systemctl start mysqld set up booting
