Home Backend Development Python Tutorial Python uses Pandas to read CSV files and write them to MySQL

Python uses Pandas to read CSV files and write them to MySQL

Mar 24, 2017 pm 05:09 PM
mysql python

Summarize the various problems I encountered recently when using Python to read and write CSV to save the database.

Recommended related mysql video tutorials: "mysql tutorial"

Code:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

reload(sys)

sys.setdefaultencoding('utf-8')

host = '127.0.0.1'

port = 3306

db = 'world'

user = 'root'

password = '123456'

 

con = MySQLdb.connect(host=host,charset="utf8",port=port,db=db,user=user,passwd=password)

try:

    df = pd.read_sql(sql=r'select * from city', con=con)

    df.to_sql('test',con=con,flavor='mysql')

except Exception as e:

    print(e.message)

Copy after login

If nothing else happens, it will print Say something: database flavor MySQL is not supported
I found the answer on stackoverflow: The flavor 'mysql' is deprecated in pandas version 0.19.

Let's try another way:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

reload(sys)

sys.setdefaultencoding('utf-8')

host = '127.0.0.1'

port = 3306

db = 'world'

user = 'root'

password = '123456'

 

engine = create_engine(str(r"mysql+mysqldb://%s:" '%s' "@%s/%s") % (user, password, host, db))

 

try:

    df = pd.read_sql(sql=r'select * from city', con=engine)

    df.to_sql('test',con=engine,if_exists='append',index=False)

except Exception as e:

    print(e.message)

Copy after login

After running, ok, you can save the index parameter to indicate whether to store the index of the DataFrame as a column. Generally speaking, it is not needed, so the value is False

Now it seems that the problem has been solved, but there is still There is a small problem.
If I have a csv file containing Chinese (my Window):
name age class
Xiao Ming 15 first grade
Xiao Zhang 18 third grade

1

2

3

4

5

6

7

8

engine = create_engine(str(r"mysql+mysqldb://%s:" '%s' "@%s/%s") % (user, password, host, db))

 

try:

    df = pd.read_csv(r'C:\Users\xx\Desktop\data.csv')

    print(df)

    df.to_sql('test', con=engine, if_exists='append', index=False)

except Exception as e:

    print(e.message)

Copy after login

After printing, the characters are garbled . It is best to specify the encoding when we read csv. My local GBK:

1

df = pd.read_csv(r'C:\Users\xx\Desktop\data.csv',encoding='gbk')

Copy after login

We can print information normally, but an error is reported again. The error is as follows:

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)

It's still an encoding problem. The reason is that we didn't specify the encoding when we saved it to the database. I was also fooled when I was trying to solve this problem. Everything on the Internet is available. I won’t talk about the process, but look at the code:

1

engine = create_engine(str(r"mysql+mysqldb://%s:" '%s' "@%s/%s?charset=utf8") % (user, password, host, db))

Copy after login

Solved

Related articles:

Real IP request Pandas detailed explanation of Python data analysis

Detailed explanation of analyzing cdn logs through the pandas library in Python

Tutorial on using Python’s pandas framework to manipulate data in Excel files

The above is the detailed content of Python uses Pandas to read CSV files and write them to MySQL. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Nordhold: Fusion System, Explained
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Mandragora: Whispers Of The Witch Tree - How To Unlock The Grappling Hook
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Clair Obscur: Expedition 33 - How To Get Perfect Chroma Catalysts
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1677
14
PHP Tutorial
1278
29
C# Tutorial
1257
24
Python vs. JavaScript: Development Environments and Tools Python vs. JavaScript: Development Environments and Tools Apr 26, 2025 am 12:09 AM

Both Python and JavaScript's choices in development environments are important. 1) Python's development environment includes PyCharm, JupyterNotebook and Anaconda, which are suitable for data science and rapid prototyping. 2) The development environment of JavaScript includes Node.js, VSCode and Webpack, which are suitable for front-end and back-end development. Choosing the right tools according to project needs can improve development efficiency and project success rate.

Explain the purpose of foreign keys in MySQL. Explain the purpose of foreign keys in MySQL. Apr 25, 2025 am 12:17 AM

In MySQL, the function of foreign keys is to establish the relationship between tables and ensure the consistency and integrity of the data. Foreign keys maintain the effectiveness of data through reference integrity checks and cascading operations. Pay attention to performance optimization and avoid common errors when using them.

Compare and contrast MySQL and MariaDB. Compare and contrast MySQL and MariaDB. Apr 26, 2025 am 12:08 AM

The main difference between MySQL and MariaDB is performance, functionality and license: 1. MySQL is developed by Oracle, and MariaDB is its fork. 2. MariaDB may perform better in high load environments. 3.MariaDB provides more storage engines and functions. 4.MySQL adopts a dual license, and MariaDB is completely open source. The existing infrastructure, performance requirements, functional requirements and license costs should be taken into account when choosing.

MySQL: The Database, phpMyAdmin: The Management Interface MySQL: The Database, phpMyAdmin: The Management Interface Apr 29, 2025 am 12:44 AM

MySQL and phpMyAdmin can be effectively managed through the following steps: 1. Create and delete database: Just click in phpMyAdmin to complete. 2. Manage tables: You can create tables, modify structures, and add indexes. 3. Data operation: Supports inserting, updating, deleting data and executing SQL queries. 4. Import and export data: Supports SQL, CSV, XML and other formats. 5. Optimization and monitoring: Use the OPTIMIZETABLE command to optimize tables and use query analyzers and monitoring tools to solve performance problems.

The Future of Python and JavaScript: Trends and Predictions The Future of Python and JavaScript: Trends and Predictions Apr 27, 2025 am 12:21 AM

The future trends of Python and JavaScript include: 1. Python will consolidate its position in the fields of scientific computing and AI, 2. JavaScript will promote the development of web technology, 3. Cross-platform development will become a hot topic, and 4. Performance optimization will be the focus. Both will continue to expand application scenarios in their respective fields and make more breakthroughs in performance.

Steps to add and delete fields to MySQL tables Steps to add and delete fields to MySQL tables Apr 29, 2025 pm 04:15 PM

In MySQL, add fields using ALTERTABLEtable_nameADDCOLUMNnew_columnVARCHAR(255)AFTERexisting_column, delete fields using ALTERTABLEtable_nameDROPCOLUMNcolumn_to_drop. When adding fields, you need to specify a location to optimize query performance and data structure; before deleting fields, you need to confirm that the operation is irreversible; modifying table structure using online DDL, backup data, test environment, and low-load time periods is performance optimization and best practice.

Detailed explanation of the installation steps of MySQL on macOS system Detailed explanation of the installation steps of MySQL on macOS system Apr 29, 2025 pm 03:36 PM

Installing MySQL on macOS can be achieved through the following steps: 1. Install Homebrew, using the command /bin/bash-c"$(curl-fsSLhttps://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)". 2. Update Homebrew and use brewupdate. 3. Install MySQL and use brewinstallmysql. 4. Start MySQL service and use brewservicesstartmysql. After installation, you can use mysql-u

Quantitative Exchange Ranking 2025 Top 10 Recommendations for Digital Currency Quantitative Trading APPs Quantitative Exchange Ranking 2025 Top 10 Recommendations for Digital Currency Quantitative Trading APPs Apr 30, 2025 pm 07:24 PM

The built-in quantization tools on the exchange include: 1. Binance: Provides Binance Futures quantitative module, low handling fees, and supports AI-assisted transactions. 2. OKX (Ouyi): Supports multi-account management and intelligent order routing, and provides institutional-level risk control. The independent quantitative strategy platforms include: 3. 3Commas: drag-and-drop strategy generator, suitable for multi-platform hedging arbitrage. 4. Quadency: Professional-level algorithm strategy library, supporting customized risk thresholds. 5. Pionex: Built-in 16 preset strategy, low transaction fee. Vertical domain tools include: 6. Cryptohopper: cloud-based quantitative platform, supporting 150 technical indicators. 7. Bitsgap:

See all articles