What packages should be installed for python crawler?
What libraries need to be installed to learn Python web crawler?
The following are the relevant libraries involved in Python crawlers
Request library, parsing library, storage library, tool library
1. Request library: urllib/re/requests
(1) urllib/re is the library that comes with python by default. It can be verified by the following command:
No error message is output, indicating that the environment is normal
(2) requests installation
2.1 Open CMD, enter pip3 install requests
2.2 Wait for installation , Verification
(3) selenium installation (driving the browser for website access behavior)
3.1 Open CMD and enter pip3 install selenium
3.2 Install chromedriver
Website: https://npm.taobao.org/
Extract the downloaded compressed package and put the exe into D:\Python3.6.0 \Scripts\
This path only needs to be in the PATH variable
3.3 Wait for the installation to be completed, verify
and press Enter to pop up chrome browser interface
3.4 Install other browsers
Interfaceless browser phantomjs
Download URL: http://phantomjs.org/
Download After completion, unzip it, put the entire directory into D:\Python3.6.0\Scripts\, and add the path to the bin directory to the PATH variable
Verification:
Open CMD
2. Parsing library:
2.1 lxml (XPATH)
Open CMD and enter pip3 install lxml or download from https://pypi.python.org, For example, lxml-4.1.1-cp36-cp36m-win_amd64.whl (md5), first download the whl file, and execute pip3 install on the command line. File name.whl
2.2 beautifulsoup
To open CMD, you need to install lxml first
pip3 install beautifulsoup4
2.3 pyquery (similar to jquery syntax)
Open CMD , pip3 install pyquery
Verify the installation results
3. Repository
3.1 pymysql (operating MySQL, relational database)
Installation: pip3 install pymysql, test after installation:
3.2 pymongo (operating MongoDB, key-value)
Installation pip3 install pymongo
Verification
The above is the detailed content of What packages should be installed for python crawler?. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1666
14
1425
52
1325
25
1272
29
1252
24
Python vs. C : Applications and Use Cases Compared
Apr 12, 2025 am 12:01 AM
Python is suitable for data science, web development and automation tasks, while C is suitable for system programming, game development and embedded systems. Python is known for its simplicity and powerful ecosystem, while C is known for its high performance and underlying control capabilities.
Python: Games, GUIs, and More
Apr 13, 2025 am 12:14 AM
Python excels in gaming and GUI development. 1) Game development uses Pygame, providing drawing, audio and other functions, which are suitable for creating 2D games. 2) GUI development can choose Tkinter or PyQt. Tkinter is simple and easy to use, PyQt has rich functions and is suitable for professional development.
Python vs. C : Learning Curves and Ease of Use
Apr 19, 2025 am 12:20 AM
Python is easier to learn and use, while C is more powerful but complex. 1. Python syntax is concise and suitable for beginners. Dynamic typing and automatic memory management make it easy to use, but may cause runtime errors. 2.C provides low-level control and advanced features, suitable for high-performance applications, but has a high learning threshold and requires manual memory and type safety management.
Python and Time: Making the Most of Your Study Time
Apr 14, 2025 am 12:02 AM
To maximize the efficiency of learning Python in a limited time, you can use Python's datetime, time, and schedule modules. 1. The datetime module is used to record and plan learning time. 2. The time module helps to set study and rest time. 3. The schedule module automatically arranges weekly learning tasks.
Python vs. C : Exploring Performance and Efficiency
Apr 18, 2025 am 12:20 AM
Python is better than C in development efficiency, but C is higher in execution performance. 1. Python's concise syntax and rich libraries improve development efficiency. 2.C's compilation-type characteristics and hardware control improve execution performance. When making a choice, you need to weigh the development speed and execution efficiency based on project needs.
Which is part of the Python standard library: lists or arrays?
Apr 27, 2025 am 12:03 AM
Pythonlistsarepartofthestandardlibrary,whilearraysarenot.Listsarebuilt-in,versatile,andusedforstoringcollections,whereasarraysareprovidedbythearraymoduleandlesscommonlyusedduetolimitedfunctionality.
Learning Python: Is 2 Hours of Daily Study Sufficient?
Apr 18, 2025 am 12:22 AM
Is it enough to learn Python for two hours a day? It depends on your goals and learning methods. 1) Develop a clear learning plan, 2) Select appropriate learning resources and methods, 3) Practice and review and consolidate hands-on practice and review and consolidate, and you can gradually master the basic knowledge and advanced functions of Python during this period.
Python: Automation, Scripting, and Task Management
Apr 16, 2025 am 12:14 AM
Python excels in automation, scripting, and task management. 1) Automation: File backup is realized through standard libraries such as os and shutil. 2) Script writing: Use the psutil library to monitor system resources. 3) Task management: Use the schedule library to schedule tasks. Python's ease of use and rich library support makes it the preferred tool in these areas.
