Backend Development
Python Tutorial
Summary of usage of urllib2 module in Python network programming
Summary of usage of urllib2 module in Python network programming
1. The most basic application
import urllib2 url = r'http://www.baidu.com' html = urllib2.urlopen(url).read() print html
The client and the server communicate through request and response. The client first sends a request to the server, and then receives the response returned by the server
urllib2 provides the request class, which allows users to construct a request object before sending a request, and then send the request through the urllib2.urlopen method
import urllib2 url = r'http://www.baidu.com' req = urllib2.Request(url) html = urllib2.urlopen(req).read() print html
Used first in the above example
req = urllib2.Request(url)
Instantiate a request object, then use
urllib2.urlopen(req)
Let’s open this page.
We noticed that when instantiating the Request object, the url is required, and there are several default parameters
Data and header are also commonly used in the base. Some websites that require login to browse often require these two parameters
import urllib
import urllib2
url = 'http://www.baidu.com/'
values = {'name' : 'Michael Foord', 'location' : 'Northampton','language' : 'Python' }
data = urllib.urlencode(values)
req = urllib2.Request(url,data)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
This example is to send some data to Baidu. This example will return an error page, which is normal, because we do not need to post any information when visiting Baidu. However, an error will occur if we post it
Baidu will report an error if it cannot find the corresponding web page.
Of course this is POST data, it can also be used in the GET method, slightly modify the above code
Baidu queries through http://www.baidu.com/s?wd=XXX, so we need to urlencode the dictionary {'wd':'xxx'}
#coding:utf-8
import urllib
import urllib2
url = 'http://www.baidu.com/s'
values = {'wd':'杨彦星'}
data = urllib.urlencode(values)
print data
url2 = url+'?'+data
response = urllib2.urlopen(url2)
the_page = response.read()
print the_page
The following is an example of simulating login to Renren and then displaying the homepage content to explain the use of cookies in detail. The following is an example given in the document. We will modify this example to achieve the functions we want
import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
#coding:utf-8
import urllib2,urllib
import cookielib
url = r'http://www.renren.com/ajaxLogin'
#创建一个cj的cookie的容器
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
#将要POST出去的数据进行编码
data = urllib.urlencode({"email":email,"password":pass})
r = opener.open(url,data)
print cj
When you see cj, it means you have visited the login page. You can’t tell whether you have logged in normally. You can check it by visiting http://www.renren.com/home
There are two points to explain in the above code. It took me a long time to understand it
r = opener.open(url,data)
This sentence, why should we use the opener object to open instead of using utllib2 and urlopen? Not only is it written like this in the example, we write it this way. Through transformation, we can also use urllib2.urlopen. In fact, it is because opener is urllib2.bulid_opener Created, but you can understand it this way. After it was built, you did not install and use it, nor did it have its attributes and methods. If you want urllib2 to also have the attributes and methods of opener, you can first use urllib2.install_opener( opener) to "install" this opener. After installation, you can use urllib2 to operate it
#coding:utf-8
import urllib2,urllib
import cookielib
url = r'http://www.renren.com/ajaxLogin'
#创建一个cj的cookie的容器
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
#将要POST出去的数据进行编码
data = urllib.urlencode({"email":email,"password":pass})
#r = opener.open(url,data)如果没有上面的urllib2.install_opener方法，就必须这样写了
r = urllib2.urlopen(url,data)
html = urllib2.urlopen('http://www.renren.com/home').read()
print html
Similarly, urllib2 also has proxy-related handles. The basic idea is similar to this.
2. Exception handling
URLError exception will be raised when urlopen() cannot process the response. The HTTPError exception is a subclass of URLError and is only raised when accessing an HTTP type URL.
1. URLError exception
The usual reasons for URLError are: no network connection (no route to the target server), and the accessed target server does not exist. In this case, the exception object will have a reason attribute (which is a tuple of (error code, error reason)).
#! /usr/bin/env python #coding=utf-8 import urllib2 url="http://www.baidu.com/" try: response=urllib2.urlopen(url) except urllib2.URLError,e: print e.reason
2. HTTPError
Every HTTP response returned from the server has a status code. Among them, some status codes indicate that the server cannot complete the corresponding request. The default handler can handle some such status codes for us (if the returned response is a redirection, urllib2 will automatically obtain information from the redirected page for us) . There are some status codes that the urllib2 module cannot handle for us, so the urlopen function will cause an HTTPError exception, of which 404/401 is typical.
Instances of HTTPError exceptions have a code attribute of integer type, which represents the error status code returned by the server.
The default handler of the urllib2 module can handle redirects (status code is in the 300 range), and status codes in the 100-299 range indicate success. Therefore, the status code range that can cause HTTPError exceptions is: 400-599.
When an error occurs, the server returns an HTTP error code and error page. You can use the HTTPError instance as the return page, which means that the HTTPError instance not only has the code attribute, but also has methods such as read, geturl, and info.
#! /usr/bin/env python #coding=utf-8 import urllib2 url="http://cs.scu.edu.cn/~duanlei" try: response=urllib2.urlopen(url) except urllib2.HTTPError,e: print e.code print e.read()
3. Summary
If you want to handle URLError and HTTPError in code, there are two methods, the code is as follows:
#! /usr/bin/env python #coding=utf-8 import urllib2 url="xxxxxx" #需要访问的URL try: response=urllib2.urlopen(url) except urllib2.HTTPError,e: #HTTPError必须排在URLError的前面 print "The server couldn't fulfill the request" print "Error code:",e.code print "Return content:",e.read() except urllib2.URLError,e: print "Failed to reach the server" print "The reason:",e.reason else: #something you should do pass #其他异常的处理 #! /usr/bin/env python #coding=utf-8 import urllib2 url="http://xxx" #需要访问的URL try: response=urllib2.urlopen(url) except urllib2.URLError,e: if hasattr(e,"reason"): print "Failed to reach the server" print "The reason:",e.reason elif hasattr(e,"code"): print "The server couldn't fulfill the request" print "Error code:",e.code print "Return content:",e.read() else: pass #其他异常的处理
In comparison, the second exception handling method is better.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
PHP and Python: Different Paradigms Explained
Apr 18, 2025 am 12:26 AM
PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.
Choosing Between PHP and Python: A Guide
Apr 18, 2025 am 12:24 AM
PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.
Python vs. JavaScript: The Learning Curve and Ease of Use
Apr 16, 2025 am 12:12 AM
Python is more suitable for beginners, with a smooth learning curve and concise syntax; JavaScript is suitable for front-end development, with a steep learning curve and flexible syntax. 1. Python syntax is intuitive and suitable for data science and back-end development. 2. JavaScript is flexible and widely used in front-end and server-side programming.
PHP and Python: A Deep Dive into Their History
Apr 18, 2025 am 12:25 AM
PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.
Can vs code run in Windows 8
Apr 15, 2025 pm 07:24 PM
VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.
Can visual studio code be used in python
Apr 15, 2025 pm 08:18 PM
VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.
How to run programs in terminal vscode
Apr 15, 2025 pm 06:42 PM
In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.
Is the vscode extension malicious?
Apr 15, 2025 pm 07:57 PM
VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.
