pythoy超级超级超级简单的一个爬虫小程序_html/css_WEB-ITnose
首次接触爬虫，这个程序可以让初学者明白爬虫是啥，爬虫的基本逻辑。（参照网易云课堂heibanke老师的程序编写的。hebanke老师的课程讲得很清楚，感兴趣的选手可以去看看）
被爬虫的网址是http://www.heibanke.com/lesson/crawler_ex00/需要完成的任务是在网址后面输入显示的数字转入下一个网址，再次读取显示的数字，转入下一个网址，以此循环
爬虫要求
在爬虫的时候需要需要用到两个库，一个urllib，一个BeautifulSoup,urllib为python自带的库，beautifulsoup需要自己安装
pip install beautifulsoup4
安装成功后导入
import urllibfrom bs4 import BeautifulSoup
urllib提供了一系列用于操作URL的功能打开要爬取得网站：
urllib.urlopen(url)
beautifulsoup主要用于解析HTML，将 HTML转化成数型结构，每个节点都是Python对象。所有对象可以归纳为4种:1.Tag2.NavigableString3.BeautifulSoup4.Comment如果一个HTML为
<html> <[head])> <title> The Dormouse's story </title>
print soup.title>>><title>The Dormouse's story</title>
print soup.title.string>>>The Dormouse's story
程序逻辑：读出页面上的数字，将数字加在url后面，继续读取新页面的数字，直到进入最后一个没有数字的页面，break程序如下：
import urllibfrom bs4 import BeautifulSoupimport re#这个库导入是为了使用正则表达式读取读取找到的内容中的数字url='http://www.heibanke.com/lesson/crawler_ex00/'number=['']#用于储存读到的数字while True: content = urllib.urlopen(url+number[0])#number为字符串，number[0]为数字 bs_obj = BeautifulSoup(content,"html.parser")#html.parser表示解析网站，不返回任何值 number = bs_obj.h3.string#网页显示出的“你需要在网址后输入数字44513”在html的h3 tag中，number在这里读出了h3里面的内容 number= re.findall(r'\d+',number)#读出了number里面的数字 if not number:#必须判断页面中还有是否还有number，没有说明已经到了最后一个页面，这时应该跳出循环，打印 bs_obj.h3.string break else: print number[0]print bs_obj.h3.string
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Is HTML easy to learn for beginners?
Apr 07, 2025 am 12:11 AM
HTML is suitable for beginners because it is simple and easy to learn and can quickly see results. 1) The learning curve of HTML is smooth and easy to get started. 2) Just master the basic tags to start creating web pages. 3) High flexibility and can be used in combination with CSS and JavaScript. 4) Rich learning resources and modern tools support the learning process.
The Roles of HTML, CSS, and JavaScript: Core Responsibilities
Apr 08, 2025 pm 07:05 PM
HTML defines the web structure, CSS is responsible for style and layout, and JavaScript gives dynamic interaction. The three perform their duties in web development and jointly build a colorful website.
Understanding HTML, CSS, and JavaScript: A Beginner's Guide
Apr 12, 2025 am 12:02 AM
WebdevelopmentreliesonHTML,CSS,andJavaScript:1)HTMLstructurescontent,2)CSSstylesit,and3)JavaScriptaddsinteractivity,formingthebasisofmodernwebexperiences.
What is an example of a starting tag in HTML?
Apr 06, 2025 am 12:04 AM
AnexampleofastartingtaginHTMLis,whichbeginsaparagraph.StartingtagsareessentialinHTMLastheyinitiateelements,definetheirtypes,andarecrucialforstructuringwebpagesandconstructingtheDOM.
Gitee Pages static website deployment failed: How to troubleshoot and resolve single file 404 errors?
Apr 04, 2025 pm 11:54 PM
GiteePages static website deployment failed: 404 error troubleshooting and resolution when using Gitee...
How to use CSS3 and JavaScript to achieve the effect of scattering and enlarging the surrounding pictures after clicking?
Apr 05, 2025 am 06:15 AM
To achieve the effect of scattering and enlarging the surrounding images after clicking on the image, many web designs need to achieve an interactive effect: click on a certain image to make the surrounding...
How to implement adaptive layout of Y-axis position in web annotation?
Apr 04, 2025 pm 11:30 PM
The Y-axis position adaptive algorithm for web annotation function This article will explore how to implement annotation functions similar to Word documents, especially how to deal with the interval between annotations...
HTML, CSS, and JavaScript: Essential Tools for Web Developers
Apr 09, 2025 am 12:12 AM
HTML, CSS and JavaScript are the three pillars of web development. 1. HTML defines the web page structure and uses tags such as, etc. 2. CSS controls the web page style, using selectors and attributes such as color, font-size, etc. 3. JavaScript realizes dynamic effects and interaction, through event monitoring and DOM operations.
