nodejs implements the function of crawling website images_node.js
I’ll give you an example to teach you how to use nodejs to implement the pictures function of crawling websites. Friends who are interested can save it.
I will explain to you how nodejs implements the function of crawling website images through examples. The following is the full content:
Principle:
Crawler is the most powerful For obvious IO-intensive application scenarios, it is obvious to use node, which makes data mining with small I/O waiting overhead more convenient.
Use the express module to build node services
and use the request module to obtain the target page html code
Download the cheerio module to process the html code (cheerio has a syntax similar to jQuery, so it is easy to use and convenient)
Environment configuration:
npm install express request cheerio --save
(1)Introduce each module
var http = require('http'); var request = require('request); var cheerio = require('cheerio'); var fs = require('fs'); //用来操作文件 var url = 'https://movie.douban.com/cinema/nowplaying/beijing/' //定义要爬的页面
(2) Send a request
http.get(function(res){
var html = '';
var titles = [];
res.setEncoding('utf-8') //防止中文乱码
res.on('data',function(chunk){
html += chrunk; //监听data事件 每次取一块数据
})
res.on('end',function(){
var $ = cheerio.load(html); //获取数据完成后，解析html
//将获取的图片存到images文件夹中
$('.mod-bd img').each(function(index, item){
//获取图片属性
var imgName = $(this).parent().next().text().trimg()
var imgfile = imgName + '.jpeg';
var imgSrc = $(this).attr('src')
//采用request模块，向服务器发起请求 获取图片资源
request.head(imgSrc, function(error, res,body){
if(error){
console.log('失败了')
}
});
//通过管道的方式用fs模块将图片写到本地的images文件下
request(imgSrc).pipe.(fs.createWriteStream('./images/' + imgfile));
})
})
})The above is all the content of this article, I hope it will be helpful to my friends! !
Related recommendations:
NodeJS crawler instance encyclopedia of embarrassing things_node.js
NodeJs’s solution to database exception handling
How to use nodejs to implement chat function
The above is the detailed content of nodejs implements the function of crawling website images_node.js. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Is nodejs a backend framework?
Apr 21, 2024 am 05:09 AM
Node.js can be used as a backend framework as it offers features such as high performance, scalability, cross-platform support, rich ecosystem, and ease of development.
How to connect nodejs to mysql database
Apr 21, 2024 am 06:13 AM
To connect to a MySQL database, you need to follow these steps: Install the mysql2 driver. Use mysql2.createConnection() to create a connection object that contains the host address, port, username, password, and database name. Use connection.query() to perform queries. Finally use connection.end() to end the connection.
What are the global variables in nodejs
Apr 21, 2024 am 04:54 AM
The following global variables exist in Node.js: Global object: global Core module: process, console, require Runtime environment variables: __dirname, __filename, __line, __column Constants: undefined, null, NaN, Infinity, -Infinity
What is the difference between npm and npm.cmd files in the nodejs installation directory?
Apr 21, 2024 am 05:18 AM
There are two npm-related files in the Node.js installation directory: npm and npm.cmd. The differences are as follows: different extensions: npm is an executable file, and npm.cmd is a command window shortcut. Windows users: npm.cmd can be used from the command prompt, npm can only be run from the command line. Compatibility: npm.cmd is specific to Windows systems, npm is available cross-platform. Usage recommendations: Windows users use npm.cmd, other operating systems use npm.
Is there a big difference between nodejs and java?
Apr 21, 2024 am 06:12 AM
The main differences between Node.js and Java are design and features: Event-driven vs. thread-driven: Node.js is event-driven and Java is thread-driven. Single-threaded vs. multi-threaded: Node.js uses a single-threaded event loop, and Java uses a multi-threaded architecture. Runtime environment: Node.js runs on the V8 JavaScript engine, while Java runs on the JVM. Syntax: Node.js uses JavaScript syntax, while Java uses Java syntax. Purpose: Node.js is suitable for I/O-intensive tasks, while Java is suitable for large enterprise applications.
Is nodejs a back-end development language?
Apr 21, 2024 am 05:09 AM
Yes, Node.js is a backend development language. It is used for back-end development, including handling server-side business logic, managing database connections, and providing APIs.
Which one to choose between nodejs and java?
Apr 21, 2024 am 04:40 AM
Node.js and Java each have their pros and cons in web development, and the choice depends on project requirements. Node.js excels in real-time applications, rapid development, and microservices architecture, while Java excels in enterprise-grade support, performance, and security.
Can nodejs write front-end?
Apr 21, 2024 am 05:00 AM
Yes, Node.js can be used for front-end development, and key advantages include high performance, rich ecosystem, and cross-platform compatibility. Considerations to consider are learning curve, tool support, and small community size.
