How to crawl website images in nodejs
Let me teach you how to use nodejs to crawl website images through an example. Friends who are interested can save it.
I will explain to you how nodejs implements the function of crawling website images through examples. The following is the full content:
Principle:
Crawler is the most obvious IO For intensive application scenarios, it is obvious to use node, which makes data mining with small I/O waiting overhead more convenient.
Use the express module to build the node service
And use the request module to obtain the html code of the target page
Download the cheerio module to process html code (cheerio has a syntax similar to jQuery, so it is easy to use and convenient)
Environment configuration:
npm install express request cheerio --save
(1)Introduce each module
var http = require('http'); var request = require('request); var cheerio = require('cheerio'); var fs = require('fs'); //用来操作文件 var url = 'https://movie.douban.com/cinema/nowplaying/beijing/' //定义要爬的页面
(2)Send a request
http.get(function(res){
var html = '';
var titles = [];
res.setEncoding('utf-8') //防止中文乱码
res.on('data',function(chunk){
html += chrunk; //监听data事件 每次取一块数据
})
res.on('end',function(){
var $ = cheerio.load(html); //获取数据完成后，解析html
//将获取的图片存到images文件夹中
$('.mod-bd img').each(function(index, item){
//获取图片属性
var imgName = $(this).parent().next().text().trimg()
var imgfile = imgName + '.jpeg';
var imgSrc = $(this).attr('src')
//采用request模块，向服务器发起请求 获取图片资源
request.head(imgSrc, function(error, res,body){
if(error){
console.log('失败了')
}
});
//通过管道的方式用fs模块将图片写到本地的images文件下
request(imgSrc).pipe.(fs.createWriteStream('./images/' + imgfile));
})
})
})The above is what I compiled for everyone, I hope it will be helpful to everyone in the future .
Related articles:
How to implement reassignment using js
How to save the image generated by canvas in js
How to implement two-way binding in js
Details introduction to the more practical functions of webpack
How to implement a menu using jQuery Add removal function
How to configure ueditor using nodejs mongodb vue
The above is the detailed content of How to crawl website images in nodejs. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
The difference between nodejs and vuejs
Apr 21, 2024 am 04:17 AM
Node.js is a server-side JavaScript runtime, while Vue.js is a client-side JavaScript framework for creating interactive user interfaces. Node.js is used for server-side development, such as back-end service API development and data processing, while Vue.js is used for client-side development, such as single-page applications and responsive user interfaces.
Is nodejs a backend framework?
Apr 21, 2024 am 05:09 AM
Node.js can be used as a backend framework as it offers features such as high performance, scalability, cross-platform support, rich ecosystem, and ease of development.
How to connect nodejs to mysql database
Apr 21, 2024 am 06:13 AM
To connect to a MySQL database, you need to follow these steps: Install the mysql2 driver. Use mysql2.createConnection() to create a connection object that contains the host address, port, username, password, and database name. Use connection.query() to perform queries. Finally use connection.end() to end the connection.
What are the global variables in nodejs
Apr 21, 2024 am 04:54 AM
The following global variables exist in Node.js: Global object: global Core module: process, console, require Runtime environment variables: __dirname, __filename, __line, __column Constants: undefined, null, NaN, Infinity, -Infinity
What is the difference between npm and npm.cmd files in the nodejs installation directory?
Apr 21, 2024 am 05:18 AM
There are two npm-related files in the Node.js installation directory: npm and npm.cmd. The differences are as follows: different extensions: npm is an executable file, and npm.cmd is a command window shortcut. Windows users: npm.cmd can be used from the command prompt, npm can only be run from the command line. Compatibility: npm.cmd is specific to Windows systems, npm is available cross-platform. Usage recommendations: Windows users use npm.cmd, other operating systems use npm.
Is there a big difference between nodejs and java?
Apr 21, 2024 am 06:12 AM
The main differences between Node.js and Java are design and features: Event-driven vs. thread-driven: Node.js is event-driven and Java is thread-driven. Single-threaded vs. multi-threaded: Node.js uses a single-threaded event loop, and Java uses a multi-threaded architecture. Runtime environment: Node.js runs on the V8 JavaScript engine, while Java runs on the JVM. Syntax: Node.js uses JavaScript syntax, while Java uses Java syntax. Purpose: Node.js is suitable for I/O-intensive tasks, while Java is suitable for large enterprise applications.
Is nodejs a back-end development language?
Apr 21, 2024 am 05:09 AM
Yes, Node.js is a backend development language. It is used for back-end development, including handling server-side business logic, managing database connections, and providing APIs.
Which one to choose between nodejs and java?
Apr 21, 2024 am 04:40 AM
Node.js and Java each have their pros and cons in web development, and the choice depends on project requirements. Node.js excels in real-time applications, rapid development, and microservices architecture, while Java excels in enterprise-grade support, performance, and security.
