nodejs crawler framework superagent

This time I will bring you the nodejs crawler framework superagent. What are the precautions of the nodejs crawler framework superagent. Here are the actual cases, let’s take a look.

Preface

I have heard of crawlers for a long time. I started to learn nodejs in the past few days and wrote a crawlerhttps://github.com/leichangchun/node-crawlers/tree/master/superagent_cheerio_demo, crawl the article title, user name, number of reads, number of recommendations and user avatar on the homepage of the blog park, and now make a short summary.

Use these points:

1. The core module of node - File system

2. Third party used for http requests Module -- superagent

3. Third-party module for parsing DOM -- cheerio

For detailed explanations and APIs of several modules, please refer to each link. There are only simple usages in the demo.

Preparation work

Use npm to manage dependencies, and dependency information will be stored in package.json

Introducing the required functional modules

Request + Parse page

If you want to climb to the content of the blog park homepage, First, you need to request the homepage address and get the returned html. Here, superagent is used to make an http request. The basic usage method is as follows:

Initiate a get request to the specified url. When the request is incorrect, an error will be returned (no In case of error, error is null or undefined), and res is the returned data.

After getting the html content, we need to use cheerio to parse the DOM in order to get the data we want. Cheerio must first load the target html and then parse it. The API is very similar to the jquery API. , familiar with jquery and getting started very quickly. Look directly at the code example

Storing data

After parsing the DOM above, the required information content has been spliced ​​and the image has been obtained URL, store it now, store the content in a txt file in the specified directory, and download the image to the specified directory

Create the directory first and use the nodejs core file system

With the specification After the directory, you can write data. The content of the txt file is already there. Just write it directly. Use writeFile()

to get the link to the picture, so you need to use superagent to download the picture and save it locally. . superagent can directly return a response stream, and then cooperate with the nodejs pipeline to directly write the image content to the local

Effect

Execute the demo and see the effect. The data has climbed down normally

A very simple demo, it may not be that rigorous, but it is always the first small step towards node.

I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the php Chinese website!

Recommended reading:

H5’s semantic tags

H5’s method of reading files and uploading them to the server

How to achieve the animation effect of picture rotation in html5

The above is the detailed content of nodejs crawler framework superagent. For more information, please follow other related articles on the PHP Chinese website!