How to get the correct number of applicants and viewers when crawling the 58.com work page?

58.com recruitment information crawling: Solve the problem of inconsistent data of applicants and viewers

When crawling the 58.com recruitment page, you often encounter a difficult problem: the number of applicants and the number of viewers displayed by the web page source code does not match the data actually displayed on the page, and the source code is often displayed as 0, while the data updated in real time on the page is consistent with the Elements content in the browser developer tool (F12). This article will explore how to solve this problem and obtain accurate applicants and viewers.

Problem analysis:

In order to prevent data from being maliciously crawled, 58.com adopted the method of dynamically loading data. The number of applicants and viewers on the page is not directly obtained from the HTML source code, but is loaded asynchronously through JavaScript. Therefore, direct parsing HTML source code cannot obtain the correct data.

Solution:

To obtain the correct number of applicants and viewers, you need to find the API interface provided by 58.com. By analyzing network requests, we can find an API interface for obtaining recruitment information statistics, with a URL similar to the following format:

 <code>https://statisticszp.58.com/position/totalcount/?infoId=27988...</code>

The infoId parameter represents the specific position ID and needs to be extracted based on the URL of the target recruitment page.

API returns data example:

The JSON data returned by the API interface contains the information we need:

 {
  "deliveryCount": 1141, // Number of applicants "commentCount": 0,
  "infoCount": 4, // Number of viewers "resumeReadPercent": 0,
  "referUrl": "",
  "nextUrl": "null"
}

The deliveryCount field indicates the number of applicants, and the infoCount field indicates the number of viewers.

Implementation steps:

1. Get Job ID (infoId): Analyze the URL of the target recruitment page and find the parameter value corresponding to the Job ID. This may require the use of regular expressions or other string processing methods.

2. Construct API request URL: Replace the extracted infoId into the API URL template to form a complete API request URL.

3. Send API requests: Use Python's requests library or other HTTP clients to send GET requests to the API URL.

4. Analyze JSON data: parse the JSON data returned by the API into a Python dictionary, extract the values ​​of deliveryCount and infoCount , that is, the correct number of applicants and number of viewers.

Through the above steps, you can bypass the dynamic loading mechanism of 58.com's web page and accurately obtain the number of applicants and viewers on the recruitment page. Please note that the address and parameter names of the API interface may change and need to be adjusted according to actual conditions. At the same time, please abide by 58.com's robots.txt rules to avoid excessive pressure on the server.

The above is the detailed content of How to get the correct number of applicants and viewers when crawling the 58.com work page?. For more information, please follow other related articles on the PHP Chinese website!