Write a Python program to crawl the fund flow of sectors

Through the above example of crawling the capital flow of individual stocks, you should be able to learn to write your own crawling code. Now consolidate it and do a similar small exercise. You need to write your own Python program to crawl the fund flow of online sectors. The crawled URL is http://data.eastmoney.com/bkzj/hy.html, and the display interface is shown in Figure 1.

# 图 1 The Faculty Stream URL interface

1, find js

# Directly press the F12 key, open the development commissioning tool and find the data The corresponding web page is shown in Figure 2.

Figure 2 Find the web page corresponding to JS

and then enter the URL into the browser. The URL is relatively long.

http://push2.eastmoney.com/api/qt/clist/get?cb=jQuery112309073354919152763_1617455258434&pn=1&pz=500&po=1&np=1&fields=f12,f13,f14,f62&fid=f62&fs=m:9 0+ t:2&ut=b2884a393a59ad64002292a3e90d46a5&_=1617455258435

At this time, you will get feedback from the website, as shown in Figure 3.

Figure 3 Obtaining sections and capital flows from the website

The content corresponding to this URL is the content we want to crawl.

2, request request and response response status

Write the crawler code, see the following code for details:

# coding=utf-8
import requests
url=" http://push2.eastmoney.com/api/qt/clist/get?cb=jQuery112309073354919152763_
1617455258436&fid=f62&po=1&pz=50&pn=1&np=1&fltt=2&invt=2&ut=b2884a393a59ad64002292a3
e90d46a5&fs=m%3A90+t%3A2&fields=f12%2Cf14%2Cf2%2Cf3%2Cf62%2Cf184%2Cf66%2Cf69%2Cf72%2
Cf75%2Cf78%2Cf81%2Cf84%2Cf87%2Cf204%2Cf205%2Cf124"
r = requests.get(url)

r.status_code displays 200, indicating that the response status is normal. r.text also has data, indicating that crawling the capital flow data is successful, as shown in Figure 4.

Figure 4 response status

3, clean str into JSON standard format

(1) Analyze r.text data. Its internal format is standard JSON, but with some extra prefixes in front. Remove the jQ prefix and use the split() function to complete this operation. See the following code for details:

r_text=r.text.split("{}".format("jQuery112309073354919152763_1617455258436"))[1]
r_text

The running results are shown in Figure 5.

##                                                                                                                                                                                                                                         . See the following code for details:

r_text_qu=r_text.rstrip(';')
r_text_json=json.loads(r_text_qu[1:-1])['data']['diff']
dfcf_code={"f12":"code","f2":"价格","f3":"涨幅","f14":"name","f62":"主净入√","f66":"超净入","f69":"超占比", "f72":"大净入","f75":"大占比","f78":"中净入","f81":"中占比","f84":"小净入","f87":"小占比","f124":"不知道","f184":"主占比√"}
result_=pd.DataFrame(r_text_json).rename(columns=dfcf_code)
result_["主净入√"]=round(result_["主净入√"]/100000000,2)#一亿，保留2位
result_=result_[result_["主净入√"]>0]
result_["超净入"]=round(result_["超净入"]/100000000,2)#一亿，保留2位
result_["大净入"]=round(result_["大净入"]/100000000,2)#一亿，保留2位
result_["中净入"]=round(result_["中净入"]/100000000,2)#一亿，保留2位
result_["小净入"]=round(result_["小净入"]/100000000,2)#一亿，保留2位
result_

The running results are shown in Figure 6.

##                                                                                                                                                                                                                                     Save it locally, as shown in Figure 7.

Through the above two examples of fund crawling, you must have understood some of the methods of using crawlers. The core idea is:

(1) Select the advantages of capital flows of individual stocks;

(2) Obtain the URL and analyze it;

(3) Use crawlers to collect data Get and save data.

Figure 6 Data Saving

Summary

JSON format data is one of the standardized data formats used by many websites. The lightweight data exchange format is very easy to read and write, and can effectively improve network transmission efficiency. The first thing to crawl is the string in str format. Through data processing and processing, it is turned into standard JSON format, and then into Pandas format.

Through case analysis and actual combat, we must learn to write our own code to crawl financial data and have the ability to convert it into JSON standard format. Complete daily data crawling and data storage work to provide effective data support for future historical testing and historical analysis of data.

Of course, capable readers can save the results to databases such as MySQL, MongoDB, or even the cloud database Mongo Atlas. The author will not focus on the explanation here. We focus entirely on the study of quantitative learning and strategy. Using txt format to save data can completely solve the problem of early data storage, and the data is also complete and effective.

The above is the detailed content of Write a Python program to crawl the fund flow of sectors. For more information, please follow other related articles on the PHP Chinese website!