Detailed introduction to Python crawler video crawling

This article brings you a detailed introduction to Python crawler crawling videos. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

I am writing an application recently and need to collect some popular videos on Weibo. These short videos generally come from Miaopai, Weipai, Meipai and Sina Video, and there is no download option, so I can only Use your brain to think of a solution.

The first step

Analyze the source code of the web page. For example: http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97, right-click to view the source code. Generally, videos have the mp4 suffix. If you search, you will not find it, but some can be directly viewed, such as Meipai videos.

Step 2

Capture the packet, analyze the request and return. This can also be achieved through the powerful chrome, or the above example, right-click->Inspect Element->NetWork, and then refresh the webpage with F5

I found that there are many requests, and I can only go one by one. After analysis, in fact, the video formats are mp4, flv, and avi. You can see it at once. Copy it to the browser and open it, and it is the download link we want.

The third step

Analyze the rules of download links and video links. That is, the relationship between http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97 and xxx.mp4. This again requires analyzing the web page source code. In fact, you can pay attention to the link with the .m3u8 suffix above. m3u8 records an index plain text file. When you open it, the playback software does not play it, but finds the corresponding audio and video based on its index. Use the network address of the file to play online, open it, and see that the download link we want is indeed recorded in it. Moreover, the link with the .m3u8 suffix is ​​in the source code of the web page.

Summary

After the analysis of the first three steps, the idea of ​​​​obtaining the video download link is to first obtain the .m3u8 suffix link from the web page source code, download the file, and Get the video download link inside, and finally download the video

Source code

#sinavideo.py
#coding=utf-8
import os
import re
import urllib2
import urllib 
from common import Common
class SinaVideo():

    URL_PIRFIX = "http://us.sinaimg.cn/"
    def getM3u8(self,html):
        reg = re.compile(r'list=([\s\S]*?)&fid')
        result = reg.findall(html)
        return result[0]


    def getName(self,url):
         return url.split('=')[1]

    def getSinavideoUrl(self,filepath):
        f = open(filepath,'r')
        lines = f.readlines()
        f.close()
        for line in lines:
            if line[0] !='#':
                return line

    def download(self,url,filepath):
        #获取名称
        name = self.getName(url)
        html = Common.getHtml(url)
        m3u8 = self.getM3u8(html)
        Common.download(urllib.unquote(m3u8),filepath,name + '.m3u8')
        url = self.URL_PIRFIX + self.getSinavideoUrl(filepath+name+'.m3u8')
        Common.download(url,filepath,name+'.mp4')

#common.py
#coding=utf-8
import urllib2
import os
import re


class Common():
    #  获取网页源码
    @staticmethod
    def getHtml(url):
        html = urllib2.urlopen(url).read()
        print  "[+]获取网页源码:"+url
        return html

    # 下载文件
    @staticmethod
    def download(url,filepath,filename):
        headers = {
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Charset': 'UTF-8,*;q=0.5',
            'Accept-Encoding': 'gzip,deflate,sdch',
            'Accept-Language': 'en-US,en;q=0.8',
            'User-Agent': 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36'
        }
        request = urllib2.Request(url,headers = headers);
        response = urllib2.urlopen(request)
        path = filepath + filename
        with open(path,'wb') as output:
            while True:
                buffer = response.read(1024*256);
                if not buffer:
                    break
                # received += len(buffer)
                output.write(buffer)

        print "[+]下载文件成功:"+path

    @staticmethod
    def isExist(filepath):
        return os.path.exists(filepath)

    @staticmethod
    def createDir(filepath):
         os.makedirs(filepath,0777)

Calling method:

 url = "http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97"sinavideo = SinaVideo()         
 sinavideo.download(url,""/Users/cheng/Documents/PyScript/res/"")

Result

The above is the detailed content of Detailed introduction to Python crawler video crawling. For more information, please follow other related articles on the PHP Chinese website!