python - 爬虫爬https站点怎么处理的？

Question

一段模拟登入知乎的问题
PS:这里不讨论验证码问题，我的账号登入没有提示验证码

代码信息：

# /usr/bin/python
#coding:utf-8
__author__ = 'eyu Fanne'

import requests,time
from bs4 import BeautifulSoup

begintime = time.time()



headers={
    "Host":"www.zhihu.com",
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36",
    "Accept":"*/*",
    "Accept-Language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3",
    "Accept-Encoding":"gzip, deflate",
    "Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
    "X-Requested-With":"XMLHttpRequest",
    "Connection":"keep-alive"
}


login_url=r'https://www.zhihu.com/login/email'

html_txt = requests.get(login_url,verify=False).text
html_soup = BeautifulSoup(html_txt,'lxml')
xsrf = html_soup.find("input",{"name":"_xsrf"})['value']

#print xsrf
#print html_txt
s = requests.session()

url_data={
    "_xsrf":xsrf,
    "email":"******",
    "password":"********",
    'remember_me':'true'
}

s_login=s.post(login_url,data=url_data,headers=headers,verify=False)
print s_login.text

endtime = time.time()
usetime =  endtime - begintime
print "执行脚本总用时 %s 秒" % usetime


#print s.get('https://www.zhihu.com',verify=False).text

执行结果：

C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
{"r":0,
 "msg": "\u767b\u9646\u6210\u529f"
}
执行脚本总用时 73.2650001049 秒

在模拟登入的时候，不知道是不是因为https的问题处理的好慢，整个模拟登入要73s才完成，不知卡在哪里了。
像类似这种问题的应该怎么处理。

Answer 1

from requests.packages.urllib3.exceptions import InsecureRequestWarning,InsecurePlatformWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
requests.packages.urllib3.disable_warnings(InsecurePlatformWarning)

可以解决告警问题，以及解决速度问题

Answer 2

可以用cProfile模块，先把所有过程封装到一个函数里，然后cProfile看看哪个函数调用的时间最长。

不过你这程序在我这里运行起来挺快的，应该和你的网络环境有关。