PHP learning CURL crawler example
Many times we need to crawl some website resources in batches, and at this time we need to use crawlers. The basis of the crawler is to use CURL to simulate HTTP requests and then parse the data. This article will lead you to learn PHP's CURL by writing a simple web crawler.
First introduce some commonly used functions.
curl_init 初始化一个curl对话 curl_setopt 设置curl参数，即传输选项 curl_exec 执行请求 curl_close 关闭一个curl对话
Mainly the above four
curl_errno 返回最后一次错误码,php已经定义了诸多错误枚举编码 curl_errror 返回一个保护当前会话最近一次错误的字符串
Let’s go directly to the examples. The relevant explanations are in the comments
1. Download a webpage on the Internet and replace "Baidu" in the content with "Diaosi" and then output
<?php
/**
* 实例描述：在网络上下载一个网页并把内容中的“百度”替换为“屌丝”之后输出
*/
$curlobj = curl_init(); // 初始化
curl_setopt($curlobj, CURLOPT_URL, "http://www.baidu.com"); // 设置访问网页的URL
curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, true); // 执行之后不直接打印出来
$output=curl_exec($curlobj); // 执行
curl_close($curlobj); // 关闭cURL
echo str_replace("百度","屌丝",$output);
?>2. Query by calling WebService The current weather in Beijing
<?php
/**
* 实例描述：通过调用WebService查询北京的当前天气
*/
$data = 'theCityName=北京';
$curlobj = curl_init();
curl_setopt($curlobj, CURLOPT_URL, "http://www.webxml.com.cn/WebServices/WeatherWebService.asmx/getWeatherbyCityName");
curl_setopt($curlobj, CURLOPT_HEADER, 0);
curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curlobj, CURLOPT_POST, 1);
curl_setopt($curlobj, CURLOPT_POSTFIELDS, $data);
curl_setopt($curlobj, CURLOPT_HTTPHEADER, array("application/x-www-form-urlencoded; charset=utf-8",
"Content-length: ".strlen($data)
));
$rtn = curl_exec($curlobj);
if(!curl_errno($curlobj)){
// $info = curl_getinfo($curlobj);
// print_r($info);
echo $rtn;
} else {
echo 'Curl error: ' . curl_error($curlobj);
}
curl_close($curlobj);
?>3. Simulate the URL that requires login and capture the content of the webpage
<?php
/**
* 实例描述：模拟需要登录的网址并抓取网页的内容
*/
$data=array('username' => 'promonkey',
'password' => '1q2w3e',
'remember'=>1);
$data='username=zjzhoufy@126.com&password=1q2w3e&remember=1';
$curlobj = curl_init(); // 初始化
curl_setopt($curlobj, CURLOPT_URL, "http://www.imooc.com/user/login"); // 设置访问网页的URL
curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, true); // 执行之后不直接打印出来
// Cookie相关设置，这部分设置需要在所有会话开始之前设置
date_default_timezone_set('PRC'); // 使用Cookie时，必须先设置时区
curl_setopt($curlobj, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($curlobj, CURLOPT_HEADER, 0);
curl_setopt($curlobj, CURLOPT_FOLLOWLOCATION, 1); // 这样能够让cURL支持页面链接跳转
curl_setopt($curlobj, CURLOPT_POST, 1);
curl_setopt($curlobj, CURLOPT_POSTFIELDS, $data);
curl_setopt($curlobj, CURLOPT_HTTPHEADER, array("application/x-www-form-urlencoded; charset=utf-8",
"Content-length: ".strlen($data)
));
curl_exec($curlobj); // 执行
curl_setopt($curlobj, CURLOPT_URL, "http://www.imooc.com/space/index");
curl_setopt($curlobj, CURLOPT_POST, 0);
curl_setopt($curlobj, CURLOPT_HTTPHEADER, array("Content-type: text/xml"
));
$output=curl_exec($curlobj); // 执行
curl_close($curlobj); // 关闭cURL
echo $output;
?>4 . Capture the information of the logged in website and download the personal space page + customize the implementation of page link jump capture
<?php
/**
* 实例描述：登录网站的信息抓取并下载个人空间页面
* 自定义实现页面链接跳转抓取
*
*/
$data='username=demo_peter@126.com&password=123qwe&remember=1';
$curlobj = curl_init(); // 初始化
curl_setopt($curlobj, CURLOPT_URL, "http://www.imooc.com/user/login"); // 设置访问网页的URL
curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, true); // 执行之后不直接打印出来
// Cookie相关设置，这部分设置需要在所有会话开始之前设置
date_default_timezone_set('PRC'); // 使用Cookie时，必须先设置时区
curl_setopt($curlobj, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($curlobj, CURLOPT_HEADER, 0);
// 注释掉这行，因为这个设置必须关闭安全模式 以及关闭open_basedir，对服务器安全不利
//curl_setopt($curlobj, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curlobj, CURLOPT_POST, 1);
curl_setopt($curlobj, CURLOPT_POSTFIELDS, $data);
curl_setopt($curlobj, CURLOPT_HTTPHEADER, array("application/x-www-form-urlencoded; charset=utf-8",
"Content-length: ".strlen($data)
));
curl_exec($curlobj); // 执行
curl_setopt($curlobj, CURLOPT_URL, "http://www.imooc.com/space/index");
curl_setopt($curlobj, CURLOPT_POST, 0);
curl_setopt($curlobj, CURLOPT_HTTPHEADER, array("Content-type: text/xml"
));
$output=curl_redir_exec($curlobj); // 执行
curl_close($curlobj); // 关闭cURL
echo $output;
/**
* 自定义实现页面链接跳转抓取
*/
function curl_redir_exec($ch,$debug="")
{
static $curl_loops = 0;
static $curl_max_loops = 20;
if ($curl_loops++ >= $curl_max_loops)
{
$curl_loops = 0;
return FALSE;
}
curl_setopt($ch, CURLOPT_HEADER, true); // 开启header才能够抓取到重定向到的新URL
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
// 分割返回的内容
$h_len = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($data,0,$h_len);
$data = substr($data,$h_len - 1);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($http_code == 301 || $http_code == 302) {
$matches = array();
preg_match('/Location:(.*?)\n/', $header, $matches);
$url = @parse_url(trim(array_pop($matches)));
// print_r($url);
if (!$url)
{
//couldn't process the url to redirect to
$curl_loops = 0;
return $data;
}
$last_url = parse_url(curl_getinfo($ch, CURLINFO_EFFECTIVE_URL));
if (!isset($url['scheme']))
$url['scheme'] = $last_url['scheme'];
if (!isset($url['host']))
$url['host'] = $last_url['host'];
if (!isset($url['path']))
$url['path'] = $last_url['path'];
$new_url = $url['scheme'] . '://' . $url['host'] . $url['path'] . (isset($url['query'])?'?'.$url['query']:'');
curl_setopt($ch, CURLOPT_URL, $new_url);
return curl_redir_exec($ch);
} else {
$curl_loops=0;
return $data;
}
}
?>Download a file from the FTP server to the local
<?php
/**
* 实例描述：把本地文件上传到FTP服务器上
*/
$curlobj = curl_init();
$localfile = 'ftp01.php';
$fp = fopen($localfile, 'r');
curl_setopt($curlobj, CURLOPT_URL, "ftp://192.168.1.100/ftp01_uploaded.php");
curl_setopt($curlobj, CURLOPT_HEADER, 0);
curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curlobj, CURLOPT_TIMEOUT, 300); // times out after 300s
curl_setopt($curlobj, CURLOPT_USERPWD, "peter.zhou:123456");//FTP用户名：密码
//上传和下载主要是下面子三个参数不一样
curl_setopt($curlobj, CURLOPT_UPLOAD, 1);
curl_setopt($curlobj, CURLOPT_INFILE, $fp);
curl_setopt($curlobj, CURLOPT_INFILESIZE, filesize($localfile));
$rtn = curl_exec($curlobj);
fclose($fp);
if(!curl_errno($curlobj)){
echo "Uploaded successfully.";
} else {
echo 'Curl error: ' . curl_error($curlobj);
}
curl_close($curlobj);
?>6. Download an HTTPS resource on the network
<?php /** * 实例描述：下载网络上面的一个HTTPS的资源 */ $curlobj = curl_init(); // 初始化 curl_setopt($curlobj, CURLOPT_URL, "https://ajax.aspnetcdn.com/ajax/jquery.validate/1.12.0/jquery.validate.js"); // 设置访问网页的URL curl_setopt($curlobj, CURLOPT_RETURNTRANSFER, true); // 执行之后不直接打印出来 // 设置HTTPS支持 date_default_timezone_set('PRC'); // 使用Cookie时，必须先设置时区 curl_setopt($curlobj, CURLOPT_SSL_VERIFYPEER, 0); // 对认证证书来源的检查从证书中检查SSL加密算法是否存在 curl_setopt($curlobj, CURLOPT_SSL_VERIFYHOST, 2); // $output=curl_exec($curlobj); // 执行 curl_close($curlobj); // 关闭cURL echo $output; ?>
Native PHP simulates http request
Yes In order to simply simulate an http request, it is a bit wasteful to use curl. In fact, PHP itself can already implement this function.
Needs to simulate POST/GET and other requests on the server side , that is, to implement simulation in a PHP program, how to do it? In other words, in a PHP program, if you are given an array, how do you POST/GET this array to another address? Of course, it's easy to do it using CURL, but what if you don't use the CURL library? In fact, there is already a related function implemented in PHP, and this function is stream_context_create() that I will talk about next.
Show you the code directly, this is the best way:
$data = array(
'foo'=>'bar',
'baz'=>'boom',
'site'=>'www.nowamagic.net',
'name'=>'nowa magic');
$data = http_build_query($data);
//$postdata = http_build_query($data);
$options = array(
'http' => array(
'method' => 'POST',
'header' => 'Content-type:application/x-www-form-urlencoded',
'content' => $data
//'timeout' => 60 * 60 // 超时时间（单位:s）
)
);
$url = "http://www.nowamagic.net/test2.php";
$context = stream_context_create($options);
$result = file_get_contents($url, false, $context);
echo $result;http://www.nowamagic.net/test2.php code For:
$data = $_POST; echo '<pre class="brush:php;toolbar:false">'; print_r( $data ); echo '';
The running result is:
Array
(
[foo] => bar
[baz] => boom
[site] => www.nowamagic.net
[name] => nowa magic
)Some key points to explain:
The above program The http_build_query() function is used to construct the URL string.
stream_context_create() is used to create context options for opening files, such as accessing with POST, using a proxy, sending headers, etc. Just create a stream, let's give another example:
$context = stream_context_create(array(
'http' => array(
'method' => 'POST',
'header' => sprintf("Authorization: Basic %s\r\n", base64_encode($username.':'.$password)).
"Content-type: application/x-www-form-urlencoded\r\n",
'content' => http_build_query(array('status' => $message)),
'timeout' => 5,
),
));
$ret = file_get_contents('http://twitter.com/statuses/update.xml', false, $context);The context options created by stream_context_create can be used for streams (streams) and file systems (file systems). It is more useful for functions like file_get_contents, file_put_contents, and readfile that operate directly on file names without file handles. Adding headers to stream_context_create is only part of the function. You can also define proxies, timeouts, etc. This makes the function of accessing the web not weaker than curl.
stream_context_create() Function: Create and return a text data stream and apply various options. It can be used for timeout settings and proxy servers of fopen(), file_get_contents() and other processes. , request method, and special process of header information setting.
stream_context_create can also solve file_get_contents timeout processing by adding the timeout option:
$opts = array(
'http'=>array(
'method'=>"GET",
'timeout'=>60,
)
);
//创建数据流上下文
$context = stream_context_create($opts);
$html =file_get_contents('http://www.nowamagic.net', false, $context);
//fopen输出文件指针处的所有剩余数据:
//fpassthru($fp); //fclose()前使用The above is the detailed content of PHP learning CURL crawler example. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
PHP 8.4 Installation and Upgrade guide for Ubuntu and Debian
Dec 24, 2024 pm 04:42 PM
PHP 8.4 brings several new features, security improvements, and performance improvements with healthy amounts of feature deprecations and removals. This guide explains how to install PHP 8.4 or upgrade to PHP 8.4 on Ubuntu, Debian, or their derivati
7 PHP Functions I Regret I Didn't Know Before
Nov 13, 2024 am 09:42 AM
If you are an experienced PHP developer, you might have the feeling that you’ve been there and done that already.You have developed a significant number of applications, debugged millions of lines of code, and tweaked a bunch of scripts to achieve op
How To Set Up Visual Studio Code (VS Code) for PHP Development
Dec 20, 2024 am 11:31 AM
Visual Studio Code, also known as VS Code, is a free source code editor — or integrated development environment (IDE) — available for all major operating systems. With a large collection of extensions for many programming languages, VS Code can be c
Explain JSON Web Tokens (JWT) and their use case in PHP APIs.
Apr 05, 2025 am 12:04 AM
JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,
PHP Program to Count Vowels in a String
Feb 07, 2025 pm 12:12 PM
A string is a sequence of characters, including letters, numbers, and symbols. This tutorial will learn how to calculate the number of vowels in a given string in PHP using different methods. The vowels in English are a, e, i, o, u, and they can be uppercase or lowercase. What is a vowel? Vowels are alphabetic characters that represent a specific pronunciation. There are five vowels in English, including uppercase and lowercase: a, e, i, o, u Example 1 Input: String = "Tutorialspoint" Output: 6 explain The vowels in the string "Tutorialspoint" are u, o, i, a, o, i. There are 6 yuan in total
How do you parse and process HTML/XML in PHP?
Feb 07, 2025 am 11:57 AM
This tutorial demonstrates how to efficiently process XML documents using PHP. XML (eXtensible Markup Language) is a versatile text-based markup language designed for both human readability and machine parsing. It's commonly used for data storage an
Explain late static binding in PHP (static::).
Apr 03, 2025 am 12:04 AM
Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.
What are PHP magic methods (__construct, __destruct, __call, __get, __set, etc.) and provide use cases?
Apr 03, 2025 am 12:03 AM
What are the magic methods of PHP? PHP's magic methods include: 1.\_\_construct, used to initialize objects; 2.\_\_destruct, used to clean up resources; 3.\_\_call, handle non-existent method calls; 4.\_\_get, implement dynamic attribute access; 5.\_\_set, implement dynamic attribute settings. These methods are automatically called in certain situations, improving code flexibility and efficiency.
