Home Backend Development PHP Tutorial PHP implements multi-threaded crawling based on curl

PHP implements multi-threaded crawling based on curl

Jun 08, 2018 pm 06:01 PM
php

This article mainly introduces PHP to implement multi-threaded crawling based on curl. Interested friends can refer to it. I hope it will be helpful to everyone.

PHP can use Curl to complete various file transfer operations, such as simulating a browser to send GET, POST requests, etc. However, because the PHP language itself does not support multi-threading, the efficiency of developing crawler programs is not high, so it is often You need to use the Curl Multi Functions function to achieve concurrent multi-threaded access to multiple url addresses to achieve concurrent multi-threaded crawling of web pages or downloading files.

The code is as follows:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

<?php

/*

curl 多线程抓取

*/

 /**

   * curl 多线程

   *

   * @param array $array 并行网址

   * @param int $timeout 超时时间

   * @return array

   */

 function Curl_http($array,$timeout){

 $res = array();

 $mh = curl_multi_init();//创建多个curl语柄

 $startime = getmicrotime();

 foreach($array as $k=>$url){

  $conn[$k]=curl_init($url);

 

    curl_setopt($conn[$k], CURLOPT_TIMEOUT, $timeout);//设置超时时间

    curl_setopt($conn[$k], CURLOPT_USERAGENT, &#39;Mozilla/5.0 (compatible; MSIE 5.01; Windows NT 5.0)&#39;);

    curl_setopt($conn[$k], CURLOPT_MAXREDIRS, 7);//HTTp定向级别

    curl_setopt($conn[$k], CURLOPT_HEADER, 0);//这里不要header,加块效率

    curl_setopt($conn[$k], CURLOPT_FOLLOWLOCATION, 1); // 302 redirect

    curl_setopt($conn[$k],CURLOPT_RETURNTRANSFER,1);

    curl_multi_add_handle ($mh,$conn[$k]);

 }

 //防止死循环耗死cpu 这段是根据网上的写法

 do {

  $mrc = curl_multi_exec($mh,$active);//当无数据,active=true

 } while ($mrc == CURLM_CALL_MULTI_PERFORM);//当正在接受数据时

 while ($active and $mrc == CURLM_OK) {//当无数据时或请求暂停时,active=true

  if (curl_multi_select($mh) != -1) {

  do {

   $mrc = curl_multi_exec($mh, $active);

  } while ($mrc == CURLM_CALL_MULTI_PERFORM);

  }

 }

 

 foreach ($array as $k => $url) {

   curl_error($conn[$k]);

    $res[$k]=curl_multi_getcontent($conn[$k]);//获得返回信息

    $header[$k]=curl_getinfo($conn[$k]);//返回头信息

    curl_close($conn[$k]);//关闭语柄

    curl_multi_remove_handle($mh , $conn[$k]);  //释放资源

 }

 

 curl_multi_close($mh);

 $endtime = getmicrotime();

 $diff_time = $endtime - $startime;

 

 return array(&#39;diff_time&#39;=>$diff_time,

   &#39;return&#39;=>$res,

   &#39;header&#39;=>$header

   );

 

 }

 //计算当前时间

 function getmicrotime() {

   list($usec, $sec) = explode(" ",microtime());

   return ((float)$usec + (float)$sec);

 }

 

 //测试一下,curl 三个网址

 $array = array(

  "http://www.weibo.com/",

  "http://www.renren.com/",

  "http://www.qq.com/"

  );

 $data = Curl_http($array,&#39;10&#39;);//调用

 var_dump($data);//输出

//如果POST的数据大于1024字节,curl并不会直接就发起POST请求

//发送请求时,header中包含一个空的Expect。curl_setopt($ch, CURLOPT_HTTPHEADER, array("Expect:"));

?>

Copy after login

Let’s look at a few more examples

(1) The following code is to capture multiple URLs, and then write the page code of the captured URLs into the specified file

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

$urls = array(

&#39;http://www.jb51.net/&#39;,

&#39;http://www.google.com/&#39;,

&#39;http://www.example.com/&#39;

); // 设置要抓取的页面URL

$save_to=&#39;/test.txt&#39;; // 把抓取的代码写入该文件

$st = fopen($save_to,"a");

$mh = curl_multi_init();

foreach ($urls as $i => $url) {

$conn[$i] = curl_init($url);

curl_setopt($conn[$i], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)");

curl_setopt($conn[$i], CURLOPT_HEADER ,0);

curl_setopt($conn[$i], CURLOPT_CONNECTTIMEOUT,60);

curl_setopt($conn[$i], CURLOPT_FILE,$st); // 将爬取的代码写入文件

curl_multi_add_handle ($mh,$conn[$i]);

} // 初始化

do {

curl_multi_exec($mh,$active);

} while ($active); // 执行

foreach ($urls as $i => $url) {

curl_multi_remove_handle($mh,$conn[$i]);

curl_close($conn[$i]);

} // 结束清理

curl_multi_close($mh);

fclose($st);

Copy after login

(2) The following code and The above is almost the same, except that this place puts the obtained code into the variable first, and then writes the obtained content to the specified file

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

$urls = array(

&#39;http://www.jb51.net/&#39;,

&#39;http://www.google.com/&#39;,

&#39;http://www.example.com/&#39;

);

$save_to=&#39;/test.txt&#39;; // 把抓取的代码写入该文件

$st = fopen($save_to,"a");

$mh = curl_multi_init();

foreach ($urls as $i => $url) {

$conn[$i] = curl_init($url);

curl_setopt($conn[$i], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)");

curl_setopt($conn[$i], CURLOPT_HEADER ,0);

curl_setopt($conn[$i], CURLOPT_CONNECTTIMEOUT,60);

curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,true); // 不将爬取代码写到浏览器,而是转化为字符串

curl_multi_add_handle ($mh,$conn[$i]);

}

do {

curl_multi_exec($mh,$active);

} while ($active);

foreach ($urls as $i => $url) {

$data = curl_multi_getcontent($conn[$i]); // 获得爬取的代码字符串

fwrite($st,$data); // 将字符串写入文件

} // 获得数据变量,并写入文件

foreach ($urls as $i => $url) {

curl_multi_remove_handle($mh,$conn[$i]);

curl_close($conn[$i]);

}

curl_multi_close($mh);

fclose($st);

Copy after login

(3) The following code is implemented using PHP Curl Functions implements concurrent multi-threaded downloading of files

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

$urls=array(

 &#39;http://www.jb51.net/5w.zip&#39;,

 &#39;http://www.jb51.net/5w.zip&#39;,

 &#39;http://www.jb51.net/5w.zip&#39;

);

$save_to=&#39;./home/&#39;;

$mh=curl_multi_init();

foreach($urls as $i=>$url){

 $g=$save_to.basename($url);

 if(!is_file($g)){

   $conn[$i]=curl_init($url);

   $fp[$i]=fopen($g,"w");

   curl_setopt($conn[$i],CURLOPT_USERAGENT,"Mozilla/4.0(compatible; MSIE 7.0; Windows NT 6.0)");

   curl_setopt($conn[$i],CURLOPT_FILE,$fp[$i]);

   curl_setopt($conn[$i],CURLOPT_HEADER ,0);

   curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);

   curl_multi_add_handle($mh,$conn[$i]);

 }

}

do{

 $n=curl_multi_exec($mh,$active);

}while($active);

foreach($urls as $i=>$url){

 curl_multi_remove_handle($mh,$conn[$i]);

 curl_close($conn[$i]);

 fclose($fp[$i]);

}

curl_multi_close($mh);$urls=array(

 &#39;http://www.jb51.net/5w.zip&#39;,

 &#39;http://www.jb51.net/5w.zip&#39;,

 &#39;http://www.jb51.net/5w.zip&#39;

);

$save_to=&#39;./home/&#39;;

$mh=curl_multi_init();

foreach($urls as $i=>$url){

 $g=$save_to.basename($url);

 if(!is_file($g)){

   $conn[$i]=curl_init($url);

   $fp[$i]=fopen($g,"w");

   curl_setopt($conn[$i],CURLOPT_USERAGENT,"Mozilla/4.0(compatible; MSIE 7.0; Windows NT 6.0)");

   curl_setopt($conn[$i],CURLOPT_FILE,$fp[$i]);

   curl_setopt($conn[$i],CURLOPT_HEADER ,0);

   curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);

   curl_multi_add_handle($mh,$conn[$i]);

 }

}

do{

 $n=curl_multi_exec($mh,$active);

}while($active);

foreach($urls as $i=>$url){

 curl_multi_remove_handle($mh,$conn[$i]);

 curl_close($conn[$i]);

 fclose($fp[$i]);

}

curl_multi_close($mh);

Copy after login

Summary:The above is the entire content of this article, I hope it will be helpful to everyone's learning.

Related recommendations:

php file upload management system

PHP email sending case

php file upload management system

The above is the detailed content of PHP implements multi-threaded crawling based on curl. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Nordhold: Fusion System, Explained
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Mandragora: Whispers Of The Witch Tree - How To Unlock The Grappling Hook
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Clair Obscur: Expedition 33 - How To Get Perfect Chroma Catalysts
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1676
14
PHP Tutorial
1278
29
C# Tutorial
1257
24
PHP and Python: Code Examples and Comparison PHP and Python: Code Examples and Comparison Apr 15, 2025 am 12:07 AM

PHP and Python have their own advantages and disadvantages, and the choice depends on project needs and personal preferences. 1.PHP is suitable for rapid development and maintenance of large-scale web applications. 2. Python dominates the field of data science and machine learning.

PHP and Python: Different Paradigms Explained PHP and Python: Different Paradigms Explained Apr 18, 2025 am 12:26 AM

PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

PHP: Handling Databases and Server-Side Logic PHP: Handling Databases and Server-Side Logic Apr 15, 2025 am 12:15 AM

PHP uses MySQLi and PDO extensions to interact in database operations and server-side logic processing, and processes server-side logic through functions such as session management. 1) Use MySQLi or PDO to connect to the database and execute SQL queries. 2) Handle HTTP requests and user status through session management and other functions. 3) Use transactions to ensure the atomicity of database operations. 4) Prevent SQL injection, use exception handling and closing connections for debugging. 5) Optimize performance through indexing and cache, write highly readable code and perform error handling.

PHP's Purpose: Building Dynamic Websites PHP's Purpose: Building Dynamic Websites Apr 15, 2025 am 12:18 AM

PHP is used to build dynamic websites, and its core functions include: 1. Generate dynamic content and generate web pages in real time by connecting with the database; 2. Process user interaction and form submissions, verify inputs and respond to operations; 3. Manage sessions and user authentication to provide a personalized experience; 4. Optimize performance and follow best practices to improve website efficiency and security.

Choosing Between PHP and Python: A Guide Choosing Between PHP and Python: A Guide Apr 18, 2025 am 12:24 AM

PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.

PHP and Python: A Deep Dive into Their History PHP and Python: A Deep Dive into Their History Apr 18, 2025 am 12:25 AM

PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.

Why Use PHP? Advantages and Benefits Explained Why Use PHP? Advantages and Benefits Explained Apr 16, 2025 am 12:16 AM

The core benefits of PHP include ease of learning, strong web development support, rich libraries and frameworks, high performance and scalability, cross-platform compatibility, and cost-effectiveness. 1) Easy to learn and use, suitable for beginners; 2) Good integration with web servers and supports multiple databases; 3) Have powerful frameworks such as Laravel; 4) High performance can be achieved through optimization; 5) Support multiple operating systems; 6) Open source to reduce development costs.

PHP's Impact: Web Development and Beyond PHP's Impact: Web Development and Beyond Apr 18, 2025 am 12:10 AM

PHPhassignificantlyimpactedwebdevelopmentandextendsbeyondit.1)ItpowersmajorplatformslikeWordPressandexcelsindatabaseinteractions.2)PHP'sadaptabilityallowsittoscaleforlargeapplicationsusingframeworkslikeLaravel.3)Beyondweb,PHPisusedincommand-linescrip

See all articles