Home Backend Development PHP Tutorial 大侠何在,采集数据被封ip该怎么处理

大侠何在,采集数据被封ip该怎么处理

Jun 13, 2016 am 10:12 AM
curl headers quot sleep

大侠何在,采集数据被封ip该怎么办,
大侠何在,采集数据被封ip该怎么办,如何判断出网站都用了哪些仿采集的程序啊,我们又改如何避让开这些仿采集程序啊

------解决方案--------------------

没办法,你可以装做客户投诉他们,
------解决方案--------------------
sleep() 你不要太暴力采集 我们现在采集 都是一秒采集一次 否则对大家都不好是不是 你一下子并发那么高 大家都不好 ,慢慢来 他不会封的。
------解决方案--------------------
这个...除了你换IP,找代理服务器,伪造IP,没其他啥好方法...
人家看到你的IP频繁访问,直接iptables屏掉你的IP都是很简单的...
------解决方案--------------------
封就疯了吧!人家辛辛苦苦地正的那些资料,凭什么给你呀
------解决方案--------------------
疯吧,既然知道封ip,那就用别的服务器采集呗
降低频率,悠着点
------解决方案--------------------
sleep(60)?要是我我也封你ip

我采集都是隔几分钟才一篇
------解决方案--------------------
隔几分钟一篇,这样采集速度会很慢的吧?
不过好像也没有办法吧,毕竟是人家的东西
------解决方案--------------------
用CURL可以假冒IP的吧
$headers['CLIENT-IP'] = $headers['X-FORWARDED-FOR'] = '192.168.1.1'; //用请求地址的IP
$headerArr = array();
foreach( $headers as $n => $v ) {
$headerArr[] = $n .':' . $v;
}
$refer = $_SERVER["HTTP_REFERER"]

curl_setopt($ch, CURLOPT_HTTPHEADER , $headerArr); //构造IP

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to realize the mutual conversion between CURL and python requests in python How to realize the mutual conversion between CURL and python requests in python May 03, 2023 pm 12:49 PM

Both curl and Pythonrequests are powerful tools for sending HTTP requests. While curl is a command-line tool that allows you to send requests directly from the terminal, Python's requests library provides a more programmatic way to send requests from Python code. The basic syntax for converting curl to Pythonrequestscurl command is as follows: curl[OPTIONS]URL When converting curl command to Python request, we need to convert the options and URL into Python code. Here is an example curlPOST command: curl-XPOST https://example.com/api

From start to finish: How to use php extension cURL to make HTTP requests From start to finish: How to use php extension cURL to make HTTP requests Jul 29, 2023 pm 05:07 PM

From start to finish: How to use php extension cURL for HTTP requests Introduction: In web development, it is often necessary to communicate with third-party APIs or other remote servers. Using cURL to make HTTP requests is a common and powerful way. This article will introduce how to use PHP to extend cURL to perform HTTP requests, and provide some practical code examples. 1. Preparation First, make sure that php has the cURL extension installed. You can execute php-m|grepcurl on the command line to check

Tutorial on updating curl version under Linux! Tutorial on updating curl version under Linux! Mar 07, 2024 am 08:30 AM

To update the curl version under Linux, you can follow the steps below: Check the current curl version: First, you need to determine the curl version installed in the current system. Open a terminal and execute the following command: curl --version This command will display the current curl version information. Confirm available curl version: Before updating curl, you need to confirm the latest version available. You can visit curl's official website (curl.haxx.se) or related software sources to find the latest version of curl. Download the curl source code: Using curl or a browser, download the source code file for the curl version of your choice (usually .tar.gz or .tar.bz2

PHP8.1 released: Introducing curl for concurrent processing of multiple requests PHP8.1 released: Introducing curl for concurrent processing of multiple requests Jul 08, 2023 pm 09:13 PM

PHP8.1 released: Introducing curl for concurrent processing of multiple requests. Recently, PHP officially released the latest version of PHP8.1, which introduced an important feature: curl for concurrent processing of multiple requests. This new feature provides developers with a more efficient and flexible way to handle multiple HTTP requests, greatly improving performance and user experience. In previous versions, handling multiple requests often required creating multiple curl resources and using loops to send and receive data respectively. Although this method can achieve the purpose

Can linux sleep delay threads? Can linux sleep delay threads? Mar 16, 2023 am 11:02 AM

sleep can be delayed. The linux sleep command can be used to delay the current action for a period of time. The syntax is "sleep [--help] [--version] number[smhd]"; by default, the sleep command will delay for a few seconds, but the suffix is ​​allowed to specify the time. Units to specify the delay in seconds, minutes, hours, or days.

How to handle 301 redirection of web pages in PHP Curl? How to handle 301 redirection of web pages in PHP Curl? Mar 08, 2024 am 11:36 AM

How to handle 301 redirection of web pages in PHPCurl? When using PHPCurl to send network requests, you will often encounter a 301 status code returned by the web page, indicating that the page has been permanently redirected. In order to handle this situation correctly, we need to add some specific options and processing logic to the Curl request. The following will introduce in detail how to handle 301 redirection of web pages in PHPCurl, and provide specific code examples. 301 redirect processing principle 301 redirect means that the server returns a 30

what is linux curl what is linux curl Apr 20, 2023 pm 05:05 PM

In Linux, curl is a very practical tool for transferring data to and from the server. It is a file transfer tool that uses URL rules to work under the command line; it supports file upload and download, and is a comprehensive transfer tool. . Curl provides a lot of very useful functions, including proxy access, user authentication, ftp upload and download, HTTP POST, SSL connection, cookie support, breakpoint resume and so on.

What is the difference between sleep and wait methods in Java? What is the difference between sleep and wait methods in Java? May 06, 2023 am 09:52 AM

1. The fundamental difference between sleep and wait methods: sleep is a method in the Thread class and will not enter the running state immediately. wait is a method in the Object class. Once an object calls the wait method, notify() and notifyAll must be used. () method wakes up the process to release the synchronization lock: sleep will release the cpu, but sleep will not release the synchronization lock resources, and wait will release the synchronization lock resource usage range: sleep can be used anywhere, but wait can only be used in the synchronized synchronization method Or use exception handling in the code block: sleep needs to catch exceptions, but wait does not need to catch exceptions 2. wa

See all articles