Data crawling of Tmall and Taobao products
This time I will bring you the data crawling of Tmall and Taobao products. What are the precautions for crawling Tmall and Taobao product data. The following is a practical case, let's take a look.
1. Ideas
I recently made a website that crawled product information from Tmall and Taobao from the URL. I first looked at the mobile web page and found that it was used. React, I don’t know much about it and I can’t do it, so I considered crawling the data from the PC portal. However, when crawling the URL to obtain the data, the price, inventory, etc. information was not obtained. After careful study, I found that another interface was requested asynchronously. However, the interface needs to use refer to obtain data, so I wrote a simple crawler in the following way to crawl the product preview and the price, inventory, etc. of the first category of the product.
2. The code to implement
is as follows:
function crawlUrl($url){ import('PhpQuery.Curl'); $curl=new \Curl(); $result = $curl->read($url); $content = mb_convert_encoding( $result['content'], 'UTF-8', 'UTF-8,GBK,GB2312,BIG5' ); $myres=array(); if(strrpos($url,'taobao.com')!=false) { //匹配是否下架 if(strpos($content,'此宝贝已下架')!==false){ return false; } preg_match("|itemId : '(.*)'|isU", $content, $match); $item_id=$match[1]; preg_match("|sellerId : '(.*)'|isU", $content, $match); $sellet_id=$match[1]; preg_match("|<title>(.*)</title>|isU",$content,$match); $title=$match[1]; //价格库存信息 $ch = curl_init(); curl_setopt ($ch, CURLOPT_URL, 'https://detailskip.taobao.com/service/getData/1/p1/item/detail/sib.htm?itemId='.$item_id.'&sellerId='.$sellet_id.'&modules=dynStock,qrcode,viewer,price,duty,xmpPromotion,delivery,upp,activity,fqg,zjys,amountRestriction,couponActivity,soldQuantity,originalPrice,tradeContract&callback=onSibRequestSuccess'); $opt[CURLOPT_HEADER]=false; $opt[CURLOPT_CONNECTTIMEOUT]=15; $opt[CURLOPT_TIMEOUT]=300; $opt[CURLOPT_AUTOREFERER]=true; $opt[CURLOPT_USERAGENT]='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11'; curl_setopt_array($ch,$opt); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ch,CURLOPT_REFERER,$url); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); $out_put=curl_exec ($ch); curl_close ($ch); $res=str_replace('onSibRequestSuccess(',"",$out_put); $res=rtrim($res,');1'); $result=json_decode($res,true); //查询出图片信息 preg_match('|<ul id="J_UlThumb" class="tb-thumb tb-clearfix">(.*)</ul>|isU', $content, $match); preg_match_all('/<img src="(.*?)" \//', $match[1], $images); $myres['title']=str_replace('-淘宝网','',$title); $myres['price']=current($result['data']['originalPrice']); $myres['act_price']=current($result['data']['promotion']['promoData']); $myres['stock']=$result['data']['dynStock']['stock']; $myres['banners']=$images[1]; }else{ //匹配是否下架 if(strpos($content,'此宝贝已下架')!==false){ return false; } $start=strpos($url,'&id='); $item_id=substr($url,$start+4,12); if(!is_numeric($item_id)){ $start=strpos($url,'?id='); $end=strpos($url,'&spm'); $item_id=substr($url,$start+4,$end-$start-4); } preg_match("|<title>(.*)</title>|isU",$content,$match); $title=$match[1]; $myurl='https://mdskip.taobao.com/core/initItemDetail.htm?cachedTimestamp=1500562177777&queryMemberRight=true&cartEnable=true&offlineShop=false&addressLevel=2&itemId='.$item_id.'&tryBeforeBuy=false&isAreaSell=false&tmallBuySupport=true&isPurchaseMallPage=false&household=false&isForbidBuyItem=false&service3C=false&isRegionLevel=false&showShopProm=false&isSecKill=false&sellerPreview=false&isUseInventoryCenter=false&isApparel=true&callback=setMdskip×tamp=1500562172109&isg=AiUlDZFWmP/sMgVurQSILU3Ytet/Zdis&isg2=Ajk51JIhRFqKzxmiNPP6dkYxSKXT7iySkzSTeVtu9WDf4ll0o5Y9yKdyEtHu'; //价格库存信息 $ch = curl_init(); curl_setopt ($ch, CURLOPT_URL, $myurl); $opt[CURLOPT_HEADER]=false; $opt[CURLOPT_CONNECTTIMEOUT]=15; $opt[CURLOPT_TIMEOUT]=300; $opt[CURLOPT_AUTOREFERER]=true; $opt[CURLOPT_USERAGENT]='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11'; curl_setopt_array($ch,$opt); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ch,CURLOPT_REFERER,$url); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); $out_put=curl_exec ($ch); curl_close ($ch); $res = mb_convert_encoding( $out_put, 'UTF-8', 'UTF-8,GBK,GB2312,BIG5' ); $res=str_replace('setMdskip',"",$res); $res=str_replace('(',"",$res); $res=str_replace(')',"",$res); $result=json_decode($res,true); $nowk=""; $nowstore=""; foreach($result['defaultModel']['inventoryDO']['skuQuantity'] as $k=>$val){ $nowk=$k; $nowstore=$val; break; } $myres['title']=str_replace('-tmall.com天猫','',$title); $myres['price']=$result['defaultModel']['itemPriceResultDO']['priceInfo'][$nowk]['price']; $myres['act_price']=isset($result['defaultModel']['itemPriceResultDO']['priceInfo'][$nowk]['suggestivePromotionList'])?$result['defaultModel']['itemPriceResultDO']['priceInfo'][$nowk]['suggestivePromotionList']:$result['defaultModel']['itemPriceResultDO']['priceInfo'][$nowk]; $myres['stock']=$result['defaultModel']['inventoryDO']['totalQuantity']?$result['defaultModel']['inventoryDO']['totalQuantity']:$nowstore['quantity']; //查询出图片信息 preg_match('|<ul id="J_UlThumb" class="tb-thumb tm-clear">(.*)</ul>|isU',$content, $match); preg_match_all('/<img src="(.*?)" \//',$match[1],$images); $myres['banners']=$images[1]; } return $myres; }
I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to the php Chinese website Other related articles!
Recommended reading:
Implementing APP payment function on the server side
Detailed explanation of API and interface development on the PHP server side
The above is the detailed content of Data crawling of Tmall and Taobao products. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

DDREASE is a tool for recovering data from file or block devices such as hard drives, SSDs, RAM disks, CDs, DVDs and USB storage devices. It copies data from one block device to another, leaving corrupted data blocks behind and moving only good data blocks. ddreasue is a powerful recovery tool that is fully automated as it does not require any interference during recovery operations. Additionally, thanks to the ddasue map file, it can be stopped and resumed at any time. Other key features of DDREASE are as follows: It does not overwrite recovered data but fills the gaps in case of iterative recovery. However, it can be truncated if the tool is instructed to do so explicitly. Recover data from multiple files or blocks to a single

0.What does this article do? We propose DepthFM: a versatile and fast state-of-the-art generative monocular depth estimation model. In addition to traditional depth estimation tasks, DepthFM also demonstrates state-of-the-art capabilities in downstream tasks such as depth inpainting. DepthFM is efficient and can synthesize depth maps within a few inference steps. Let’s read about this work together ~ 1. Paper information title: DepthFM: FastMonocularDepthEstimationwithFlowMatching Author: MingGui, JohannesS.Fischer, UlrichPrestel, PingchuanMa, Dmytr

If you need to know how to use filtering with multiple criteria in Excel, the following tutorial will guide you through the steps to ensure you can filter and sort your data effectively. Excel's filtering function is very powerful and can help you extract the information you need from large amounts of data. This function can filter data according to the conditions you set and display only the parts that meet the conditions, making data management more efficient. By using the filter function, you can quickly find target data, saving time in finding and organizing data. This function can not only be applied to simple data lists, but can also be filtered based on multiple conditions to help you locate the information you need more accurately. Overall, Excel’s filtering function is a very practical

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

Facing lag, slow mobile data connection on iPhone? Typically, the strength of cellular internet on your phone depends on several factors such as region, cellular network type, roaming type, etc. There are some things you can do to get a faster, more reliable cellular Internet connection. Fix 1 – Force Restart iPhone Sometimes, force restarting your device just resets a lot of things, including the cellular connection. Step 1 – Just press the volume up key once and release. Next, press the Volume Down key and release it again. Step 2 – The next part of the process is to hold the button on the right side. Let the iPhone finish restarting. Enable cellular data and check network speed. Check again Fix 2 – Change data mode While 5G offers better network speeds, it works better when the signal is weaker

I cry to death. The world is madly building big models. The data on the Internet is not enough. It is not enough at all. The training model looks like "The Hunger Games", and AI researchers around the world are worrying about how to feed these data voracious eaters. This problem is particularly prominent in multi-modal tasks. At a time when nothing could be done, a start-up team from the Department of Renmin University of China used its own new model to become the first in China to make "model-generated data feed itself" a reality. Moreover, it is a two-pronged approach on the understanding side and the generation side. Both sides can generate high-quality, multi-modal new data and provide data feedback to the model itself. What is a model? Awaker 1.0, a large multi-modal model that just appeared on the Zhongguancun Forum. Who is the team? Sophon engine. Founded by Gao Yizhao, a doctoral student at Renmin University’s Hillhouse School of Artificial Intelligence.

The latest video of Tesla's robot Optimus is released, and it can already work in the factory. At normal speed, it sorts batteries (Tesla's 4680 batteries) like this: The official also released what it looks like at 20x speed - on a small "workstation", picking and picking and picking: This time it is released One of the highlights of the video is that Optimus completes this work in the factory, completely autonomously, without human intervention throughout the process. And from the perspective of Optimus, it can also pick up and place the crooked battery, focusing on automatic error correction: Regarding Optimus's hand, NVIDIA scientist Jim Fan gave a high evaluation: Optimus's hand is the world's five-fingered robot. One of the most dexterous. Its hands are not only tactile

This week, FigureAI, a robotics company invested by OpenAI, Microsoft, Bezos, and Nvidia, announced that it has received nearly $700 million in financing and plans to develop a humanoid robot that can walk independently within the next year. And Tesla’s Optimus Prime has repeatedly received good news. No one doubts that this year will be the year when humanoid robots explode. SanctuaryAI, a Canadian-based robotics company, recently released a new humanoid robot, Phoenix. Officials claim that it can complete many tasks autonomously at the same speed as humans. Pheonix, the world's first robot that can autonomously complete tasks at human speeds, can gently grab, move and elegantly place each object to its left and right sides. It can autonomously identify objects
