Capture the supplier program of 'Weiku Electronic Market'

Home

Backend Development

PHP Tutorial

Capture the supplier program of 'Weiku Electronic Market'_PHP Tutorial

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jul 13, 2016 pm 05:48 PM

l php supplier market crawl electronic program

/**
* Capture the supplier main program of "Weiku Electronic Market"
* author Lee.
* Last modify $Date: 2012-2-3 9:30:21 $
* Note: This program is executed according to the encoding GB2312, because the "Weiku Electronic Market" website is encoded in GB2312, and the database must also be consistent
*/
class weiku {
    private $key; // 型号
    private $pageNum; // 页码

    /**
* Entry program
*/
    public function go($key) {
        $this->key = $key;
        $this->pageNum = $this->getPageNum();
        $this->getInfo();
    }

    /**
* Get the supplier url link array
* @return ArrayObject
*/
    private function getInfo() {
        if ($this->pageNum==1) { # 处理只有一页的情况
            $arr = $this->shopAddContact($this->shopUrlMatchReArr($this->getContent()));
            $this->isAddSuccess($arr);
        } elseif ($this->pageNum>1) { # 多页
            for ($i=1; $i<=$this->pageNum; $i++) {
                $arr = $this->shopAddContact($this->shopUrlMatchReArr($this->getContent($i)));
                $this->isAddSuccess($arr);
            }
        }
    }

    /**
* Print whether the addition was successful
* @param ArrayObject $arr
* @return string
*/
    private function isAddSuccess($arr) {
        foreach ($arr as $k=>$v) {
            if ($this->execAdd($this->getInfoByShopUrl($v))) {
                echo 'Add Success!!';
            } else {
                echo 'Add Faild!!';
            }
        }
    }

    /**
* Execute adding to database
* @param ArrayObject $infoArr
* @return Number The number of affected rows
*/
    private function execAdd($infoArr) {
        $mysqli = $this->getDb();
        if (!emptyempty($infoArr['company'])) {
            if (!$this->isExists($mysqli, $infoArr)) {
                $num = $mysqli->query("INSERT INTO weiku(company,person,phone,fax,mobile,qq,msn,email,address,shopUrl) VALUES ('{$infoArr['company']}','{$infoArr['person']}','{$infoArr['phone']}','{$infoArr['fax']}','{$infoArr['mobile']}','{$infoArr['qq']}','{$infoArr['msn']}','{$infoArr['email']}','{$infoArr['address']}','{$infoArr['shopUrl']}')");
                return $num;
            } else {
                return false; # 表示数据已经存在
            }
        } else {
            return false;
        }
    }

    /**
* Connect to database
*/
    private function getDb() {
        $mysqli = new mysqli('localhost', 'root', '1715544', 'weiku');
        $mysqli->query('SET NAMES GB2312');
        return $mysqli;
    }

    /**
* Check if the company already exists
* @param Resource $mysqli
* @param ArrayObject $infoArr
* @return bool
*/
    private function isExists($mysqli, $infoArr) {
        $mysqli->query("SELECT company FROM weiku WHERE company = '{$infoArr['company']}'");
        if ($mysqli->affected_rows) {
            return true;
        } else {
            return false;
        }
    }

    /**
* Grab information
* @param $url
* @return ArrayObject
*/
    private function getInfoByShopUrl($url) {
        $re = $this->getUrlInfo($url);
        preg_match_all('/公司名称：(.*)/Usi', $re, $companyArr);
        preg_match_all('/联系人：(.*)/Usi', $re, $personArr);
        preg_match_all('/电话：(.*)/Usi', $re, $phoneArr);
        preg_match_all('/传真：(.*)/Usi', $re, $faxArr);
        preg_match_all('/手机：(.*)/Usi', $re, $mobileArr);
        preg_match_all('/QQ：(.*)/Usi', $re, $qqArr);
        preg_match_all('/MSN：(.*)/Usi', $re, $msnArr);
        preg_match_all('/E-Mail：(.*)/Usi', $re, $emailArr);
        preg_match_all('/公司地址：(.*)/Usi', $re, $addressArr);
        $infoArr = array(
            'company'=>$this->stripATags($companyArr[1][0]),
            'person'=>trim($personArr[1][0]),
            'phone'=>trim($phoneArr[1][0]),
            'fax'=>trim($faxArr[1][0]),
            'mobile'=>trim($mobileArr[1][0]),
            'qq'=>$this->formatQqMsn($qqArr[1][0]),
            'msn'=>$this->formatQqMsn($msnArr[1][0], 'MSN'),
            'email'=>$this->stripATags($emailArr[1][0]),
            'address'=>trim($addressArr[1][0]),
            'shopUrl'=>$url
        );
        return $infoArr;
    }

    /**
* Get the supplier url array based on the page
* @param string $re
* @return ArrayObject
*/
    private function shopUrlMatchReArr($re) {
        $re = preg_replace('//', '', $re);
        $re = preg_replace('//', '', $re);
        $re = preg_replace('/[A-Z]/', '', $re);
        $re = preg_replace('/[0-9]/', '', $re);
        $re = preg_replace('/.*/', '', $re);
        $re = preg_replace('/.*/', '', $re);
        $re = preg_replace('/营业执照/', '', $re);
        $re = preg_replace('/该企业更多资质>>/', '', $re);
        $re = preg_replace('/点此反馈/', '', $re);
        $re = preg_replace('/首页/', '', $re);
        $re = preg_replace('/IC/', '', $re);
        $re = preg_replace('/简洁/', '', $re);
        $re = preg_replace('/信用/', '', $re);
        $re = preg_replace('/.*更多报价信息>>/', '', $re);
        $re = preg_replace('/.*/', '', $re);
        $re = preg_replace('/

.*/', '', $re);
        preg_match_all('/.+/Usi', $re, $arr);
        $arr = $this->formatUrlArr(array_unique($arr[1]));
        return $arr;
    }

    /**
* * Format array
* @param Array $arr
* @return ArrayObject
*/
    private function formatUrlArr($arr) {
        $newArr = array();
        foreach ($arr as $key=>$value) {
            if ($this->isExistsHttp($value)) {
                $newArr[$key] = $value;
            }
        }
        return $newArr;
    }

    /**
* Format QQ
* @param string $str
* @return string
*/
    private function formatQqMsn($str, $e='QQ') {
        if (emptyempty($str)) return '';
        preg_match_all('/alt="'.$e.':(.+)"/Usi', $str, $arr);
        if (count($arr[1])==1) return $arr[1][0];
        $newStr = null;
        foreach ($arr[1] as $value) {
            $newStr .= $value . ' ';
        }
        return rtrim($newStr, ' ');
    }

    /**
* Added supplier store link contact.html
* @param array $arr
* @return string
*/
    private function shopAddContact($arr) {
        foreach ($arr as $k=>$v) {
            $arr[$k] = $v . '/contact.html';
        }
        return $arr;
    }

    /**
* Remove the A tag from the URL
* @param string $site
* @return string
*/
    private function stripATags($site) {
        $site = preg_replace('/(.+)/', '1', $site);
        return $site;
    }

    /**
* Check if the url has http
* @param string $url
* @return bool
*/
    private function isExistsHttp($url) {
        if (stristr($url, 'http://')) {
            return true;
        } else {
            return false;
        }
    }

    /**
* Get page content
* @param Number $page
* @return string
*/
    private function getContent($page=1) {
        $re = file_get_contents($this->getUrl($this->key, $page));
        return $re;
    }

    /**
* Get page number
* @return Number
*/
    private function getPageNum() {
        $i = 1;
        while (true) {
            $re = $this->getContent($i);
            # 处理单页避免处理死循环
            if (!strstr($re, '下一页')) {
                break;
                                                                                # Multiple pages, calculate the page number
If (stristr($re, 'next page')) break;
$i++;
                                                                                                                                               }
         return $i;
}

/**
* Get URL link
* @param string $str
* @param int $page page number
* @return string
*/
Private function getUrl($str, $page=1) {
Return "http://www.dzsc.com/ic/sell_search.html?keyword={$str}&ic_sel=supplygoods&Submit=%26%23160%3B&page={$page}"; }

/**
* Get page content
* @param string $url
* @return string
*/
Private function getUrlInfo($url) {
          $re = file_get_contents($url);
         return $re;
}
}

/*
Program running idea: According to the IC search function of "Huaqiang Electronic Network", enter the model number to search, and then grab the supplier information

Database structure
CREATE TABLE `weiku` (
`id` mediumint(8) unsigned NOT NULL auto_increment COMMENT 'ID',
`company` varchar(300) default NULL COMMENT 'company name',
`person` varchar(200) default NULL COMMENT 'Contact',
`phone` varchar(300) default NULL COMMENT 'phone',
`fax` varchar(300) default NULL COMMENT 'Fax',
`mobile` varchar(300) default NULL COMMENT 'mobile phone',
`qq` varchar(200) default NULL COMMENT 'QQ',
`msn` varchar(200) default NULL COMMENT 'MSN',
`email` varchar(300) default NULL COMMENT 'email',
`address` varchar(500) default NULL COMMENT 'Company address',
`shopUrl` varchar(200) default NULL COMMENT 'Weiku.com store address',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=gb2312
*/

$k = new weiku();
$arr = array_unique(array('MAX3232', 'AML8613', 'MT6225A', 'OM8373PS/N3/A', 'PT7313', 'MAX8212ESA', 'TL431', 'S3C2440', 'TMS320F2812PGFA', 'PCM1704' 'AN6717', 'CA3162E', 'CA3161E', 'LM393N', 'DS18B20', 'SHT10', 'AML8613', 'AN6717', 'LM393N', 'CA3161E', 'CA3162E', 'PCM1704', ' STK392-040', 'K1667', 'MAX232', 'STM32F103', 'LM358'));
foreach ($arr as $v) {
$k->go($v);
}
?>
/**
* Capture the supplier main program of "Weiku Electronic Market"
* author Lee.
* Last modify $Date: 2012-2-3 9:30:21 $
* Note: This program is executed according to the encoding GB2312, because the "Weiku Electronic Market" website is encoded in GB2312, and the database must also be consistent
*/
class weiku {
private $key; //Model
private $pageNum; // Page number

/**

* Entry program

*/
public function go($key) {
$this->key = $key;
$this->pageNum = $this->getPageNum();
$this->getInfo();
}

/**
* Get supplier url link array
* @return ArrayObject
*/
private function getInfo() {
  if ($this->pageNum==1) { # 处理只有一页的情况
   $arr = $this->shopAddContact($this->shopUrlMatchReArr($this->getContent()));
   $this->isAddSuccess($arr);
  } elseif ($this->pageNum>1) { # 多页
   for ($i=1; $i<=$this->pageNum; $i++) {
    $arr = $this->shopAddContact($this->shopUrlMatchReArr($this->getContent($i)));
    $this->isAddSuccess($arr);
   }
  }
}

/**
* Print whether the addition was successful
* @param ArrayObject $arr
* @return string
*/
private function isAddSuccess($arr) {
  foreach ($arr as $k=>$v) {
   if ($this->execAdd($this->getInfoByShopUrl($v))) {
    echo 'Add Success!!';
   } else {
    echo 'Add Faild!!';
   }
  }
}

/**
* Execute adding to database
* @param ArrayObject $infoArr
* @return Number Number of affected rows
*/
private function execAdd($infoArr) {
  $mysqli = $this->getDb();
  if (!empty($infoArr['company'])) {
   if (!$this->isExists($mysqli, $infoArr)) {
    $num = $mysqli->query("INSERT INTO weiku(company,person,phone,fax,mobile,qq,msn,email,address,shopUrl) VALUES ('{$infoArr['company']}','{$infoArr['person']}','{$infoArr['phone']}','{$infoArr['fax']}','{$infoArr['mobile']}','{$infoArr['qq']}','{$infoArr['msn']}','{$infoArr['email']}','{$infoArr['address']}','{$infoArr['shopUrl']}')");
    return $num;
   } else {
    return false; # 表示数据已经存在
   }
  } else {
   return false;
  }
}

/**
* Connect to database
*/
private function getDb() {
  $mysqli = new mysqli('localhost', 'root', '1715544', 'weiku');
  $mysqli->query('SET NAMES GB2312');
  return $mysqli;
}

/**
* Check if the company already exists
* @param Resource $mysqli
* @param ArrayObject $infoArr
* @return bool
*/
private function isExists($mysqli, $infoArr) {
  $mysqli->query("SELECT company FROM weiku WHERE company = '{$infoArr['company']}'");
  if ($mysqli->affected_rows) {
   return true;
  } else {
   return false;
  }
}

/**
* Grab information
* @param $url
* @return ArrayObject
*/
private function getInfoByShopUrl($url) {
  $re = $this->getUrlInfo($url);
  preg_match_all('/公司名称：(.*)/Usi', $re, $companyArr);
  preg_match_all('/联系人：(.*)/Usi', $re, $personArr);
  preg_match_all('/电话：(.*)/Usi', $re, $phoneArr);
  preg_match_all('/传真：(.*)/Usi', $re, $faxArr);
  preg_match_all('/手机：(.*)/Usi', $re, $mobileArr);
  preg_match_all('/QQ：(.*)/Usi', $re, $qqArr);
  preg_match_all('/MSN：(.*)/Usi', $re, $msnArr);
  preg_match_all('/E-Mail：(.*)/Usi', $re, $emailArr);
  preg_match_all('/公司地址：(.*)/Usi', $re, $addressArr);
  $infoArr = array(
   'company'=>$this->stripATags($companyArr[1][0]),
   'person'=>trim($personArr[1][0]),
   'phone'=>trim($phoneArr[1][0]),
   'fax'=>trim($faxArr[1][0]),
   'mobile'=>trim($mobileArr[1][0]),
   'qq'=>$this->formatQqMsn($qqArr[1][0]),
   'msn'=>$this->formatQqMsn($msnArr[1][0], 'MSN'),
   'email'=>$this->stripATags($emailArr[1][0]),
   'address'=>trim($addressArr[1][0]),
   'shopUrl'=>$url
  );
  return $infoArr;
}

/**
* Get the supplier url array based on the page
* @param string $re
* @return ArrayObject
*/
private function shopUrlMatchReArr($re) {
  $re = preg_replace('//', '', $re);
  $re = preg_replace('//', '', $re);
  $re = preg_replace('/[A-Z]/', '', $re);
  $re = preg_replace('/[0-9]/', '', $re);
  $re = preg_replace('/.*/', '', $re);
  $re = preg_replace('/.*/', '', $re);
  $re = preg_replace('/营业执照/', '', $re);
  $re = preg_replace('/该企业更多资质>>/', '', $re);
  $re = preg_replace('/点此反馈/', '', $re);
  $re = preg_replace('/首页/', '', $re);
  $re = preg_replace('/IC/', '', $re);
  $re = preg_replace('/简洁/', '', $re);
  $re = preg_replace('/信用/', '', $re);
  $re = preg_replace('/.*更多报价信息>>/', '', $re);
  $re = preg_replace('/.*/', '', $re);
  $re = preg_replace('/

.*/', '', $re);
  preg_match_all('/.+/Usi', $re, $arr);
  $arr = $this->formatUrlArr(array_unique($arr[1]));
  return $arr;
}

/**
* Format array
* @param Array $arr
* @return ArrayObject
*/
private function formatUrlArr($arr) {
  $newArr = array();
  foreach ($arr as $key=>$value) {
   if ($this->isExistsHttp($value)) {
    $newArr[$key] = $value;
   }
  }
  return $newArr;
}

/**
* Format QQ
* @param string $str
* @return string
*/
private function formatQqMsn($str, $e='QQ') {
  if (empty($str)) return '';
  preg_match_all('/alt="'.$e.':(.+)"/Usi', $str, $arr);
  if (count($arr[1])==1) return $arr[1][0];
  $newStr = null;
  foreach ($arr[1] as $value) {
   $newStr .= $value . ' ';
  }
  return rtrim($newStr, ' ');
}

/**
*Add supplier store link contact.html
* @param array $arr
* @return string
*/
private function shopAddContact($arr) {
  foreach ($arr as $k=>$v) {
   $arr[$k] = $v . '/contact.html';
  }
  return $arr;
}

/**
* Remove the A tag from the URL
* @param string $site
* @return string
*/
private function stripATags($site) {
$site = preg_replace('/(.+)/', '1', $site);
return $site;
}

/**
* Check if the url has http
* @param string $url
* @return bool
*/
private function isExistsHttp($url) {
if (stristr($url, 'http://')) {
Return true;
} else {
Return false;
}
}

/**
* Get page content
* @param Number $page
* @return string
*/
private function getContent($page=1) {
$re = file_get_contents($this->getUrl($this->key, $page));
return $re;
}

/**
* Get page number
* @return Number
*/
private function getPageNum() {
$i = 1;
while (true) {
$re = $this->getContent($i);
# Processing single pages to avoid endless loops
if (!strstr($re, 'next page')) {
Break;
} else {
# Multiple pages, calculate the page number
If (stristr($re, 'next page')) break;
$i++;
}
}
Return $i;
}

/**
* Get URL link
* @param string $str
* @param int $page page number
* @return string
*/
private function getUrl($str, $page=1) {
retrn "http://www.dzsc.com/ic/sell_search.html?keyword={$str}&ic_sel=supplygoods&Submit=%26%23160%3B&page={$page}"u;

}

/**
* Get page content
* @param string $url
* @return string
*/
private function getUrlInfo($url) {
$re = file_get_contents($url);
return $re;
}
}

/*
Program running idea: According to the IC search function of "Huaqiang Electronic Network", enter the model number to search, and then grab the supplier information

Database structure
CREATE TABLE `weiku` (
`id` mediumint(8) unsigned NOT NULL auto_increment COMMENT 'ID',
`company` varchar(300) default NULL COMMENT 'company name',
`person` varchar(200) default NULL COMMENT 'Contact',
`phone` varchar(300) default NULL COMMENT 'phone',
`fax` varchar(300) default NULL COMMENT 'Fax',
`mobile` varchar(300) default NULL COMMENT 'mobile phone',
`qq` varchar(200) default NULL COMMENT 'QQ',
`msn` varchar(200) default NULL COMMENT 'MSN',
`email` varchar(300) default NULL COMMENT 'email',
`address` varchar(500) default NULL COMMENT 'Company address',
`shopUrl` varchar(200) default NULL COMMENT 'Weiku.com store address',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=gb2312
*/

$k = new weiku();
$arr = array_unique(array('MAX3232', 'AML8613', 'MT6225A', 'OM8373PS/N3/A', 'PT7313', 'MAX8212ESA', 'TL431', 'S3C2440', 'TMS320F2812PGFA', 'PCM1704' 'AN6717', 'CA3162E', 'CA3161E', 'LM393N', 'DS18B20', 'SHT10', 'AML8613', 'AN6717', 'LM393N', 'CA3161E', 'CA3162E', 'PCM1704', ' STK392-040', 'K1667', 'MAX232', 'STM32F103', 'LM358'));
foreach ($arr as $v) {
$k->go($v);
}
?>

Excerpted from Lee.’s column

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

How to fix KB5055612 fails to install in Windows 10?

4 weeks ago By DDD

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys

4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Roblox: Grow A Garden - Complete Mutation Guide

3 weeks ago By DDD

Nordhold: Fusion System, Explained

4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Mandragora: Whispers Of The Witch Tree - How To Unlock The Grappling Hook

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial

1675

CakePHP Tutorial

1429

Laravel Tutorial

1333

PHP Tutorial

1278

C# Tutorial

1257

Related knowledge

PHP vs. Python: Understanding the Differences Apr 11, 2025 am 12:15 AM

PHP and Python each have their own advantages, and the choice should be based on project requirements. 1.PHP is suitable for web development, with simple syntax and high execution efficiency. 2. Python is suitable for data science and machine learning, with concise syntax and rich libraries.

PHP: A Key Language for Web Development Apr 13, 2025 am 12:08 AM

PHP is a scripting language widely used on the server side, especially suitable for web development. 1.PHP can embed HTML, process HTTP requests and responses, and supports a variety of databases. 2.PHP is used to generate dynamic web content, process form data, access databases, etc., with strong community support and open source resources. 3. PHP is an interpreted language, and the execution process includes lexical analysis, grammatical analysis, compilation and execution. 4.PHP can be combined with MySQL for advanced applications such as user registration systems. 5. When debugging PHP, you can use functions such as error_reporting() and var_dump(). 6. Optimize PHP code to use caching mechanisms, optimize database queries and use built-in functions. 7

PHP and Python: Comparing Two Popular Programming Languages Apr 14, 2025 am 12:13 AM

PHP and Python each have their own advantages, and choose according to project requirements. 1.PHP is suitable for web development, especially for rapid development and maintenance of websites. 2. Python is suitable for data science, machine learning and artificial intelligence, with concise syntax and suitable for beginners.

PHP in Action: Real-World Examples and Applications Apr 14, 2025 am 12:19 AM

PHP is widely used in e-commerce, content management systems and API development. 1) E-commerce: used for shopping cart function and payment processing. 2) Content management system: used for dynamic content generation and user management. 3) API development: used for RESTful API development and API security. Through performance optimization and best practices, the efficiency and maintainability of PHP applications are improved.

The Enduring Relevance of PHP: Is It Still Alive? Apr 14, 2025 am 12:12 AM

PHP is still dynamic and still occupies an important position in the field of modern programming. 1) PHP's simplicity and powerful community support make it widely used in web development; 2) Its flexibility and stability make it outstanding in handling web forms, database operations and file processing; 3) PHP is constantly evolving and optimizing, suitable for beginners and experienced developers.

PHP and Python: Different Paradigms Explained Apr 18, 2025 am 12:26 AM

PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

PHP and Python: Code Examples and Comparison Apr 15, 2025 am 12:07 AM

PHP and Python have their own advantages and disadvantages, and the choice depends on project needs and personal preferences. 1.PHP is suitable for rapid development and maintenance of large-scale web applications. 2. Python dominates the field of data science and machine learning.

PHP vs. Other Languages: A Comparison Apr 13, 2025 am 12:19 AM

PHP is suitable for web development, especially in rapid development and processing dynamic content, but is not good at data science and enterprise-level applications. Compared with Python, PHP has more advantages in web development, but is not as good as Python in the field of data science; compared with Java, PHP performs worse in enterprise-level applications, but is more flexible in web development; compared with JavaScript, PHP is more concise in back-end development, but is not as good as JavaScript in front-end development.

See all articles