How to use async in node to control concurrency
This time I will show you how to use async in node to control concurrency, and what are the precautions for using async in node to control concurrency. The following is a practical case, let's take a look.
Objective
Create a lesson5 project and write code in it. The entry point of the code is app.js. When node app.js is called, it will output the titles of all topics on the CNode (https://cnodejs.org/) community homepage. Link and first comment in json format.
Note: Unlike the previous lesson, the number of concurrent connections needs to be controlled at 5. Output example:[
{
"title": "【公告】发招聘帖的同学留意一下这里",
"href": "http://cnodejs.org/topic/541ed2d05e28155f24676a12",
"comment1": "呵呵呵呵"
},
{
"title": "发布一款 Sublime Text 下的 JavaScript 语法高亮插件",
"href": "http://cnodejs.org/topic/54207e2efffeb6de3d61f68f",
"comment1": "沙发！"
}
]Knowledge points##Learn async(https://github.com/caolan/async ) usage of. Here is a detailed async demo: https://github.com/alsotang/async_demo
Learn to use async to control the number of concurrent connections.
Course content#lesson4’s code is actually imperfect. The reason why we say this is because in lesson4, we sent 40 concurrent requests at one time. You must know that, except for CNode, other websites may treat you as a malicious request because you send too many concurrent connections. , block your IP.
When we write a crawler, if there are 1,000 links to crawl, it is impossible to send out 1,000 concurrent links at the same time, right? We need to control the number of concurrencies, for example, 10 concurrencies, and then slowly capture these 1,000 links.
Doing this with async is easy.
This time we are going to introduce the
mapLimit(arr, limit, iterator, callback) interface of async. In addition, there is a commonly used interface for controlling the number of concurrent connections: queue(worker, concurrency). You can go to https://github.com/caolan/async#queueworker-concurrency for instructions. This time I won’t take you to crawl the website. Let’s focus on the knowledge point: controlling the number of concurrent connections.
By the way, another question is, when to use eventproxy and when to use async? Aren't they all used for asynchronous
process controlWhen you need to go to multiple sources (usually less than 10)
to summarize data, it is convenient to use eventproxy; when you need to use Use async when you want to queue, need to control the number of concurrency, or if you like functional programming thinking. Most scenarios are the former, so I personally use eventproxy most of the time. The main topic begins.
First, we forge a
fetchUrl(url, callback) function. The function of this function is that when you call it through <div class="code" style="position:relative; padding:0px; margin:0px;"><pre class="brush:php;toolbar:false">fetchUrl('http://www.baidu.com', function (err, content) {
// do something with `content`
});</pre><div class="contentsignin">Copy after login</div></div>, it will return http: //The page content of www.baidu.com returns.
Of course, the return content here is false, and the return delay is random. And when it is called, it will tell you how many places it is being called concurrently.
// 并发连接数的计数器
var concurrencyCount = 0;
var fetchUrl = function (url, callback) {
// delay 的值在 2000 以内，是个随机的整数
var delay = parseInt((Math.random() * 10000000) % 2000, 10);
concurrencyCount++;
console.log('现在的并发数是', concurrencyCount, '，正在抓取的是', url, '，耗时' + delay + '毫秒');
setTimeout(function () {
concurrencyCount--;
callback(null, url + ' html content');
}, delay);
};Let’s then forge a set of links
var urls = [];
for(var i = 0; i < 30; i++) {
urls.push('http://datasource_' + i);
}This set of links looks like this:
Next, we use async.mapLimit to concurrently crawl and obtain results.
async.mapLimit(urls, 5, function (url, callback) {
fetchUrl(url, callback);
}, function (err, result) {
console.log('final:');
console.log(result);
});The running output is like this:
It can be seen that at the beginning, the number of concurrent links starts to grow from 1, and when it grows to 5, It will no longer increase. When one of the tasks is completed, continue fetching. The number of concurrent connections is always limited to 5.
I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the php Chinese website!
Recommended reading:
How to replace the callback function with promise in nodeHow to use Vue better-scroll to implement alphabetical index navigationThe above is the detailed content of How to use async in node to control concurrency. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Using PHP to control the camera: analysis of the entire process from connection to shooting
Jul 30, 2023 pm 03:21 PM
Use PHP to control the camera: Analyze the entire process from connection to shooting. Camera applications are becoming more and more widespread, such as video calls, surveillance systems, etc. In web applications, we often need to control and operate cameras through PHP. This article will introduce how to use PHP to realize the entire process from camera connection to shooting. Confirm the connection status of the camera. Before starting to operate the camera, we first need to confirm the connection status of the camera. PHP provides an extension library video to operate the camera. We can pass the following code
Is async for es6 or es7?
Jan 29, 2023 pm 05:36 PM
async is es7. async and await are new additions to ES7 and are solutions for asynchronous operations; async/await can be said to be syntactic sugar for co modules and generator functions, solving js asynchronous code with clearer semantics. As the name suggests, async means "asynchronous". Async is used to declare that a function is asynchronous; there is a strict rule between async and await. Both cannot be separated from each other, and await can only be written in async functions.
How to delete node in nvm
Dec 29, 2022 am 10:07 AM
How to delete node with nvm: 1. Download "nvm-setup.zip" and install it on the C drive; 2. Configure environment variables and check the version number through the "nvm -v" command; 3. Use the "nvm install" command Install node; 4. Delete the installed node through the "nvm uninstall" command.
How to use express to handle file upload in node project
Mar 28, 2023 pm 07:28 PM
How to handle file upload? The following article will introduce to you how to use express to handle file uploads in the node project. I hope it will be helpful to you!
An in-depth analysis of Node's process management tool 'pm2”
Apr 03, 2023 pm 06:02 PM
This article will share with you Node's process management tool "pm2", and talk about why pm2 is needed, how to install and use pm2, I hope it will be helpful to everyone!
![How to disable media volume control popups [permanently]](https://img.php.cn/upload/article/000/000/164/168493981948502.png?x-oss-process=image/resize,m_fill,h_207,w_330)
How to disable media volume control popups [permanently]
May 24, 2023 pm 10:50 PM
When you use the corresponding shortcut key to fine-tune the volume level, a media volume control pop-up will appear on the screen. This can be annoying, so read on to find out different ways to permanently disable media volume control pop-ups. How to disable media volume control popup? 1. Click the Windows icon on the taskbar in Google Chrome, type chrome in the search bar at the top, and select the relevant search results to launch Google Chrome. Type or copy-paste the following into the address bar and press the key. Enterchrome://flags type media keys in the search box at the top and select Disable in the Hardware Media Key Handling drop-down list. Now exit the Google Chrome app and relaunch it. Google
Pi Node Teaching: What is a Pi Node? How to install and set up Pi Node?
Mar 05, 2025 pm 05:57 PM
Detailed explanation and installation guide for PiNetwork nodes This article will introduce the PiNetwork ecosystem in detail - Pi nodes, a key role in the PiNetwork ecosystem, and provide complete steps for installation and configuration. After the launch of the PiNetwork blockchain test network, Pi nodes have become an important part of many pioneers actively participating in the testing, preparing for the upcoming main network release. If you don’t know PiNetwork yet, please refer to what is Picoin? What is the price for listing? Pi usage, mining and security analysis. What is PiNetwork? The PiNetwork project started in 2019 and owns its exclusive cryptocurrency Pi Coin. The project aims to create a one that everyone can participate
What to do if npm node gyp fails
Dec 29, 2022 pm 02:42 PM
npm node gyp fails because "node-gyp.js" does not match the version of "Node.js". The solution is: 1. Clear the node cache through "npm cache clean -f"; 2. Through "npm install -g n" Install the n module; 3. Install the "node v12.21.0" version through the "n v12.21.0" command.
