Home Database Mysql Tutorial HBase解决Region Server Compact过程占用大量网络出口带宽的问题

HBase解决Region Server Compact过程占用大量网络出口带宽的问题

Jun 07, 2016 pm 05:56 PM
compact hbase server solve

HBase解决Region Server Compact过程占用大量网络出口带宽的问题 作者: 大圆那些事 | 文章可以转载,请以超链接形式标明文章原始出处和作者信息 网址: 推荐:《HBase权威指南》 HBase 0.92版本之后,Region Server的Compact过程根据待合并的文件大小分为smal

HBase解决Region Server Compact过程占用大量网络出口带宽的问题

作者: 大圆那些事 | 文章可以转载,请以超链接形式标明文章原始出处和作者信息

网址:

推荐:《HBase权威指南》

HBase 0.92版本之后,Region Server的Compact过程根据待合并的文件大小分为small compaction和large compaction两种,由此可能导致在集群写入量大的时候Compact占用过多的网络出口带宽。本文将详细描述集群使用过程中遇到这一问题的排查过程及其解决方法。

1. 发现问题

HBase集群(版本为0.94.0)运行过程中,发现5台Region Server的网络出口带宽经常维持在100MB/s以上,香港虚拟主机,接近到网卡的极限;同时Region Server的机器load负载也很高,高峰时候能够达到30~50。

2. 排查问题

1、集群实际运行过程中,观察到Region Server服务端的网卡,平均每台写入流量大概60MB/s(此时写入量已经很大了);读出流量90MB/s,有时甚至突破100MB/s(注:每台机器都是千兆网卡);

2、观察实际的写入数据量在每秒5w tps左右,单条记录平均大小为1KB,大概会占用50MB/s左右的网卡入口带宽请求量,和观察到的现象一致;

3、观察查询量在每秒6w qps左右,单条记录平均大小为1KB,大概会占用60MB/s左右的网卡出口带宽请求量,奇怪的是实际观察到有接近甚至超过100MB/s的网络出口带宽请求量,多出了40MB/s左右的网络出口带宽

4、经分析排查确定导致上述过程的原因,可能是HBase服务端由于写入量过大频繁触发compaction过程,而compaction是需要读HBase数据的,因此占据了相当部分的网络出口带宽;

5、结合对相关源码org/apache/hadoop/hbase/regionserver/CompactSplitThread.java的分析,决定对HBase集群配置做出变更(具体见下一小节),主要目的是减少compaction的发生;

6、接下来,观察到Region Server的网络利用率明显降低,一般进出口带宽能维持在70MB/s以下。

3. 解决问题

HBase 0.92版本之后增加了关于compact的配置选项,compact分为small compaction和large compaction两个线程池执行(默认都是各有1个线程,虚拟主机,具体源代码见:org/apache/hadoop/hbase/regionserver/CompactSplitThread.java),由于compact过程需要从HBase集群读取数据,因此实际运行中导致了compact占用大量网络出口流量,解决方案为选择性地关闭small compaction或large compaction。有以下两种变更方式均可解决:

1)方案一

(1) 修改hbase.regionserver.thread.compaction.throttle为一个很大的值(如50GB),香港虚拟主机,强制让所有compact都变为small compaction,减少compact的压力;

(2) 将small compaction和large compaction线程数均设置为1,减少compact的压力(可不配置,系统默认也会将其初始化为1)。

操作步骤:

准备hbase-site.xml文件,添加或修改如下选项:

hbase.regionserver.thread.compaction.throttle53687091200hbase.regionserver.thread.compaction.small1hbase.regionserver.thread.compaction.large1

重启集群使配置生效。

2)方案二

将small compaction线程数均设置为0,从而关闭small compaction,只剩下large compaction,也可减少compact的压力。

操作步骤:

准备hbase-site.xml文件,添加或修改如下选项:

hbase.regionserver.thread.compaction.small0

重启集群使配置生效。

posted on

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Solution to the problem that Win11 system cannot install Chinese language pack Solution to the problem that Win11 system cannot install Chinese language pack Mar 09, 2024 am 09:48 AM

Solution to the problem that Win11 system cannot install Chinese language pack With the launch of Windows 11 system, many users began to upgrade their operating system to experience new functions and interfaces. However, some users found that they were unable to install the Chinese language pack after upgrading, which troubled their experience. In this article, we will discuss the reasons why Win11 system cannot install the Chinese language pack and provide some solutions to help users solve this problem. Cause Analysis First, let us analyze the inability of Win11 system to

Five tips to teach you how to solve the problem of Black Shark phone not turning on! Five tips to teach you how to solve the problem of Black Shark phone not turning on! Mar 24, 2024 pm 12:27 PM

As smartphone technology continues to develop, mobile phones play an increasingly important role in our daily lives. As a flagship phone focusing on gaming performance, the Black Shark phone is highly favored by players. However, sometimes we also face the situation that the Black Shark phone cannot be turned on. At this time, we need to take some measures to solve this problem. Next, let us share five tips to teach you how to solve the problem of Black Shark phone not turning on: Step 1: Check the battery power. First, make sure your Black Shark phone has enough power. It may be because the phone battery is exhausted

The driver cannot be loaded on this device. How to solve it? (Personally tested and valid) The driver cannot be loaded on this device. How to solve it? (Personally tested and valid) Mar 14, 2024 pm 09:00 PM

Everyone knows that if the computer cannot load the driver, the device may not work properly or interact with the computer correctly. So how do we solve the problem when a prompt box pops up on the computer that the driver cannot be loaded on this device? The editor below will teach you two ways to easily solve the problem. Unable to load the driver on this device Solution 1. Search for "Kernel Isolation" in the Start menu. 2. Turn off Memory Integrity, and it will prompt "Memory Integrity has been turned off. Your device may be vulnerable." Click behind to ignore it, and it will not affect the use. 3. The problem can be solved after restarting the machine.

How to solve the problem of automatically saving pictures when publishing on Xiaohongshu? Where is the automatically saved image when posting? How to solve the problem of automatically saving pictures when publishing on Xiaohongshu? Where is the automatically saved image when posting? Mar 22, 2024 am 08:06 AM

With the continuous development of social media, Xiaohongshu has become a platform for more and more young people to share their lives and discover beautiful things. Many users are troubled by auto-save issues when posting images. So, how to solve this problem? 1. How to solve the problem of automatically saving pictures when publishing on Xiaohongshu? 1. Clear the cache First, we can try to clear the cache data of Xiaohongshu. The steps are as follows: (1) Open Xiaohongshu and click the "My" button in the lower right corner; (2) On the personal center page, find "Settings" and click it; (3) Scroll down and find the "Clear Cache" option. Click OK. After clearing the cache, re-enter Xiaohongshu and try to post pictures to see if the automatic saving problem is solved. 2. Update the Xiaohongshu version to ensure that your Xiaohongshu

How to solve Chinese garbled characters in Linux How to solve Chinese garbled characters in Linux Feb 21, 2024 am 10:48 AM

The Linux Chinese garbled problem is a common problem when using Chinese character sets and encodings. Garbled characters may be caused by incorrect file encoding settings, system locale not being installed or set, and terminal display configuration errors, etc. This article will introduce several common workarounds and provide specific code examples. 1. Check the file encoding setting. Use the file command to view the file encoding. Use the file command in the terminal to view the encoding of the file: file-ifilename. If there is "charset" in the output

Share the method to solve the problem that PyCharm cannot be opened Share the method to solve the problem that PyCharm cannot be opened Feb 22, 2024 am 09:03 AM

Title: How to solve the problem that PyCharm cannot be opened. PyCharm is a powerful Python integrated development environment, but sometimes we may encounter the problem that PyCharm cannot be opened. In this article, we'll share some common workarounds and provide specific code examples. Hope this helps those who encounter this problem. Method 1: Clear the cache Sometimes PyCharm’s cache files may cause the program to fail to open normally. We can try clearing the cache to solve this problem. Tool

How to solve the problem of default gateway automatically disappearing How to solve the problem of default gateway automatically disappearing Feb 24, 2024 pm 04:18 PM

How to solve the problem that the default gateway disappears automatically. In modern society, the Internet has become an indispensable part of people's lives. Whether for work or entertainment, we all need stable network connections to complete various tasks. The default gateway is one of the key elements connecting the local network to the external Internet. However, sometimes we may encounter the problem that the default gateway disappears automatically, resulting in the inability to access the Internet. So, how should we solve this problem when the default gateway disappears? First, we should clarify the concept of default gateway. The default gateway is a network route

Fujifilm X-M5 price expectations rise as leaked selfie display adds complexity to the formerly affordable camera line Fujifilm X-M5 price expectations rise as leaked selfie display adds complexity to the formerly affordable camera line Sep 07, 2024 am 09:34 AM

Fujifilm fans were recently very excited at the prospect of the X-T50, since it presented a relaunch of the budget-oriented Fujifilm X-T30 II that had become quite popular in the sub-$1,000 APS-C category. Unfortunately, as the Fujifilm X-T50's launc

See all articles