Home Database Mysql Tutorial HBase新特性—Stripe Compaction

HBase新特性—Stripe Compaction

Jun 07, 2016 pm 04:30 PM
hbase stripe new features characteristic

借鉴于LevelDB、Cassandra的Compaction方法,https://issues.apache.org/jira/browse/HBASE-7667 提出了Stripe Compaction的方法。 Motivation: 1)过多Region会增大RS维护的开销,降低RS的读写性能。随着数据量的增大,在一定程度上增加Region个数,会提高

借鉴于LevelDB、Cassandra的Compaction方法,https://issues.apache.org/jira/browse/HBASE-7667 提出了Stripe Compaction的方法。

Motivation:
1)过多Region会增大RS维护的开销,降低RS的读写性能。随着数据量的增大,在一定程度上增加Region个数,会提高系统的吞吐率。然而,RS上服务的Region个数增多,增加了RS下内存维护的开销,尤其每个Store下都配置有一个MemStore,从而会造成频率更高的Flush操作,影响系统的读写性能。因此,如果能够提出更轻量级的mini-Region,不仅能够降低服务多个Region的开销,而且能够提升读写数据的效率。

2) Region Compaction容易”放大”。例如,Region区间为[1FFF,2FFF),在该区间内仅有[1FFF,21FF)区间有大量的写操作(put、delete),但是,在触及MajorCompaction条件时,却需要对所有的文件执行Major Compaction,从而引起大量的IO。

3) Region Split操作代价较大。

需要了解之前HBase的Compaction与Flush过程,可以参考:HBaseCompaction机制 以及 HBase Flush对读写的影响

Stripe-Compaction设计的核心思想:
1)对于Region下的rowkey区间进行二次切分,例如[1FFF,2FFF),切分成[1FFF,24FF),[24FF,2FFF)两个区间,每个区间成为Stripe。
2)Region下的数据文件分为Level-0和Level-1两层。其中Level-0主要用来存储临时的数据文件(例如使用bulkload或者执行mem flush操作之后的数据), Level-1层的数据是按照Stripe的分区来区分。
3)支持两种方式的配置:Mini-regions的个数设置、或者以Size-based为大小触发因子的自动切分机制。
4)容错机制。如果在Stripes之间存在空洞。那么可以根据在Store当中的设置,将所有的处于Level-1层的文件回归到Level-0重新进行compaction。
5)Get操作时,一个Row所涉及到文件有:MemStore、Level-0下所有文件、以及Level-1下对应Stripe区下的文件。根据Stack的意见,最终Level-0下的文件只是一个暂时的状态,大部分文件都位于Level-1 Stripe下,因此,这样随机读时,需要涉及到的文件更聚集。
6)Scan操作时,需要定位startrow即可。在扫描过程中,会按照Stripe的row区间的排序,依次进行。
7)Compaction,是Level-0上升到Level-1的过程,同时,在Level-1层次的数据,也会进行相关的合并。
8)在Split操作时,定位Rowkey区间的中心点,可以根据Stripe记录的位置,进一步查找,因此,使用预置的Stripe会有利于Split操作的进行,可以实现多数HFile文件直接拷贝到子Region目录,从而加快了Split操作的效率。

下面对于Cassandra以及LevelDB中使用的多层次Compaction算法做一个介绍。

1)分层式压缩方式将数据分成条个层,最底层的叫L0,其上分别是L1,L2….,每一层的数据大小是其上的那一层数据最大大小的10倍,其中最底层L0的大小为5M (可以配置)
2) 当level层次大于0时,同一层的各个文件之间的Rowkey区间不会重叠。所以在level n与level n+1的数据块进行合并时,可以明确的知道某个key值处在哪个数据块中,可以一个数据块一个数据块的合并,合并后生成新块就丢掉老块。不用一直到所有合并完成后才能删除老的块。
3)整体执行流程是从L0->L1->L2,依次合并的过程,如下图所示。

compaction由上图,我们可以得知,越是level较低的块,它的数据就越新,在满足向下归约合并的过程中,就会按照文件的Rowkey的区间,进行合并,去除多余的版本,或者执行相关删除操作。因此,在读请求最极端的情况下,从Level0开始读数据,一直读到最下层Level n。

这种Compaction的优势在于:
1)大部分的读操作如果有LRU特性,都会落入较低的Level上。因此,数据越"热",Level就越低。从而有利于未来HFile多种存储介质的定位问题。
2)在合并的过程中,仅需在由上到下的部分文件参与,而不是要对所有文件执行Compaction操作。这样会加快Compaction执行的效率。

劣势在于,如果层次太多,在递归合并的过程中,容易造成某个区间的Compaction风暴,影响该区间数据操作的吞吐。
因此,HBase-Stripe Compaction的方案中,只有两层,Level 0和Level1,这种方法在保留分层压缩的优势的同时,降低了总文件个数,有利于RS执行Split、Merge等操作。

参考文献:
[1] HBase-7667 https://issues.apache.org/jira/browse/HBASE-7667

本系列文章属于Binos_ICT在Binospace个人技术博客原创,原文链接为http://www.binospace.com/index.php/hbase-new-features-stripe-compaction,未经允许,不得转载。

From Binospace, post HBase新特性—Stripe Compaction

文章的脚注信息由WordPress的wp-posturl插件自动生成


Copyright © 2008
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Master the key concepts of Spring MVC: Understand these important features Master the key concepts of Spring MVC: Understand these important features Dec 29, 2023 am 09:14 AM

Understand the key features of SpringMVC: To master these important concepts, specific code examples are required. SpringMVC is a Java-based web application development framework that helps developers build flexible and scalable structures through the Model-View-Controller (MVC) architectural pattern. web application. Understanding and mastering the key features of SpringMVC will enable us to develop and manage our web applications more efficiently. This article will introduce some important concepts of SpringMVC

PHP 8.3 released: new features at a glance PHP 8.3 released: new features at a glance Nov 27, 2023 pm 12:52 PM

PHP8.3 released: Overview of new features As technology continues to develop and needs change, programming languages ​​are constantly updated and improved. As a scripting language widely used in web development, PHP has been constantly improving to provide developers with more powerful and efficient tools. The recently released PHP 8.3 version brings many long-awaited new features and improvements. Let’s take a look at an overview of these new features. Initialization of non-null properties In past versions of PHP, if a class property was not explicitly assigned a value, its value

A guide to learn the new features of PHP8 and gain an in-depth understanding of the latest technology A guide to learn the new features of PHP8 and gain an in-depth understanding of the latest technology Dec 23, 2023 pm 01:16 PM

An in-depth analysis of the new features of PHP8 to help you master the latest technology. As time goes by, the PHP programming language has been constantly evolving and improving. The recently released PHP8 version provides developers with many exciting new features and improvements, bringing more convenience and efficiency to our development work. In this article, we will analyze the new features of PHP8 in depth and provide specific code examples to help you better master these latest technologies. JIT compiler PHP8 introduces JIT (Just-In-Time) compilation

Are there any class-like object-oriented features in Golang? Are there any class-like object-oriented features in Golang? Mar 19, 2024 pm 02:51 PM

There is no concept of a class in the traditional sense in Golang (Go language), but it provides a data type called a structure, through which object-oriented features similar to classes can be achieved. In this article, we'll explain how to use structures to implement object-oriented features and provide concrete code examples. Definition and use of structures First, let's take a look at the definition and use of structures. In Golang, structures can be defined through the type keyword and then used where needed. Structures can contain attributes

Choose the applicable Go version, based on needs and features Choose the applicable Go version, based on needs and features Jan 20, 2024 am 09:28 AM

With the rapid development of the Internet, programming languages ​​are constantly evolving and updating. Among them, Go language, as an open source programming language, has attracted much attention in recent years. The Go language is designed to be simple, efficient, safe, and easy to develop and deploy. It has the characteristics of high concurrency, fast compilation and memory safety, making it widely used in fields such as web development, cloud computing and big data. However, there are currently different versions of the Go language available. When choosing a suitable Go language version, we need to consider both requirements and features. head

What are the new features of php8 What are the new features of php8 Sep 25, 2023 pm 01:34 PM

New features of php8 include JIT compiler, type deduction, named parameters, union types, properties, error handling improvements, asynchronous programming support, new standard library functions and anonymous class extensions. Detailed introduction: 1. JIT compiler, PHP8 introduces the JIT compiler, which is an important performance improvement. The JIT compiler can compile and optimize some high-frequency execution codes in real time, thereby improving the running speed; 2. Type derivation , PHP8 introduces the type inference function, allowing developers to automatically deduce the type of variables when declaring variables, etc.

C++ function types and characteristics C++ function types and characteristics Apr 11, 2024 pm 03:30 PM

C++ functions have the following types: simple functions, const functions, static functions, and virtual functions; features include: inline functions, default parameters, reference returns, and overloaded functions. For example, the calculateArea function uses π to calculate the area of ​​a circle of a given radius and returns it as output.

Interpretation of new features of Go language: making programming more efficient Interpretation of new features of Go language: making programming more efficient Mar 10, 2024 pm 12:27 PM

[Interpretation of new features of Go language: To make programming more efficient, specific code examples are needed] In recent years, Go language has attracted much attention in the field of software development, and its simple and efficient design concept has attracted more and more developers. As a statically typed programming language, Go language continues to introduce new features to improve development efficiency and simplify the code writing process. This article will provide an in-depth explanation of the latest features of the Go language and discuss how to experience the convenience brought by these new features through specific code examples. Modular development (GoModules) Go language from 1

See all articles