Home Database Mysql Tutorial hadoop 2.0 参数调优小结

hadoop 2.0 参数调优小结

Jun 07, 2016 pm 03:08 PM
hadoop parameter Tuning

1.io.file.buffer.size 用来设置IO操作的缓存大小,单位为byte,默认为4KB,建议设置成64KB,即65536 2.dfs.balance.bandwithPerSec 集群做balance时dn间的带宽大小,在做balance时可以通过-threshold指定平衡的阀,但是平衡时的带宽是由此参数静态指定的,

1.io.file.buffer.size

用来设置IO操作的缓存大小,单位为byte,默认为4KB,建议设置成64KB,即65536

2.dfs.balance.bandwithPerSec

集群做balance时dn间的带宽大小,在做balance时可以通过-threshold指定平衡的阀值,但是平衡时的带宽是由此参数静态指定的,因此在配置集群时应该综合考虑集群所在网络环境的带宽和繁忙程度,因为一旦设置,需要调整时就要重启集群,而设置的过低将导致平衡耗时较长。

3.dfs.datanode.du.reserved

此参数设置的目的是为了给mapred的local计算预留一些空间,因为当datanode发现所有挂载的磁盘都写满时,将进入只读模式,此时任务将无法运行。建议每个磁盘预留10GB给mapred中间数据,单位为byte,即10737418240

4.dfs.namenode.handler.count

nn处理dn的rpc和守护进程时需要使用工作线程池的大小,过小容易导致dn连接nn超时或者被拒,过大导致rpc延迟增大,数值设置建议lg(N)*20,N为集群规模。

可使用python脚本计算:

import math;
print int (math.log(N)*20)
Copy after login
5.dfs.datanode.failed.volumes.tolerated

dn上挂载的disk出现故障时,默认整个dn退出服务,此时nn将重分布block,代价过高,建议此值设置为1,即允许最多1个disk发生故障时,dn仍继续服务,此时需要对dn上的日志进行监控,以便尽早发现故障更换磁盘。

6.fs.trash.interval

设置回收站保留时间,列出此参数的目的是为了说明即使在设置了回收站保留时间的情况下,在删除数据时如何跳过回收站直接删除,即类似windows中的shift+delete功能:

hdfs dfs -rm -skipTrash yourfilename
Copy after login


Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

C++ function parameter type safety check C++ function parameter type safety check Apr 19, 2024 pm 12:00 PM

C++ parameter type safety checking ensures that functions only accept values ​​of expected types through compile-time checks, run-time checks, and static assertions, preventing unexpected behavior and program crashes: Compile-time type checking: The compiler checks type compatibility. Runtime type checking: Use dynamic_cast to check type compatibility, and throw an exception if there is no match. Static assertion: Assert type conditions at compile time.

i9-12900H parameter evaluation list i9-12900H parameter evaluation list Feb 23, 2024 am 09:25 AM

i9-12900H is a 14-core processor. The architecture and technology used are all new, and the threads are also very high. The overall work is excellent, and some parameters have been improved. It is particularly comprehensive and can bring users Excellent experience. i9-12900H parameter evaluation review: 1. i9-12900H is a 14-core processor, which adopts the q1 architecture and 24576kb process technology, and has been upgraded to 20 threads. 2. The maximum CPU frequency is 1.80! 5.00ghz, which mainly depends on the workload. 3. Compared with the price, it is very suitable. The price-performance ratio is very good, and it is very suitable for some partners who need normal use. i9-12900H parameter evaluation and performance running scores

C++ program to find the value of the inverse hyperbolic sine function taking a given value as argument C++ program to find the value of the inverse hyperbolic sine function taking a given value as argument Sep 17, 2023 am 10:49 AM

Hyperbolic functions are defined using hyperbolas instead of circles and are equivalent to ordinary trigonometric functions. It returns the ratio parameter in the hyperbolic sine function from the supplied angle in radians. But do the opposite, or in other words. If we want to calculate an angle from a hyperbolic sine, we need an inverse hyperbolic trigonometric operation like the hyperbolic inverse sine operation. This course will demonstrate how to use the hyperbolic inverse sine (asinh) function in C++ to calculate angles using the hyperbolic sine value in radians. The hyperbolic arcsine operation follows the following formula -$$\mathrm{sinh^{-1}x\:=\:In(x\:+\:\sqrt{x^2\:+\:1})}, Where\:In\:is\:natural logarithm\:(log_e\:k)

The open source model wins GPT-4 for the first time! Arena's latest battle report has sparked heated debate, Karpathy: This is the only list I trust The open source model wins GPT-4 for the first time! Arena's latest battle report has sparked heated debate, Karpathy: This is the only list I trust Apr 10, 2024 pm 03:16 PM

An open source model that can beat GPT-4 has appeared! The latest battle report of the large model arena: the 104 billion parameter open source model CommandR+ climbed to 6th place, tying with GPT-4-0314 and surpassing GPT-4-0613. Image This is also the first open-weight model to beat GPT-4 in the large model arena. The large model arena is one of the only test benchmarks that the master Karpathy trusts. Image CommandR+ from AI unicorn Cohere. The co-founder and CEO of this large model startup is none other than Aidan Gomez, the youngest author of Transformer (referred to as the wheat harvester). As soon as this battle report came out, another wave of big model clubs started

Advanced usage of reference parameters and pointer parameters in C++ functions Advanced usage of reference parameters and pointer parameters in C++ functions Apr 21, 2024 am 09:39 AM

Reference parameters in C++ functions (essentially variable aliases, modifying the reference modifies the original variable) and pointer parameters (storing the memory address of the original variable, modifying the variable by dereferencing the pointer) have different usages when passing and modifying variables. Reference parameters are often used to modify original variables (especially large structures) to avoid copy overhead when passed to constructors or assignment operators. Pointer parameters are used to flexibly point to memory locations, implement dynamic data structures, or pass null pointers to represent optional parameters.

vivox200ultra parameters and price details vivox200ultra parameters and price details Jun 28, 2024 pm 01:23 PM

The latest official news of vivox200ultra has exposed the parameters and price details of vivox200ultra. It is reported that vivox200ultra will be equipped with a 10x periscope super telephoto lens, and the price starts at about 6999 yuan. It can be seen that it occupies an absolute advantage in photography performance. The following are the parameters and prices of vivox200ultra Come and see the details. 1. Parameter configuration details of vivox200ultra 1. Vivox200ultra rendering From the vivo X200 Ultra rendering, the front of the phone adopts a borderless full-screen design, and the visual effect of the entire front of the phone can be said to be very invincible. 2. vivox200ultra has Blackhawk frame

Detailed explanation of tuning practices to improve Go language website access speed Detailed explanation of tuning practices to improve Go language website access speed Aug 26, 2023 pm 07:27 PM

Detailed explanation of tuning practices to improve Go language website access speed Abstract: In the rapidly developing Internet era, website access speed has become one of the important factors for users to choose a website. This article will introduce in detail how to use Go language to optimize website access speed, including practical experience in optimizing network requests, using cache, and concurrent processing. The article will also provide code examples to help readers better understand and apply these optimization techniques. 1. Optimize network requests In website development, network requests are an inevitable link. And optimizing network requests can

How to perform system tuning and performance testing of Linux systems How to perform system tuning and performance testing of Linux systems Nov 07, 2023 am 11:33 AM

Operating system performance optimization is one of the keys to ensuring efficient system operation. In Linux systems, we can perform performance tuning and testing through various methods to ensure the best performance of the system. This article will introduce how to perform system tuning and performance testing of Linux systems, and provide corresponding specific code examples. 1. System tuning System tuning is to optimize the performance of the system by adjusting various parameters of the system. The following are some common system tuning methods: 1. Modify the kernel parameters. The kernel parameters of the Linux system control the system operation.

See all articles