Hadoop For Windows
作者: Dong | 新浪微博： 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/mapreduce/hadoop-for-windows/ 本博客的文章集合:http://dongxicheng.org/recommend/ 重大消息：我的Hadoop新书《Hadoop
作者:Dong | 新浪微博：西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明
网址:http://dongxicheng.org/mapreduce/hadoop-for-windows/
本博客的文章集合:http://dongxicheng.org/recommend/
重大消息：我的Hadoop新书《Hadoop技术内幕：深入解析MapReduce架构设计与实现原理》已经开始在各大网站销售了，购书链接地址： 当当购书网址，京东购书网址，卓越购书网址。新书官方宣传主页： http://hadoop123.com/。
Hadoop从一出生就只面向Linux操作系统，且一直未曾考虑支持Windows操作系统。但随着Hadoop的普及，微软不得不开始为Hadoop增加patch以推进Windows成为生产环境。经过各方努力，Apache Hadoop已经同意将这些windows patch融入到核心代码中（http://hortonworks.com/blog/hadoop-in-windows/），目前，Hadoop svn上已经增加了两个分支：branch-1-win和branch-trunk-win，且Hortonworks联合微软发布了Hadoop windows安装版（http://hortonworks.com/blog/installing-hadoop-on-windows/）。
对于Hadoop平台支持问题，很多读者可能有这样的疑问：既然Hadoop是采用JAVA语言编写的，而JAVA具有跨平台特性，那么为什么官方只推荐Linux作为生产环境，而不推荐Windows呢？ 在本文中，我将简单介绍目前Hadoop的哪些特性不能够得到windows平台的支持。
总体上讲，Hadoop中的以下几个特性无法得到windows平台的支持，换句话说，为了支持windows平台，必须修改这几个特性的实现（https://issues.apache.org/jira/browse/HADOOP-8079）：
（1）管理脚本。 在Hadoop 1.0的安装目录中有一个bin目录，里面存放着各种shell管理脚本。很明显，对于Windows而言，无法运行直接这些shell脚本，需要对这些脚本进行修改。 在Hadoop Windows中，重新添加了后缀为.cmd的一系列管理脚本。
（2）HDFS权限管理。 HDFS的文件权限管理采用了Linux操作系统的权限管理，如果移植到Hadoop windows上，需要将对应的文件权限管理改为windows中的权限管理机制。
（3）Hadoop native library。 Hadoop中有一种native library机制，允许用户直接采用本地库（一般为C/C++编写）进行某种操作以优化性能，比如压缩解压库，Native IO等。如果将Hadoop移植到windows，这些对应的库需要跟着移植。
（4）Native TaskController。 TaskTracker启动时，会调用一个插拔式组件TaskController运行任务，TaskController一般会添加一些任务控制机制，比如设定任务启动用户。默认实现是DefaultTaskController，另外一个可使用的是LinuxTaskController，该TaskController将以提交用户的身份运行任务（默认情况下是以TaskTracker启动用户运行任务）。
（5）单元测试用例。原始的Hadoop很多测试用例无法再windows上运行通过，需要进行修改。
（6）其他零散问题。比如路径问题、资源泄露等（具体参考：https://issues.apache.org/jira/browse/HADOOP-8079）。
参考资料：
http://hortonworks.com/blog/hadoop-in-windows/
https://issues.apache.org/jira/browse/HADOOP-8079
原创文章，转载请注明： 转载自董的博客
本文链接地址: http://dongxicheng.org/mapreduce/hadoop-for-windows/
作者：Dong，作者介绍：http://dongxicheng.org/about/
本博客的文章集合:http://dongxicheng.org/recommend/
Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Can I install mysql on Windows 7
Apr 08, 2025 pm 03:21 PM
Yes, MySQL can be installed on Windows 7, and although Microsoft has stopped supporting Windows 7, MySQL is still compatible with it. However, the following points should be noted during the installation process: Download the MySQL installer for Windows. Select the appropriate version of MySQL (community or enterprise). Select the appropriate installation directory and character set during the installation process. Set the root user password and keep it properly. Connect to the database for testing. Note the compatibility and security issues on Windows 7, and it is recommended to upgrade to a supported operating system.
How to solve mysql cannot connect to local host
Apr 08, 2025 pm 02:24 PM
The MySQL connection may be due to the following reasons: MySQL service is not started, the firewall intercepts the connection, the port number is incorrect, the user name or password is incorrect, the listening address in my.cnf is improperly configured, etc. The troubleshooting steps include: 1. Check whether the MySQL service is running; 2. Adjust the firewall settings to allow MySQL to listen to port 3306; 3. Confirm that the port number is consistent with the actual port number; 4. Check whether the user name and password are correct; 5. Make sure the bind-address settings in my.cnf are correct.
MySQL can't be installed after downloading
Apr 08, 2025 am 11:24 AM
The main reasons for MySQL installation failure are: 1. Permission issues, you need to run as an administrator or use the sudo command; 2. Dependencies are missing, and you need to install relevant development packages; 3. Port conflicts, you need to close the program that occupies port 3306 or modify the configuration file; 4. The installation package is corrupt, you need to download and verify the integrity; 5. The environment variable is incorrectly configured, and the environment variables must be correctly configured according to the operating system. Solve these problems and carefully check each step to successfully install MySQL.
Solutions to the errors reported by MySQL on a specific system version
Apr 08, 2025 am 11:54 AM
The solution to MySQL installation error is: 1. Carefully check the system environment to ensure that the MySQL dependency library requirements are met. Different operating systems and version requirements are different; 2. Carefully read the error message and take corresponding measures according to prompts (such as missing library files or insufficient permissions), such as installing dependencies or using sudo commands; 3. If necessary, try to install the source code and carefully check the compilation log, but this requires a certain amount of Linux knowledge and experience. The key to ultimately solving the problem is to carefully check the system environment and error information, and refer to the official documents.
Unable to access mysql from terminal
Apr 08, 2025 pm 04:57 PM
Unable to access MySQL from the terminal may be due to: MySQL service not running; connection command error; insufficient permissions; firewall blocks connection; MySQL configuration file error.
How to copy and paste mysql
Apr 08, 2025 pm 07:18 PM
Copy and paste in MySQL includes the following steps: select the data, copy with Ctrl C (Windows) or Cmd C (Mac); right-click at the target location, select Paste or use Ctrl V (Windows) or Cmd V (Mac); the copied data is inserted into the target location, or replace existing data (depending on whether the data already exists at the target location).
Can vs code run in Windows 8
Apr 15, 2025 pm 07:24 PM
VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.
Can mysql run on Windows
Apr 08, 2025 pm 01:54 PM
Running MySQL on Windows is feasible, but challenges such as port conflicts, permission issues, and environment variable settings need to be considered. Installation issues can be solved by customizing configuration files, adjusting user permissions, and setting environment variables correctly. Additionally, the appropriate storage engine should be selected, tweaked configuration files, and SSDs should be used to optimize performance.
