浅谈Borg/YARN/Mesos/Torca/Corona一类系统
作者: Dong | 新浪微博： 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/mapreduce-nextgen/borg-yarn-mesos-torca-corona/ Borg（来自Google），YARN（来自Apache，属于Hadoop下面的一个分支，
作者:Dong | 新浪微博：西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明
网址:http://dongxicheng.org/mapreduce-nextgen/borg-yarn-mesos-torca-corona/
Borg（来自Google），YARN（来自Apache，属于Hadoop下面的一个分支，开源），Mesos（来自Twitter，开源），Torca（来自腾讯搜搜），Corona（来自Facebook，开源）一类系统被称为资源统一管理系统或者资源统一调度系统，它们是大数据时代的必然产物。概括起来，这类系统设计动机是解决以下两类问题：
（1） 提高集群资源利用率
在大数据时代，为了存储和处理海量数据，需要规模较大的服务器集群或者数据中心，一般说来，这些集群上运行着数量众多类型纷杂的应用程序和服务，比如离线作业，流式作业，迭代式作业，crawler server，web server等，传统的做法是，每种类型的作业或者服务对应一个单独的集群，以避免相互干扰。这样，集群被分割成数量众多的小集群，有的集群运行Hadoop，有的运行Storm，有的运行Spark，有的运行web server，然而，由于不同类型的作业/服务需要的资源量不同，因此，这些小集群的利用率通常很不均衡，有的集群满负荷、资源紧张，而另外一些则长时间闲置、资源利用率极低，为了提高资源整体利用率，一种解决方案是将这些小集群合并成一个大集群，让它们共享这个大集群的资源，并由一个资源统一调度系统进行资源管理和分配，这就诞生了Borg，YARN，Mesos，Torca，Corona。从集群共享角度看，这类系统实际上将公司的所有硬件资源抽象成一个台大型计算机，供所有用户使用。
（2） 服务自动化部署
一旦将所有计算资源抽象成一个“大型计算机”后，就会产生一个问题：公司的各种服务如何进行部署？同样，Borg/YARN/Mesos/Torca/Corona一类系统需要具备服务自动化部署的功能，因此，从服务部署的角度看，这类系统实际上是服务统一管理系统，这类系统提供服务资源申请，服务自动化部署，服务容错等动能。
以上只是简单的介绍了这一类系统的设计动机和产生背景，接下来从两个角度解析这类系统。
角度一：数据中心编程
任何一个公司内部所有的硬件资源均可看做一个数据中心，通过Borg/YARN/Mesos/Torca/Corona一类系统对这些资源进行统一管理后，用户所有的程序和服务将通过一个统一入口进入数据中心，并由这类系统为之分配资源、监控程序和服务运行状态，并在失败时启用必要的容错机制，汇报程序的执行进度等，而至于应用程序或者服务运行在具体哪台机器上，所在机器的ip、端口号是什么，则用户无需管理，全部交由统一管理系统进行管理（用户也许可以查询到）。
具体说来，采用此类系统之后，当用户执行应用程序或者部署服务时，只需通过一个配置文件描述应用程序或服务需要的资源（比如CPU、内存、磁盘、操作系统类型等）、待执行的命令、依赖的外部文件等信息，然后通过一个客户端提交到Borg/YARN/Mesos/Torca/Corona上，剩下的工作则完全交给系统。
角度二：生态系统
从另外一个角度看，Borg/YARN/Mesos/Torca/Corona一类系统可以为公司构建一个内部的生态系统，所有应用程序和服务可以“和平而友好”地运行在该生态系统上。有了这类系统之后，你不必忧愁使用Hadoop的哪个版本，是Hadoop 0.20.2还是 Hadoop 1.0，你也不必为选择何种计算模型而苦恼，因此各种软件版本，各种计算模型可以一起运行在一台“超级计算机”上了。
从开源角度看，YARN的提出，从一定程度上弱化了多计算框架的优劣之争。YARN是在Hadoop MapReduce基础上演化而来的，在MapReduce时代，很多人批评MapReduce不适合迭代计算和流失计算，于是出现了Spark和Storm等计算框架，而这些系统的开发者则在自己的网站上或者论文里与MapReduce对比，鼓吹自己的系统多么先进高效，而出现了YARN之后，则形势变得明朗：MapReduce只是运行在YARN之上的一类应用程序抽象，Spark和Storm本质上也是，他们只是针对不同类型的应用开发的，没有优劣之别，各有所长，合并共处，而且，今后所有计算框架的开发，不出意外的话，也应是在YARN之上。这样，一个以YARN为底层资源管理平台，多种计算框架运行于其上的生态系统诞生了。
这一篇和上一篇《多集群下资源共享方案介绍》内容有些重复，我一直在反反复复强调资源管理/调度系统，目的只有一个，我想告诉大家：YARN时代来了！(所有的软件和服务都在往YARN上移，包括MapReduce，Spark，Storm，MPI，HBase部署等…..)
参考资料：
Borg：http://www.quora.com/What-is-Borg-at-Google
YARN：http://dongxicheng.org/mapreduce-nextgen/nextgen-mapreduce-introduction/
Mesos：http://www.mesosproject.org/
Torca：http://djt.qq.com/thread-29998-1-2.html
Mesos与YARN比较：http://dongxicheng.org/mapreduce-nextgen/mesos_vs_yarn/
Corona：http://dongxicheng.org/hadoop-corona/hadoop-corona/
原创文章，转载请注明： 转载自董的博客
本文链接地址: http://dongxicheng.org/mapreduce-nextgen/borg-yarn-mesos-torca-corona/
作者：Dong，作者介绍：http://dongxicheng.org/about/
Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1662
14
1418
52
1311
25
1261
29
1234
24
When might a full table scan be faster than using an index in MySQL?
Apr 09, 2025 am 12:05 AM
Full table scanning may be faster in MySQL than using indexes. Specific cases include: 1) the data volume is small; 2) when the query returns a large amount of data; 3) when the index column is not highly selective; 4) when the complex query. By analyzing query plans, optimizing indexes, avoiding over-index and regularly maintaining tables, you can make the best choices in practical applications.
Can I install mysql on Windows 7
Apr 08, 2025 pm 03:21 PM
Yes, MySQL can be installed on Windows 7, and although Microsoft has stopped supporting Windows 7, MySQL is still compatible with it. However, the following points should be noted during the installation process: Download the MySQL installer for Windows. Select the appropriate version of MySQL (community or enterprise). Select the appropriate installation directory and character set during the installation process. Set the root user password and keep it properly. Connect to the database for testing. Note the compatibility and security issues on Windows 7, and it is recommended to upgrade to a supported operating system.
Can mysql and mariadb coexist
Apr 08, 2025 pm 02:27 PM
MySQL and MariaDB can coexist, but need to be configured with caution. The key is to allocate different port numbers and data directories to each database, and adjust parameters such as memory allocation and cache size. Connection pooling, application configuration, and version differences also need to be considered and need to be carefully tested and planned to avoid pitfalls. Running two databases simultaneously can cause performance problems in situations where resources are limited.
MySQL: Simple Concepts for Easy Learning
Apr 10, 2025 am 09:29 AM
MySQL is an open source relational database management system. 1) Create database and tables: Use the CREATEDATABASE and CREATETABLE commands. 2) Basic operations: INSERT, UPDATE, DELETE and SELECT. 3) Advanced operations: JOIN, subquery and transaction processing. 4) Debugging skills: Check syntax, data type and permissions. 5) Optimization suggestions: Use indexes, avoid SELECT* and use transactions.
Laravel Eloquent ORM in Bangla partial model search)
Apr 08, 2025 pm 02:06 PM
LaravelEloquent Model Retrieval: Easily obtaining database data EloquentORM provides a concise and easy-to-understand way to operate the database. This article will introduce various Eloquent model search techniques in detail to help you obtain data from the database efficiently. 1. Get all records. Use the all() method to get all records in the database table: useApp\Models\Post;$posts=Post::all(); This will return a collection. You can access data using foreach loop or other collection methods: foreach($postsas$post){echo$post->
RDS MySQL integration with Redshift zero ETL
Apr 08, 2025 pm 07:06 PM
Data Integration Simplification: AmazonRDSMySQL and Redshift's zero ETL integration Efficient data integration is at the heart of a data-driven organization. Traditional ETL (extract, convert, load) processes are complex and time-consuming, especially when integrating databases (such as AmazonRDSMySQL) with data warehouses (such as Redshift). However, AWS provides zero ETL integration solutions that have completely changed this situation, providing a simplified, near-real-time solution for data migration from RDSMySQL to Redshift. This article will dive into RDSMySQL zero ETL integration with Redshift, explaining how it works and the advantages it brings to data engineers and developers.
The relationship between mysql user and database
Apr 08, 2025 pm 07:15 PM
In MySQL database, the relationship between the user and the database is defined by permissions and tables. The user has a username and password to access the database. Permissions are granted through the GRANT command, while the table is created by the CREATE TABLE command. To establish a relationship between a user and a database, you need to create a database, create a user, and then grant permissions.
MySQL: The Ease of Data Management for Beginners
Apr 09, 2025 am 12:07 AM
MySQL is suitable for beginners because it is simple to install, powerful and easy to manage data. 1. Simple installation and configuration, suitable for a variety of operating systems. 2. Support basic operations such as creating databases and tables, inserting, querying, updating and deleting data. 3. Provide advanced functions such as JOIN operations and subqueries. 4. Performance can be improved through indexing, query optimization and table partitioning. 5. Support backup, recovery and security measures to ensure data security and consistency.
