[prev in list] [next in list] [prev in thread] [next in thread] 

List:       incubator-cvs
Subject:    =?utf-8?q?=5BIncubator_Wiki=5D_Update_of_=22CoralProposal=22_by_Byung-gon?= =?utf-8?q?Chun?=
From:       Apache Wiki <wikidiffs () apache ! org>
Date:       2018-01-31 8:59:01
Message-ID: 151738914177.17861.9815515707829594151 () moin-vm ! apache ! org
[Download RAW message or body]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change \
notification.

The "CoralProposal" page has been changed by Byung-gonChun:
https://wiki.apache.org/incubator/CoralProposal?action=diff&rev1=3&rev2=4

- ## page was renamed from OnyxProposal
+ ## page was renamed from CoralProposal
- = OnyxProposal =
+ = CoralProposal =
  
  == Abstract ==
- Onyx is a data processing system for flexible employment with different execution \
scenarios for various deployment characteristics on clusters. + Coral is a data \
processing system for flexible employment with different execution scenarios for \
various deployment characteristics on clusters.  
  == Proposal ==
  Today, there is a wide variety of data processing systems with different designs \
for better performance and datacenter efficiency. They include processing data on \
specific resource environments and running jobs with specific attributes. Although \
each system successfully solves the problems it targets, most systems are designed in \
the way that runtime behaviors are built tightly inside the system core to hide the \
complexity of distributed computing. This makes it hard for a single system to \
support different deployment characteristics with different runtime behaviors without \
substantial effort.  
- Onyx is a data processing system that aims to flexibly control the runtime \
behaviors of a job to adapt to varying deployment characteristics. Moreover, it \
provides a means of extending the system's capabilities and incorporating the \
extensions to the flexible job execution. + Coral is a data processing system that \
aims to flexibly control the runtime behaviors of a job to adapt to varying \
deployment characteristics. Moreover, it provides a means of extending the system's \
capabilities and incorporating the extensions to the flexible job execution.  
- In order to be able to easily modify runtime behaviors to adapt to varying \
deployment characteristics, Onyx exposes runtime behaviors to be flexibly configured \
and modified at both compile-time and runtime through a set of high-level graph pass \
interfaces.  + In order to be able to easily modify runtime behaviors to adapt to \
varying deployment characteristics, Coral exposes runtime behaviors to be flexibly \
configured and modified at both compile-time and runtime through a set of high-level \
graph pass interfaces.   
  We hope to contribute to the big data processing community by enabling more \
flexibility and extensibility in job executions. Furthermore, we can benefit more \
together as a community when we work together as a community to mature the system \
with more use cases and understanding of diverse deployment characteristics. The \
Apache Software Foundation is the perfect place to achieve these aspirations.  
@@ -24, +24 @@

  
  For an application writer to optimize an application to perform well on a certain \
system engraved with its underlying behaviors, it requires a deep understanding of \
the system itself, which is an overhead that often requires a lot of time and effort. \
Moreover, for a developer to modify such system behaviors, it requires modifications \
of the system core, which requires an even deeper understanding of the system itself. \
                
- With this background, Onyx is designed to represent all of its jobs as an \
Intermediate Representation (IR) DAG. In the Onyx compiler, user applications from \
various programming models (ex. Apache Beam) are submitted, transformed to an IR DAG, \
and optimized/customized for the deployment characteristics. In the IR DAG \
optimization phase, the DAG is modified through a series of compiler "passes" which \
reshape or annotate the DAG with an expression of the underlying runtime behaviors. \
The IR DAG is then submitted as an execution plan for the Onyx runtime. The runtime \
includes the unmodified parts of data processing in the backbone which is \
transparently integrated with configurable components exposed for further extension. \
+ With this background, Coral is designed to represent all of its jobs as an \
Intermediate Representation (IR) DAG. In the Coral compiler, user applications from \
various programming models (ex. Apache Beam) are submitted, transformed to an IR DAG, \
and optimized/customized for the deployment characteristics. In the IR DAG \
optimization phase, the DAG is modified through a series of compiler "passes" which \
reshape or annotate the DAG with an expression of the underlying runtime behaviors. \
The IR DAG is then submitted as an execution plan for the Coral runtime. The runtime \
includes the unmodified parts of data processing in the backbone which is \
transparently integrated with configurable components exposed for further extension.  \
  == Rationale ==
- Onyx's vision lies in providing means for flexibly supporting a wide variety of job \
execution scenarios for users while facilitating system developers to extend the \
execution framework with various functionalities at the same time. The capabilities \
of the system can be extended as it grows to meet a more variety of execution \
scenarios. We require inputs from users and developers from diverse domains in order \
to make it a more thriving and useful project. The Apache Software Foundation \
provides the best tools and community to support this vision. + Coral's vision lies \
in providing means for flexibly supporting a wide variety of job execution scenarios \
for users while facilitating system developers to extend the execution framework with \
various functionalities at the same time. The capabilities of the system can be \
extended as it grows to meet a more variety of execution scenarios. We require inputs \
from users and developers from diverse domains in order to make it a more thriving \
and useful project. The Apache Software Foundation provides the best tools and \
community to support this vision.  
  == Initial Goals ==
  Initial goals will be to move the existing codebase to Apache and integrate with \
the Apache development process. We further plan to develop our system to meet the \
needs for more execution scenarios for a more variety of deployment characteristics.  \
  == Current Status ==
- Onyx codebase is currently hosted in a repository at github.com. The current \
version has been developed by system developers at Seoul National University, Viva \
Republica, Samsung, and LG. + Coral codebase is currently hosted in a repository at \
github.com. The current version has been developed by system developers at Seoul \
National University, Viva Republica, Samsung, and LG.  
  == Meritocracy ==
- We plan to strongly support meritocracy. We will discuss the requirements in an \
open forum, and those that continuously contribute to Onyx with the passion to \
strengthen the system will be invited as committers. Contributors that enrich Onyx by \
providing various use cases, various implementations of the configurable components \
including ideas for optimization techniques will be especially welcome. Committers \
with a deep understanding of the system's technical aspects as a whole and its \
philosophy will definitely be voted as the PMC. We will monitor community \
participation so that privileges can be extended to those that contribute. + We plan \
to strongly support meritocracy. We will discuss the requirements in an open forum, \
and those that continuously contribute to Coral with the passion to strengthen the \
system will be invited as committers. Contributors that enrich Coral by providing \
various use cases, various implementations of the configurable components including \
ideas for optimization techniques will be especially welcome. Committers with a deep \
understanding of the system's technical aspects as a whole and its philosophy will \
definitely be voted as the PMC. We will monitor community participation so that \
privileges can be extended to those that contribute.  
  == Community ==
- We hope to expand our contribution community by becoming an Apache incubator \
project. The contributions will come from both users and system developers interested \
in flexibility and extensibility of job executions that Onyx can support. We expect \
users to mainly contribute to diversify the use cases and deployment characteristics, \
and developers to  contribute to implement them. + We hope to expand our contribution \
community by becoming an Apache incubator project. The contributions will come from \
both users and system developers interested in flexibility and extensibility of job \
executions that Coral can support. We expect users to mainly contribute to diversify \
the use cases and deployment characteristics, and developers to  contribute to \
implement them.  
  == Alignment ==
- Apache Spark is one of many popular data processing frameworks. The system is \
designed towards optimizing jobs using RDDs in memory and many other optimizations \
built tightly within the framework. In contrast to Spark, Onyx aims to provide more \
flexibility for job execution in an easy manner. + Apache Spark is one of many \
popular data processing frameworks. The system is designed towards optimizing jobs \
using RDDs in memory and many other optimizations built tightly within the framework. \
In contrast to Spark, Coral aims to provide more flexibility for job execution in an \
easy manner.  
- Apache Tez enables developers to build complex task DAGs with control over the \
control plane of job execution. In Onyx, a high-level programming layer (ex. Apache \
Beam) is automatically converted to a basic IR DAG and can be converted to any IR DAG \
through a series of easy user writable passes, that can both reshape and modify the \
annotation (of execution properties) of the DAG. Moreover, Onyx leaves more parts of \
the job execution configurable, such as the scheduler and the data plane. As opposed \
to providing a set of properties for solid optimization, Onyx's configurable parts \
can be easily extended and explored by implementing the pre-defined interfaces. For \
example, an arbitrary intermediate data store can be added.   + Apache Tez enables \
developers to build complex task DAGs with control over the control plane of job \
execution. In Coral, a high-level programming layer (ex. Apache Beam) is \
automatically converted to a basic IR DAG and can be converted to any IR DAG through \
a series of easy user writable passes, that can both reshape and modify the \
annotation (of execution properties) of the DAG. Moreover, Coral leaves more parts of \
the job execution configurable, such as the scheduler and the data plane. As opposed \
to providing a set of properties for solid optimization, Coral's configurable parts \
can be easily extended and explored by implementing the pre-defined interfaces. For \
example, an arbitrary intermediate data store can be added.    
- Onyx currently supports Apache Beam programs and we are working on supporting \
Apache Spark programs as well. Onyx also utilizes Apache REEF for container \
management, which allows Onyx to run in Apache YARN and Apache Mesos clusters. If \
necessary, we plan to contribute to and collaborate with these other Apache projects \
for the benefit of all. We plan to extend such integrations with more Apache \
softwares. Apache software foundation already hosts many major big-data systems, and \
we expect to help further growth of the big-data community by having Onyx within the \
Apache foundation.  + Coral currently supports Apache Beam programs and we are \
working on supporting Apache Spark programs as well. Coral also utilizes Apache REEF \
for container management, which allows Coral to run in Apache YARN and Apache Mesos \
clusters. If necessary, we plan to contribute to and collaborate with these other \
Apache projects for the benefit of all. We plan to extend such integrations with more \
Apache softwares. Apache software foundation already hosts many major big-data \
systems, and we expect to help further growth of the big-data community by having \
Coral within the Apache foundation.   
  == Known Risks ==
  === Orphaned Products ===
- The risk of the Onyx project being orphaned is minimal. There is already plenty of \
work that arduously support different deployment characteristics, and we propose a \
general way to implement them with flexible and extensible configuration knobs. The \
domain of data processing is already of high interest, and this domain is expected to \
evolve continuously with various other purposes, such as resource disaggregation and \
using transient resources for better datacenter resource utilization. + The risk of \
the Coral project being orphaned is minimal. There is already plenty of work that \
arduously support different deployment characteristics, and we propose a general way \
to implement them with flexible and extensible configuration knobs. The domain of \
data processing is already of high interest, and this domain is expected to evolve \
continuously with various other purposes, such as resource disaggregation and using \
transient resources for better datacenter resource utilization.  
  === Inexperience with Open Source ===
  The initial committers include PMC members and committers of other Apache projects. \
They have experience with open source projects, starting from their incubation to the \
top-level. They have been involved in the open source development process, and are \
familiar with releasing code under an open source license. @@ -62, +62 @@

  Many developers are from Seoul National University. This is not applicable.
  
  === Relationships with Other Apache Products ===
- Onyx positions itself among multiple Apache products. It runs on Apache REEF for \
container management. It also utilizes many useful development tools including Apache \
Maven, Apache Log4J, and multiple Apache Commons components. Onyx supports the Apache \
Beam programming model for user applications. We are currently working on supporting \
the Apache Spark programming APIs as well. + Coral positions itself among multiple \
Apache products. It runs on Apache REEF for container management. It also utilizes \
many useful development tools including Apache Maven, Apache Log4J, and multiple \
Apache Commons components. Coral supports the Apache Beam programming model for user \
applications. We are currently working on supporting the Apache Spark programming \
APIs as well.  
  === An Excessive Fascination with the Apache Brand ===
- We hope to make Onyx a powerful system for data processing, meeting various needs \
for different deployment characteristics, under a more variety of environments. We \
see the limitations of simply putting code on GitHub, and we believe the Apache \
community will help the growth of Onyx for the project to become a positively \
impactful and innovative open source software. We believe Onyx is a great fit for the \
Apache Software Foundation due to the collaboration it aims to achieve from the big \
data processing community. + We hope to make Coral a powerful system for data \
processing, meeting various needs for different deployment characteristics, under a \
more variety of environments. We see the limitations of simply putting code on \
GitHub, and we believe the Apache community will help the growth of Coral for the \
project to become a positively impactful and innovative open source software. We \
believe Coral is a great fit for the Apache Software Foundation due to the \
collaboration it aims to achieve from the big data processing community.  
  == Documentation ==
- The current documentation for Onyx is at https://snuspl.github.io/onyx/.
+ The current documentation for Coral is at https://snuspl.github.io/coral/.
  
  == Initial Source ==
- The Onyx codebase is currently hosted at https://github.com/snuspl/onyx.
+ The Coral codebase is currently hosted at https://github.com/snuspl/coral.
  
  == External Dependencies ==
- To the best of our knowledge, all Onyx dependencies are distributed under Apache \
compatible licenses. Upon acceptance to the incubator, we would begin a thorough \
analysis of all transitive dependencies to verify this fact and further introduce \
license checking into the build and release process. + To the best of our knowledge, \
all Coral dependencies are distributed under Apache compatible licenses. Upon \
acceptance to the incubator, we would begin a thorough analysis of all transitive \
dependencies to verify this fact and further introduce license checking into the \
build and release process.  
  == Cryptography ==
  Not applicable.
@@ -82, +82 @@

  == Required Resources ==
  === Mailing Lists ===
  We will operate two mailing lists as follows: 
-    * Onyx PMC discussions: private@onyx.incubator.apache.org
+    * Coral PMC discussions: private@coral.incubator.apache.org
-    * Onyx developers: dev@onyx.incubator.apache.org
+    * Coral developers: dev@coral.incubator.apache.org
  
  === Git Repositories ===
- Upon incubation: https://github.com/apache/incubator-onyx.  
+ Upon incubation: https://github.com/apache/incubator-coral.  
- After the incubation, we would like to move the existing repo \
https://github.com/snuspl/onyx to the Apache infrastructure + After the incubation, \
we would like to move the existing repo https://github.com/snuspl/coral to the Apache \
infrastructure  
  === Issue Tracking ===
- Onyx currently tracks its issues using the Github issue tracker: \
https://github.com/snuspl/onyx/issues. We plan to migrate to Apache JIRA. + Coral \
currently tracks its issues using the Github issue tracker: \
https://github.com/snuspl/coral/issues. We plan to migrate to Apache JIRA.  
  == Initial Committers ==
    * Byung-Gon Chun 
@@ -137, +137 @@

  === Mentors ===
    * Hyunsik Choi
    * Byung-Gon Chun
+   * Jean-Baptiste Onofré
    * Markus Weimer 
    * Reynold Xin
-   * Jean-Baptiste Onofré
  
  === Sponsoring Entity ===
  The Apache Incubator  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]