Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce. In the short term the major focus is to support Hive and Pig, specifically to enable performance improvements to batch and ad-hoc interactive queries.
Tez provides runtime components:
An execution environment that can handle traditional map-reduce jobsAn execution environment that handles DAG-based jobs comprising various built-in and extendable primitivesCluster-side determination of input piecesRuntime planning such as task cardinality determination and dynamic modification to the DAG structureTez provides APIs to access these services:
Traditional map-reduce functionality is accessed via java classes written to the Job interface: org.apache.hadoop.mapred.Job and/or org.apache.hadoop.mapreduce.v2.app.job.Job; and by specifying in yarn-site that the map-reduce framework should be Tez.DAG-based execution is accessed via the new Tez DAG API: org.apache.tez.dag.api.*, org.apache.tez.engine.api.*.Tez provides pre-made primitives for use with the DAG API (org.apache.tez.engine.common.*)
Vertex InputVertex OutputSortingShufflingMergingData transfer
In the above figure Tez is represented by the red components: client-side API, an AppMaster, and multiple containers that execute child processes under the control of the AppMaster.
Three separate software stacks are involved in the execution of a Tez job, each using components from the clientapplication, Tez, and YARN:
The following terminology is used:
Job Vertex: A “stage” in the job plan. 逻辑顶点, 可以理解成stage Job Edge: The logical connections between Job Vertices. 逻辑边, 关联 Vertex: A materialized stage at runtime comprising a certain number of materialized tasks. 物理顶点, 由并行的tasks节点组成 Edge: Represents actual data movement between tasks. 物理边, 代表实际数据流向 Task: A process performing computation within a YARN container. Task, 一个执行节点 Task cardinality: The number of materialized tasks in a Vertex. Task基数, Vertex的并发度 Static plan: Planning decisions fixed before job submission. Dynamic plan: Planning decisions made at runtime in the AppMaster process.
The Tez API comprises many services that support applications to run DAG-style jobs. An application that makes use of Tez will need to: 1. Create a job plan (the DAG) comprising vertices, edges, and data source references 2. Create task implementations that perform computations and interact with the DAG AppMaster 3. Configure Yarn and Tez appropriately
抽象DAG的定义接口
public class DAG{ DAG(); void addVertex(Vertex); void addEdge(Edge); void addConfiguration(String, String); void setName(String); void verify(); DAGPlan createDaG(); } public class Vertex { Vertex(String vertexName, String processorName, int parallelism); void setTaskResource(); void setTaskLocationsHint(TaskLocationHint[]); void setJavaOpts(String); String getVertexName(); String getProcessorName(); int getParallelism(); Resource getTaskResource(); TaskLocationHint[] getTaskLocationsHint(); String getJavaOpts(); } public class Edge { Edge(Vertex inputVertex, Vertex outputVertex, EdgeProperty edgeProperty); String getInputVertex(); String getOutputVertex(); EdgeProperty getEdgeProperty(); String getId(); }Task作为Tez的执行者, 遵循input, output, processor的模式
public interface Master //a context object for task execution. currently only stub public interface Input{ void initialize(Configuration conf, Master master) boolean hasNext() Object getNextKey() Iterable<Object> getNextValues() float getProgress() void close() } public interface Output{ void initialize(Configuration conf, Master master); void write(Object key, Object value); OutputContext getOutputContext(); void close(); } public interface Partitioner { int getPartition(Object key, Object value, int numPartitions); } public interface Processor { void initialize(Configuration conf, Master master) void process(Input[] in, Output[] out) void close() } public interface Task{ void initialize(Configuration conf, Master master) Input[] getInputs(); Processor getProcessor(); Output[] getOutputs(); void run() void close() } 本文章摘自博客园，原文发布日期： 2013-10-19 相关资源：敏捷开发V1.0.pptx