apache falcon documentation

12
Contents  Architecture Control flow Modes Of Deployment Entity Management actions Instance Management actions Retention Replication Cross entity validations Updating process and feed definition Handling late input data Idempotency Falcon EL Expressions Lineage Security Recipes Monitoring Backwards Compatibility Instructions  Ar c hi t ec tu r e Introduction Falcon is a feed and process management platform over hadoop. Falcon essentially transforms user's feed and process configurations into repeated actions through a standard workflow engine. Falcon by itself doesn't do any heavy lifting. All the functions and workflow state management requirements are delegated to the workflow scheduler. The only thing that Falcon maintains is the dependencies and r elationship between these entities. This is adequate to provide integrated and seamless experience to the developers using the falcon platform. Fa lcon Arch itectur e - Overview Scheduler Falcon system has picked Oozie as the default scheduler. However the system is open for integration with other schedulers. Lot of the data processing in hadoop requires scheduling to be based on both data availability as well as time. Oozie currently supports these capabilities off the shelf and hence the choice. Control flow Though the actual responsibility of the workflow is with the scheduler (Oozie), Falcon remains in the execution path, by subscribing to messages that each of the workflow may generate. When Falcon generates a workflow in Oozie, it does so, after instrumenting the workflow with additional steps which includes messaging via JMS. Falcon system itself subscribes to these control messages and can perform actions such as retries, handling late input arrival etc. Feed Schedule flow Falcon - Contents http://falcon.apache.org/0.6-incubating/FalconDocumentation.html 1 of 12 4/7/2015 3:40 AM

Upload: yuniantok7235

Post on 08-Jul-2018

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 1/12

Page 2: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 2/12

Page 3: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 3/12

Page 4: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 4/12

Page 5: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 5/12

Page 6: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 6/12

Page 7: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 7/12

Page 8: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 8/12

Page 9: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 9/12

Page 10: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 10/12

Page 11: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 11/12

Page 12: Apache Falcon Documentation

8/19/2019 Apache Falcon Documentation

http://slidepdf.com/reader/full/apache-falcon-documentation 12/12