faz documentation - media.readthedocs.org · faz documentation, release 0.1.7 faz without...
TRANSCRIPT
![Page 1: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/1.jpg)
faz DocumentationRelease 0.1.7
Hugo Martiniano
July 21, 2016
![Page 2: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/2.jpg)
![Page 3: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/3.jpg)
Contents
1 faz 31.1 Why? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Task file basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Installation 5
3 Usage 7
4 Contributing 94.1 Types of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 Get Started! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3 Pull Request Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.4 Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5 Credits 135.1 Development Lead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.2 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6 History 15
7 0.1.0 (2014-01-11) 17
8 0.1.1 (2015-03-20) 19
9 0.1.2 (2015-10-17) 21
10 0.1.3 (2016-03-26) 23
11 0.1.4 (2016-05-19) 25
12 0.1.5 (2016-05-19) 27
13 0.1.6 (2016-07-18) 29
14 0.1.7 (2016-07-20) 31
15 Indices and tables 33
i
![Page 4: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/4.jpg)
ii
![Page 5: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/5.jpg)
faz Documentation, Release 0.1.7
Contents:
Contents 1
![Page 6: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/6.jpg)
faz Documentation, Release 0.1.7
2 Contents
![Page 7: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/7.jpg)
CHAPTER 1
faz
Faz is a data workflow tool heavily inspired in .. _Drake: https://github.com/Factual/drake
The intended use is combining data treatment scripts in bash, python, ruby (or anything else, with a little coding) intoa single text file.
The name “faz” is portuguese for “do” or “make”.
The various steps can be separated into tasks, with defined inputs and outputs. Dependencies between the tasks aredetermined from inputs and outputs of every task. The program executes all tasks in the appropriate order, checkingfor the existence of output and input files.
1.1 Why?
• Because I like Drake but can’t stand the startup time of java.
• Because I can (actually to see if I can, but it turns out I can).
1.2 Features
• simple but robust functionality
• easy to use and extend (the code, minus the tests, is around 300 lines of python)
• fast startup time (compared to Drake)
• Documentation: https://faz.readthedocs.org.
1.3 Installation
Using pypi
pip install faz
1.4 Usage
From the command line, just type
3
![Page 8: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/8.jpg)
faz Documentation, Release 0.1.7
faz
without arguments, the program will read the tasks from a file called “fazfile”. If you want to use another filename,just give that as an argument to the program
faz <filename>
to get a list of command line arguments type
faz -h
1.5 Task file basics
The task file is a plain text file, with a syntax similar to Drake input files. The following is an example with two tasks
# file1 <-touch file1
# file2 <- file1cat file1 > file2
Lines starting with “#” and having the symbols “<-” signal a task. On the left of the “<-” is a (comma separated)list of the files produced by the task. On the right are the task dependencies, the files needed to run that task. In theabove example the first task has no dependencies, and produces a file called “file1”. The second task has “file1” as adependency, and has as output a file called “file2”.
The outputs and inputs and inputs of each task are used by the program to estabilish the order by which the tasks haveto be run, and if they need to be run. In the example above, if a file called “file1” was already present in the directorythe program was run, the first task would not be executed.
The code sections are all the lines in betweeen the two task lines. In these two tasks, they are just are just plain bashcommands but could be, for example, python code
# file1 <-touch file1
# file2 <- file1 :pythonf1 = open("file1")text = file1.read()f2 = open("file2", "w")f2.write(text)
note that, in the second task, there’s an extra option ”:python”, wich indicates to the program that the code from thistask is python code. Options are a list of (comma separated) keywords following the ”:”, and must be placed after theinputs.
4 Chapter 1. faz
![Page 9: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/9.jpg)
CHAPTER 2
Installation
From pipy:
$ pip install faz
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv faz$ pip install faz
5
![Page 10: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/10.jpg)
faz Documentation, Release 0.1.7
6 Chapter 2. Installation
![Page 11: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/11.jpg)
CHAPTER 3
Usage
From the command line, just type
faz
without arguments, the program will read the tasks from a file called “fazfile”. If you want to use another filename,just give that as an argument to the program
faz <filename>
to get a list of command line arguments type
faz -h
7
![Page 12: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/12.jpg)
faz Documentation, Release 0.1.7
8 Chapter 3. Usage
![Page 13: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/13.jpg)
CHAPTER 4
Contributing
Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
4.1 Types of Contributions
4.1.1 Report Bugs
Report bugs at https://github.com/hmartiniano/faz/issues.
If you are reporting a bug, please include:
• Your operating system name and version.
• Any details about your local setup that might be helpful in troubleshooting.
• Detailed steps to reproduce the bug.
4.1.2 Fix Bugs
Look through the GitHub issues for bugs. Anything tagged with “bug” is open to whoever wants to implement it.
4.1.3 Implement Features
Look through the GitHub issues for features. Anything tagged with “feature” is open to whoever wants to implementit.
4.1.4 Write Documentation
faz could always use more documentation, whether as part of the official faz docs, in docstrings, or even on the web inblog posts, articles, and such.
4.1.5 Submit Feedback
The best way to send feedback is to file an issue at https://github.com/hmartiniano/faz/issues.
If you are proposing a feature:
9
![Page 14: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/14.jpg)
faz Documentation, Release 0.1.7
• Explain in detail how it would work.
• Keep the scope as narrow as possible, to make it easier to implement.
• Remember that this is a volunteer-driven project, and that contributions are welcome :)
4.2 Get Started!
Ready to contribute? Here’s how to set up faz for local development.
1. Fork the faz repo on GitHub.
2. Clone your fork locally:
$ git clone [email protected]:your_name_here/faz.git
3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set upyour fork for local development:
$ mkvirtualenv faz$ cd faz/$ python setup.py develop
4. Create a branch for local development:
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
5. When you’re done making changes, check that your changes pass flake8 and the tests, including testing otherPython versions with tox:
$ flake8 faz tests$ python setup.py test$ tox
To get flake8 and tox, just pip install them into your virtualenv.
6. Commit your changes and push your branch to GitHub:
$ git add .$ git commit -m "Your detailed description of your changes."$ git push origin name-of-your-bugfix-or-feature
7. Submit a pull request through the GitHub website.
4.3 Pull Request Guidelines
Before you submit a pull request, check that it meets these guidelines:
1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a functionwith a docstring, and add the feature to the list in README.rst.
3. The pull request should work for Python 2.6, 2.7, 3.3, and 3.4, and for PyPy. Check https://travis-ci.org/hmartiniano/faz/pull_requests and make sure that the tests pass for all supported Python versions.
10 Chapter 4. Contributing
![Page 15: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/15.jpg)
faz Documentation, Release 0.1.7
4.4 Tips
To run a subset of tests:
$ python -m unittest tests.test_faz
4.4. Tips 11
![Page 16: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/16.jpg)
faz Documentation, Release 0.1.7
12 Chapter 4. Contributing
![Page 17: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/17.jpg)
CHAPTER 5
Credits
5.1 Development Lead
• Hugo Martiniano <[email protected]>
5.2 Contributors
None yet. Why not be the first?
13
![Page 18: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/18.jpg)
faz Documentation, Release 0.1.7
14 Chapter 5. Credits
![Page 19: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/19.jpg)
CHAPTER 6
History
15
![Page 20: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/20.jpg)
faz Documentation, Release 0.1.7
16 Chapter 6. History
![Page 21: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/21.jpg)
CHAPTER 7
0.1.0 (2014-01-11)
• First release.
17
![Page 22: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/22.jpg)
faz Documentation, Release 0.1.7
18 Chapter 7. 0.1.0 (2014-01-11)
![Page 23: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/23.jpg)
CHAPTER 8
0.1.1 (2015-03-20)
• Bug fixes.
19
![Page 24: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/24.jpg)
faz Documentation, Release 0.1.7
20 Chapter 8. 0.1.1 (2015-03-20)
![Page 25: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/25.jpg)
CHAPTER 9
0.1.2 (2015-10-17)
• Project name change.
21
![Page 26: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/26.jpg)
faz Documentation, Release 0.1.7
22 Chapter 9. 0.1.2 (2015-10-17)
![Page 27: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/27.jpg)
CHAPTER 10
0.1.3 (2016-03-26)
• NetworkX dependency removed.
23
![Page 28: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/28.jpg)
faz Documentation, Release 0.1.7
24 Chapter 10. 0.1.3 (2016-03-26)
![Page 29: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/29.jpg)
CHAPTER 11
0.1.4 (2016-05-19)
• Input and output names added to task environment.
25
![Page 30: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/30.jpg)
faz Documentation, Release 0.1.7
26 Chapter 11. 0.1.4 (2016-05-19)
![Page 31: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/31.jpg)
CHAPTER 12
0.1.5 (2016-05-19)
• Bug Fixes in variable expansion code.
27
![Page 32: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/32.jpg)
faz Documentation, Release 0.1.7
28 Chapter 12. 0.1.5 (2016-05-19)
![Page 33: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/33.jpg)
CHAPTER 13
0.1.6 (2016-07-18)
• Added a mechanism to include other task files.
29
![Page 34: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/34.jpg)
faz Documentation, Release 0.1.7
30 Chapter 13. 0.1.6 (2016-07-18)
![Page 35: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/35.jpg)
CHAPTER 14
0.1.7 (2016-07-20)
• dependencies and outputs can now be on different directories.
31
![Page 36: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/36.jpg)
faz Documentation, Release 0.1.7
32 Chapter 14. 0.1.7 (2016-07-20)
![Page 37: faz Documentation - media.readthedocs.org · faz Documentation, Release 0.1.7 faz without arguments, the program will read the tasks from a file called “fazfile”. If you want](https://reader034.vdocument.in/reader034/viewer/2022050500/5f93592ee8298973726fa242/html5/thumbnails/37.jpg)
CHAPTER 15
Indices and tables
• genindex
• modindex
• search
33