open science workshop - handout · open science workshop - handout december 4, 2018 1 so, you want...

Open Science Workshop - Handout

December 4, 2018

1 So, you want to go open Science ?

1.1 Version, colaborate, share, test, document !

We will manly show you how to do things, and give you the full code of what we did

• https://c4science.ch/source/so-workshop/• https://c4science.ch/source/so-workshop-test/• https://c4science.ch/source/so-slides/

SCITAS training with expertise from the EPFL Library Nicolas Richart and Jean-BaptisteAubort - SCITAS

2 First thing first : Version your stuffs

3 Create a repository

c4s repo

3.1 Some basics on git

• Let this represent a repository

• Clone an existing repository

1

https://c4science.ch/source/so-workshop/

https://c4science.ch/source/so-workshop-test/

https://c4science.ch/source/so-slides/

In [1]: cd ..rm -rf so-workshop

In [2]: git clone https://c4science.ch/source/so-workshop.gitcd so-workshop

Cloning into 'so-workshop'...remote: Counting objects: 191, done.remote: Compressing objects: 100% (181/181), done.remote: Total 191 (delta 87), reused 0 (delta 0)Receiving objects: 100% (191/191), 17.90 KiB | 8.95 MiB/s, done.Resolving deltas: 100% (87/87), done.

In [3]: echo "Add a modification to the readme" >> README

In [4]: # edit READMEgit status

On branch masterYour branch is up to date with 'origin/master'.

Changes not staged for commit:(use "git add <file>..." to update what will be committed)(use "git checkout -- <file>..." to discard changes in working directory)

modified: README

no changes added to commit (use "git add" and/or "git commit -a")

In [5]: git add README

In [6]: git status

On branch masterYour branch is up to date with 'origin/master'.

Changes to be committed:(use "git reset HEAD <file>..." to unstage)

modified: README

In [7]: git commit -m "Added a change to README"

[master 9130a85] Added a change to README1 file changed, 1 insertion(+)

2

In [8]: git log --graph -n3

* commit 9130a85bff15c2b8800fd6cf6c5e4723ce2cc09a (HEAD -> master)| Author: Nicolas Richart <[email protected]>| Date: Mon Dec 3 15:19:30 2018 +0100|| Added a change to README|

* commit 59738410b9ec50d0a20cbdc5639e045084bbc5a1 (origin/master, origin/HEAD)| Author: Jean-Baptiste Aubort <[email protected]>| Date: Fri Nov 30 16:52:44 2018 +0100|| Fix branch name|

* commit b896907f7b01e9490c47f85f57783cb36040c39d| Author: Jean-Baptiste Aubort <[email protected]>| Date: Fri Nov 30 16:51:12 2018 +0100|| Update branch names

In [9]: # In order to push without changing the repository origingit remote set-url origin ssh://[email protected]/source/so-workshop-test.gitgit push --force -u origin master

Enumerating objects: 36, done.Counting objects: 100% (36/36), done.Delta compression using up to 4 threadsCompressing objects: 100% (23/23), done.Writing objects: 100% (36/36), 3.68 KiB | 3.68 MiB/s, done.Total 36 (delta 6), reused 28 (delta 5)To ssh://c4science.ch/source/so-workshop-test.git+ bdb19ee...9130a85 master -> master (forced update)Branch 'master' set up to track remote branch 'master' from 'origin'.

• git commit

• often someone else already commited

3

• git pull

• git push

4 Collaborate through projects

4.1 Why projects

• To tidy your repositories• To handle permissions• To define tasks• To attach information

c4s projects

5 Now that your are all setup : add a workflow

5.1 Why a workflow

• Helps to separate development• Helps to release a code• Helps to review to code• Helps to maintain different versions

5.2 Minimalist workflow

• Feature branches

In [10]: git checkout -b feature/workflow

4

c4s project view

c4s repository tags

c4s project tasks

5

Switched to a new branch 'feature/workflow'

In [11]: echo "Adding my feature" >> new_featuregit add new_featuregit commit -m "Added a feature to demonstrate a simple workflow"

[feature/workflow 7ab596d] Added a feature to demonstrate a simple workflow1 file changed, 1 insertion(+)create mode 100644 new_feature

• Merge your changes back in master

In [12]: git checkout mastergit merge --no-ff --no-edit feature/workflow

Switched to branch 'master'Your branch is up to date with 'origin/master'.Merge made by the 'recursive' strategy.new_feature | 1 +1 file changed, 1 insertion(+)create mode 100644 new_feature


* commit 503033e6a6cfab06ddb4873b975e62b23d1a9bc8 (HEAD -> master)|\ Merge: 9130a85 7ab596d| | Author: Nicolas Richart <[email protected]>| | Date: Mon Dec 3 15:19:34 2018 +0100| || | Merge branch 'feature/workflow'| || * commit 7ab596dd68f757a29260e962721ee14e541c0216 (feature/workflow)|/ Author: Nicolas Richart <[email protected]>| Date: Mon Dec 3 15:19:33 2018 +0100|| Added a new feature to demonstrate a simple workflow|

* commit 9130a85bff15c2b8800fd6cf6c5e4723ce2cc09a (origin/master, origin/HEAD)| Author: Nicolas Richart <[email protected]>| Date: Mon Dec 3 15:19:30 2018 +0100|| Added a change to README

In [14]: git checkout mastergit push

6

Already on 'master'Your branch is ahead of 'origin/master' by 2 commits.

(use "git push" to publish your local commits)Enumerating objects: 5, done.Counting objects: 100% (5/5), done.Delta compression using up to 4 threadsCompressing objects: 100% (3/3), done.Writing objects: 100% (4/4), 425 bytes | 425.00 KiB/s, done.Total 4 (delta 2), reused 0 (delta 0)To ssh://c4science.ch/source/so-workshop-test.git

9130a85..503033e master -> master

• Review the code using arcanist

In [15]: git checkout -b feature/reviewecho "Adding my feature that needs review" >> new_featuregit add new_featuregit commit -m "Added a new feature to demonstrate a review"touch new_filegit add new_filegit commit -m "Adding a file"

Switched to a new branch 'feature/review'[feature/review f192911] Added a new feature to demonstrate a review1 file changed, 1 insertion(+)[feature/review e06ca93] Adding a file1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 new_file

In [16]: cat > ../.arcmessage << EOFAdding a reviewed featureTest Plan: none yetReviewers: #SoOpenEOF

In [18]: arc diff master -F ../.arcmessage --no-amend

Linting...No lint engine configured for this project.Running unit tests...No unit test engine is configured for this project.SKIP STAGING No staging area is configured for this repository.Created a new Differential revision:

Revision URI: https://c4science.ch/D222

Included changes:M new_feature

7

A new_file

c4s differential

• Once the review is accepted it can be landed

In [21]: arc land --merge --revision D222

Landing current branch 'feature/review'.TARGET Landing onto "master", the default target under git.REMOTE Using remote "origin", the default remote under git.FETCH Fetching origin/master...These commits will be landed:

- e06ca93 Adding a file- f192911 Added a new feature to demonstrate a review

Landing revision 'D222: Adding a reviewed feature'...BUILDS PASSED Harbormaster builds for the active diff completed successfully.PUSHING Pushing changes to "origin/master".Enumerating objects: 9, done.Counting objects: 100% (9/9), done.Delta compression using up to 4 threadsCompressing objects: 100% (6/6), done.Writing objects: 100% (7/7), 760 bytes | 760.00 KiB/s, done.Total 7 (delta 3), reused 0 (delta 0)To ssh://c4science.ch/source/so-workshop-test.git

503033e..7a166f2 7a166f2b01bf778f941a1a0722c23b887da77e04 -> masterUPDATE Local "master" tracks target remote "origin/master", checking out and pulling changes.PULL Checking out and pulling "master".Cleaning up branch "feature/review"...(Use `git checkout -b feature/review e06ca93df6ab4b3fa3b231a7e5f3f0e2e38d6018` if you want it back.)

8

DONE Landed changes.


* commit 52ec4a6c006cd9b0b95957dc810d98bc2cc84139|\ Merge: 19f5c5c 71efe12| | Author: Nicolas Richart <[email protected]>| | Date: Mon Dec 3 04:32:52 2018 +0100| || | Adding a reviewed feature| || | Test Plan: none yet| || | Reviewers: nrichart| || | Reviewed By: nrichart| || | Differential Revision: https://c4science.ch/D220| || * commit 71efe12719648965e96ad146412d0c3726dc79a0| | Author: Nicolas Richart <[email protected]>| | Date: Mon Dec 3 04:32:52 2018 +0100| || | Adding a file| || * commit a00f49cdd2ff81a83d40398ccb29082ad14ec80e|/ Author: Nicolas Richart <[email protected]>| Date: Mon Dec 3 04:32:52 2018 +0100|| Added a new feature to demonstrate a review

5.3 Adding a public version of the code

In [25]: git remote add public [email protected]:epfl-scitas/so-workshop.gitgit fetch public

From github.com:epfl-scitas/so-workshop

* [new branch] master -> public/master

In [26]: git checkout -b public public/master

Branch 'public' set up to track remote branch 'master' from 'public'.Switched to a new branch 'public'

In [27]: git merge --squash --allow-unrelated-histories master

9

Squash commit -- not updating HEADAutomatic merge went well; stopped before committing as requested

In [28]: git commit -m "First public revision"git push -u public public:master

[public 46ccdaa] First public revision5 files changed, 32 insertions(+)create mode 100644 .gitignorecreate mode 100644 READMEcreate mode 100644 code/mylib.pycreate mode 100644 new_featurecreate mode 100644 new_fileEnumerating objects: 9, done.Counting objects: 100% (9/9), done.Delta compression using up to 4 threadsCompressing objects: 100% (5/5), done.Writing objects: 100% (8/8), 885 bytes | 885.00 KiB/s, done.Total 8 (delta 0), reused 2 (delta 0)To github.com:epfl-scitas/so-workshop.git

eaf9eaf..46ccdaa public -> masterBranch 'public' set up to track remote branch 'master' from 'public'.


* commit 46ccdaa253a2960c2c8cb77dea0885b232d0044a (HEAD -> public, public/master)| Author: Nicolas Richart <[email protected]>| Date: Mon Dec 3 15:38:14 2018 +0100|| First public revision|

* commit eaf9eaf6260bfda406becd0adb39cc90474b3691Author: Nicolas Richart <[email protected]>Date: Mon Dec 3 15:37:16 2018 +0100

Create README.md

6 Continuous Integration (CI)

6.1 What is CI ?

• Automation of build and tests• Run tests at each commit• You need to write tests !

10

6.2 Why use CI ?

• Detect bugs when they are introduced• Detect when fixed bugs appear again (regression)• Published code is tested and should be working• Maintain code quality• Linters• Test Coverage

7 Jenkins on c4science

7.1 https://c4science.ch/jobs

c4s jenkinsjob

7.2 https://jenkins.c4science.ch/

c4s jenkins

11

https://c4science.ch/jobs

https://jenkins.c4science.ch/

8 Example

8.1 Jenkins base

In [36]: cd ../so-workshop

Branch 'feature/jenkins-base' set up to track remote branch 'feature/jenkins-base' from 'origin'.Switched to a new branch 'feature/jenkins-base'.codeaa mylib.pyaa test.pyJenkinsfileREADME

1 directory, 4 files

In [4]: git checkout feature/jenkins-basetree

Switched to branch 'feature/jenkins-base'Your branch is up to date with 'origin/feature/jenkins-base'..codeaa mylib.pyaa test.pyJenkinsfileREADME


In [37]: cat code/mylib.py

#!/usr/bin/env python3

class MyLib():

def __init__(self):pass

def MyFunc(self):return 'test'

if __name__ == '__main__':pass

In [38]: cat code/test.py

12

#!/usr/bin/env python3import unittest, pytestimport mylib

class Test(unittest.TestCase):

def test_return(self):m = mylib.MyLib()self.assertTrue(m.MyFunc() == 'test')

if __name__ == '__main__':pytest.main(['./test.py'])

In [39]: cat Jenkinsfile

pipeline {

agent {docker {

image 'python:3.7'}

}

environment {HOME = "$WORKSPACE"

}

stages {stage('install dependencies') {

steps {sh 'pip3 install --user pytest'

}}stage('run tests') {

steps {sh '$HOME/.local/bin/pytest code/test.py'

}}

}

}

13

jenkins base

9 Example

9.1 Jenkins dockerfile

In [40]: git checkout feature/jenkins-dockerfiletree

Switched to branch 'feature/jenkins-dockerfile'Your branch is up to date with 'origin/feature/jenkins-dockerfile'..codeaa mylib.pyaa test.pyDockerfileJenkinsfileREADME


In [41]: cat Dockerfile

FROM python:3.7RUN pip3 install --no-cache-dir pytest


pipeline {

agent {dockerfile true

}

14

stages {stage('run tests') {

steps {sh 'pytest code/test.py'

}}

}

}

9.2 Example

9.2.1 Jenkins junit

In [43]: git checkout feature/jenkins-junittree

Switched to branch 'feature/jenkins-junit'Your branch is up to date with 'origin/feature/jenkins-junit'..codeaa mylib.pyaa test.pyDockerfileJenkinsfileREADME



pipeline {


}


steps {sh 'pytest --junitxml results.xml code/test.py'

}}

}

post {

15

always {junit 'results.xml'

}}

}

jenkins junit

jenkins junit 2

10 C4science

10.1 Arcanist

Command line tool to interact with c4science - Code review

16

arc diffarc listarc coverarc patcharc land

• Code quality

arc lintarc unit

• Other tools

arc call-conduitarc uploadarc downloadarc pastearc anoid

In [1]: arc list

Needs Review D73: Add Wikimedia Sprint extensionNeeds Review D86: Add mysql logs to rsyslogNeeds Review D130: Enable automation on all repositoriesNeeds Review D151: Separate name and hashtag for Project allowing to have multiple projects with the same nameNeeds Review D200: Every command run using ExecFuture use a custom cgroup 'future'Needs Review D209: Migrate from GlusterFS to native Phabricator clustered repo

In [7]: echo hello > /tmp/helloarc upload --temporary /tmp/helloarc download --as /tmp/hello-new F10024647cat /tmp/hello | arc paste

F10024657 hello: https://c4science.ch/F10024657

Done.Getting file information...Downloading file 'hello' (6 bytes)...Saved file as '/tmp/hello-new'.P6: https://c4science.ch/P6

In [2]: arc tasks

T1411 Huge repositories and push Unbreak Now! OpenT2377 c4science Winter Update Unbreak Now! OpenT2143 C4science: One of your repository is too big Unbreak Now! Open

17

T2027 Cannot download binary file from web browser interface Unbreak Now! OpenT1477 Couldn't set refs/heads/master - failed to write Unbreak Now! OpenT1882 Externals can't view Conpherence room High OpenT1607 Download raw files High OpenT1531 SSH access for Administrator fatal High OpenT2320 fatal: premature end of pack file, XX bytes missing High OpenT2386 Test OOMd from Facebook instea of stock OOMKiller High OpenT2002 Pygmentize stuck at CPU 100% for >60h High OpenT1485 Repository don't appear in the new Home High OpenT1079 Realtime replication for Gluster and MySQL High OpenT816 Leaving university keeps the user in Creators group High OpenT666 MySQL database table schema need update High OpenT2317 Problem with fs load leads to filesystem deconnection High OpenT1790 Remove users when Switch AAI account is not valid anymore Normal OpenT1699 Add image in README.md on c4science Normal OpenT1696 Automatic badges Normal OpenT1597 Upload file deletes typed issue description Normal OpenT1525 Better project page Normal OpenT1519 Rename All Users (Creators) to Swiss Universities Normal OpenT1476 warning: remote HEAD refers to nonexistent ref, unable to checkout. Normal OpenT1310 Escape closes new issue Normal OpenT2131 C4science: One of your repository is too big Normal OpenT2212 Images in readme Normal OpenT2384 Project members of child project cannot edit Normal OpenT2334 List repositories and access fields via web request Normal OpenT2322 Many tags Normal OpenT1975 .pdf files in SVN repository Normal OpenT1099 Faire une annonce sur les écrans polynex Normal OpenT1024 Utilisation de Jenkins en remote Normal OpenT2106 C4science: One of your repository is too big Normal OpenT948 Error message for Shibboleth authentication Normal OpenT945 Problem with Herald rules Normal OpenT919 Allow All Users to create Bot user Normal OpenT750 segmentation fault when arc install-certificate Normal OpenT632 Plugins in jenkins, `xUnit` `Warnings` Normal OpenT2223 No markdown rendering in Jupyter Notebooks Low OpenT1514 Edit a file as admin fatal with Undefined variable: can_manage Low OpenT1504 Uninstall Owners and Packages Low OpenT1437 Some items in mgmt panels of a repo are greyed Low OpenT1120 Remove Spaces Low OpenT969 Markdown doesn't render nested blockquotes Low OpenT1731 Cannot tag a repository Wishlist OpenT1571 Better visibility for wiki pages Wishlist OpenT1480 Is it possible to render md files that are not called README? Wishlist OpenT1459 Project global namespace Wishlist OpenT1375 Allow downloading code directly from the web interface Wishlist OpenT2486 GitHub API or something similar Wishlist OpenT1231 Rendering of arbitrary markdown files within repos Wishlist Open

18

T754 subprojects and member inheritance Wishlist OpenT333 Repo statistics Wishlist OpenT311 Markdown links for local files not interpreted Wishlist OpenT307 Show tags/branches in repo history Wishlist Open

11 C4science

11.1 Arcanist

11.1.1 Linters

A linter or lint refers to tools that analyze source code to flag programming errors,bugs, stylistic errors, and suspicious constructs.

• https://en.wikipedia.org/wiki/Lint_(software)• https://c4science.ch/w/c4science/lint/

In [45]: git checkout feature/arcanist-lintertree -a -I ".git"

Switched to branch 'feature/arcanist-linter'Your branch is up to date with 'origin/feature/arcanist-linter'...arclintcodeaa mylib.pyaa test.pyDockerfile.gitignoreJenkinsfileREADME


In [46]: arc linters

CONFIGURED text (Basic Text Linter)AVAILABLE csharp (C#)AVAILABLE cpplint (C++ Google's Styleguide)AVAILABLE cppcheck (C++ linter)AVAILABLE lessc (CSS pre-processor)AVAILABLE csslint (CSSLint)AVAILABLE chmod (Chmod)AVAILABLE gjslint (Closure Linter)AVAILABLE coffeelint (CoffeeLint)AVAILABLE composer (Composer Dependency Manager)CONFIGURED filename (Filename)

19

https://en.wikipedia.org/wiki/Lint_(software)

https://c4science.ch/w/c4science/lint/

AVAILABLE generated (Generated Code)AVAILABLE golint (Golint)AVAILABLE hlint (Haskell Linter)AVAILABLE jsonlint (JSON Lint)AVAILABLE json (JSON Lint)AVAILABLE jscs (JavaScript Code Style)AVAILABLE jshint (JavaScript error checking)AVAILABLE nolint (Lint Disabler)AVAILABLE merge-conflict (Merge Conflicts)AVAILABLE phpcs (PHP_CodeSniffer)AVAILABLE phutil-library (Phutil Library Linter)CONFIGURED pylint (PyLint)AVAILABLE flake8 (Python Flake8 multi-linter)AVAILABLE pep8 (Python PEP 8)AVAILABLE pyflakes (Python PyFlakes)AVAILABLE ruby (Ruby)AVAILABLE rubocop (Ruby static code analyzer)AVAILABLE script-and-regex (Script and Regex)AVAILABLE xml (SimpleXML Linter)CONFIGURED spelling (Spellchecker)AVAILABLE xhpast (XHPAST Lint)AVAILABLE php (php -l)AVAILABLE puppet-lint (puppet-lint)(Run `arc linters --verbose` for more details.)

In [47]: cat .arclint

{"linters": {"spelling-linter": {"type": "spelling"

},"filename-linter": {"type": "filename"

},"text-linter": {"type": "text"

},"python-checks": {"type": "pylint","include": "(\\.py$)"

}}

}

In [48]: arc lint --never-apply-patches --output summary --everything

20

Jenkinsfile:4:Auto-Fix: Trailing WhitespaceREADME:3:Auto-Fix: Trailing Whitespace at EOFcode/mylib.py:1:Advice: Missing Docstringcode/mylib.py:3:Advice: Missing Docstringcode/mylib.py:3:Advice: Old Style Classcode/mylib.py:3:Advice: Too Few Public Methodscode/mylib.py:8:Advice: Invalid Namecode/mylib.py:8:Advice: Missing Docstringcode/mylib.py:8:Advice: No Self Usecode/mylib.py:11:Warning: Mixed Indentationcode/mylib.py:11:Error: Tab Literalcode/mylib.py:11:Warning: Unreachablecode/mylib.py:11:Advice: Invalid Namecode/mylib.py:11:Advice: Missing Docstringcode/mylib.py:11:Warning: Unused Variablecode/mylib.py:11:Warning: Unused Argumentcode/mylib.py:12:Warning: Mixed Indentationcode/test.py:1:Advice: Missing Docstringcode/test.py:5:Advice: Missing Docstringcode/test.py:9:Advice: Missing Docstringcode/test.py:12:Advice: Missing Docstringcode/test.py:13:Error: No Member

12 C4science

12.1 Arcanist

12.1.1 Unit-Tests

Using the TAP (Test Anything Protocol) extension of Arcanist, we can report test units result toc4science

In [49]: git checkout feature/arcanist-unittestsgit submodule update --inittree -a -I ".git|.arcanist-extensions"

Switched to branch 'feature/arcanist-unittests'Your branch is up to date with 'origin/feature/arcanist-unittests'.Submodule path '.arcanist-extensions': checked out 'b89c749a90154676967a1601b62563fed0cbd572'..arcconfig.arclintcodeaa mylib.pyaa test.pyDockerfile

21

.gitignore

.gitmodulesJenkinsfileREADME


In [50]: cat .arcconfig

{"load": [".arcanist-extensions/tap_test_engine"

],

"unit.engine": "TAPTestEngine","unit.engine.tap.command": "pytest --tap-stream code/test.py"

}

In [51]: pytest --tap-stream ./code/test.py

ok 1 - Test.test_capitalizenot ok 2 - Test.test_capitalize_fail## self = <test.Test testMethod=test_capitalize_fail>## def test_capitalize_fail(self):# > self.assertTrue(self.m.Capitalize('TeSt') == 'test')# E AssertionError: False is not true## code/test.py:16: AssertionErrorok 3 - Test.test_return1..3

In [52]: arc unit --rev HEAD^

ok 1 - Test.test_capitalizenot ok 2 - Test.test_capitalize_fail## self = <test.Test testMethod=test_capitalize_fail>## def test_capitalize_fail(self):# > self.assertTrue(self.m.Capitalize('TeSt') == 'test')# E AssertionError: False is not true#

22

# code/test.py:16: AssertionErrorok 3 - Test.test_return1..3

PASS <1ms Test.test_capitalizeFAIL Test.test_capitalize_fail

#PASS <1ms Test.test_return

In [53]: cat Dockerfile

FROM dkdde/arcanistRUN apk add --update bash \

git \php7 \php7-curl \php7-json \py2-pip \

&& pip install pytest pytest-tap tap.py \&& rm -rf /var/cache/apk/*

RUN ln -s /arc/arcanist/bin/arc /bin/arc

ENTRYPOINT []


pipeline {


}


steps {sh 'git submodule update --init'sh 'arc unit --rev HEADˆ'

}}

}

post {always {

archiveArtifacts artifacts: '*.tap'step([$class: "TapPublisher", testResults: "*.tap"])

}

23

}

}

jenkins tap

13 That’s all folks !

13.1 Explore, Experiment and Test !

14 Questions ?

24

open science workshop - handout · open science workshop - handout december 4, 2018 1 so, you want...

Documents