software ac output - dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.carolinejay3.slides.pdf ·...
TRANSCRIPT
![Page 1: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/1.jpg)
So#ware as academic output
Caroline Jay and Robert Haines
![Page 2: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/2.jpg)
So#ware is now considered an academic output
• When does so#ware ‘count’ as an output?
![Page 3: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/3.jpg)
Historically (currently)
• So#ware is hidden – Describe what it does – Describing how it does it (the method) – ImplementaCon details • Pseudocode not sufficient (Thimbleby, 2003)
• So#ware is an add-‐on • So#ware should be required – it’s part of the scienCfic process
![Page 4: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/4.jpg)
What’s the ideal?
• Can we define a gold (or minimum) standard?
• Case study – HCI
![Page 5: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/5.jpg)
So#ware’s role in research
• So#ware is the tool that enables the research
• So#ware is enabling research in a new way or to a new group – So it is used for doing research AND is part of the research
• So#ware is the research
![Page 6: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/6.jpg)
So#ware is… …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
![Page 7: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/7.jpg)
So#ware is… (Computer ScienCst) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Workflow management
tools
![Page 8: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/8.jpg)
So#ware is… (BioinformaCcian) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Workflow management
tools
![Page 9: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/9.jpg)
So#ware is… (Biologist) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Workflow management
tools
![Page 10: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/10.jpg)
So#ware is… (Computer ScienCst) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Reasoners
![Page 11: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/11.jpg)
So#ware is… (Ontology author) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Reasoners
![Page 12: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/12.jpg)
So#ware is… (Computer ScienCst) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Data collecCon
app
![Page 13: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/13.jpg)
So#ware is… (Humanist) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Data collecCon
app
![Page 14: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/14.jpg)
So#ware is… (Computer ScienCst) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Analysis scripts
![Page 15: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/15.jpg)
So#ware is… (Clinician) …a tool to enable research
…the research
Developed Upfront ConCnuously
Becomes an output RetrospecCvely ConCnuously
Analysis scripts
![Page 16: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/16.jpg)
CHI 2016 best papers
• 23 papers – Pseudocode – Full analysis results of analysis – Data, source code – Tool, source code
![Page 17: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/17.jpg)
Example 1
• Can we predict if someone will view an item of dynamically updaCng web content?
![Page 18: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/18.jpg)
Final analysis: CHAID in SPSS – all data online.
Analysis process
• Step through the video – Manually define areas of interest – Log start and end Cmes
![Page 19: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/19.jpg)
![Page 20: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/20.jpg)
Example 2 • 100,000 fossils sibng in boxes • Photographed by volunteers • Can we catalogue them through crowdsourcing?
![Page 21: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/21.jpg)
![Page 22: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/22.jpg)
HCI research quesCon
• Should people be required to register before they can contribute to a study?
• No – but they should have the opCon.
Jay et al (2016). To Sign up or Not to Sign Up? Maximising ContribuCon Rates through OpConal RegistraCon. CHI 16.
![Page 23: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/23.jpg)
Extending the placorm
![Page 24: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/24.jpg)
Example 3
• hdp://idinteracCon.cs.manchester.ac.uk • ‘ABC: Using Object Tracking to Automate Behavioural Coding.’ CHI 2016.
• SSI blog post: hdp://goo.gl/kNVtFt Funded by EPSRC grant EP/M017133/1
![Page 25: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/25.jpg)
IDInteracCon
![Page 26: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/26.jpg)
So#ware as academic output
• So#ware citaCon • Credit • License
• So#ware isn’t a separate thing – So#ware could exist without the paper – Paper couldn’t exist without the so#ware
![Page 27: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/27.jpg)
So#ware as Academic Output • ComputaConal research should be: – Findable – Accessible – Interoperable – Reusable
• Academic so#ware should be: – Findable – Accessible – Reusable – Citable? – Extensible
And probably other things… This is a starCng point.
![Page 28: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/28.jpg)
Video processing: The Horror
• Install Linux (Ubuntu 14.04.2) • Install mencoder (version 1.1-‐4.8) – Plus all 33 of its dependencies • And all of their dependencies, and so on…
• Run, for every input file, one a#er another: $ mencoder <input> -‐vf crop=640:360:640:0 -‐nosound -‐ovc x264 -‐o <front-‐view-‐output> $ mencoder <input> -‐vf crop=640:360:0:0 -‐nosound -‐ovc x264 -‐o <side-‐view-‐output> $ mencoder <input> -‐vf crop=640:360:640:360 -‐nosound -‐ovc x264 -‐o <rear-‐view-‐output>
![Page 29: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/29.jpg)
Video processing: IDInteracCon
• Install docker – Works on Windows, OS X or Linux
• Run, once:
– This downloads the video processing pipeline and runs it over all of the input data automaCcally
• Come back tomorrow – For new data, just re-‐run the docker command above and it will only process new inputs
$ docker run -‐it -‐v <input-‐dir> -‐v <output-‐dir> idinteraction/video
![Page 30: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/30.jpg)
Object tracking: The Horror • Assuming that Linux is already installed • Install OpenCV (version 2.4.8)
– Plus all 34 of its dependencies • Plus all of the development libraries and header files • And all of their dependencies, and so on…
• Install C++ compiler • Install CMake • Install Git • Install CppMT (C++ version of CMT tracker)
– Clone from GitHub • Get the IDInteracCon specific version
– Configure the build – Compile to produce ‘cmt’ executable
• Run, for every output of the video processing stage, one a#er another:
$ cmt -‐-‐bbox=<initial bounding box> -‐-‐output <output dir> <video file>
![Page 31: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/31.jpg)
Object tracking: IDInteracCon
• Assuming that docker is already installed • Run, once:
– This downloads the object tracking pipeline and runs it over all of the input data automaCcally
• Come back tomorrow – For new data, just re-‐run the docker command above and it will only process new inputs
$ docker run -‐it -‐v <input-‐dir> -‐v <output-‐dir> idinteraction/tracking
![Page 32: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/32.jpg)
Open source so#ware
• Specify coding start point • Outputs data into files • Batch process videos • Define, adjust and edit the bounding box
• Changes merged back
![Page 33: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/33.jpg)
Making so#ware available
• Package (exe, Ruby gem, Python egg) • Docker image, virtual machine • Online service
• Source code
![Page 34: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/34.jpg)
To make it easier for ourselves and other invesCgators to use our automated pipelines, we have used so#ware containers to wrap up each stage of our method with everything needed for it to run. This guarantees that our tools will always run the same way, regardless of the environment they are running in, and ensures that any research is as reproducible as possible.
Our code and scripts
Standard tools and
dependencies
So#ware container Results
Automated Behavioural Coding
![Page 35: software ac output - Dagstuhlmaterials.dagstuhl.de/files/16/16252/16252.CarolineJay3.Slides.pdf · $ mencoder vfcrop=640:360:640:0 nosound"0ovcx264o](https://reader034.vdocument.in/reader034/viewer/2022051605/600ef6a01614f63cc3321aeb/html5/thumbnails/35.jpg)
The trouble with pseudo code • Thimbleby (2003) has illustrated an example which explains the
problem of pseudo code and why it is not suitable to be presented instead of source code in scienCfic papers. In the case of Porter’s stemming algorithm which was invented to find the canonical form of any word, for example programmable and programming words will both refers to “program” word. The problem of Porter’s algorithm is that, it was published in a nonprogrammable form thus many incorrect implementaCons were produced based on that form. Porter himself recognized misunderstanding as one of his algorithm’s problems. Consequently, many researchers have presented incorrect results by applying an incorrect version of Porter’s stemming algorithm in their published work. Regarding this, Thimbleby (2003) believes that it is much easier to provide the source code on a website, for example, rather than trying to extract a descripCon or pseudo code from the source code.