walkthrough occam · computer systems relies almost exclusively on artifacts compilers, analyzers,...
TRANSCRIPT
Walkthrough OCCAM
Be on the lookout for this fellow: The call-
outs are ACTIONs for you to do!
When you see the check mark, compare
your work to the marked element
Objectives
In this presentation you’ll
1. Learn what is OCCAM
◦ Why is it needed?
◦ What does it do?
2. Learn how to use OCCAM
◦ As an experimentalist
◦ Using the web interface
◦ Running and configuring experiments
2
OCCAM and the
curation of artifacts
Computer Systems
Relies almost exclusively on artifacts
◦ Compilers, analyzers, VMs, …
◦ Software simulation, …
◦ Hardware emulation, …
◦ Benchmarks, mini-apps, …
◦ Traces, data sets, …
◦ And a cast of a thousand other tools…
Empirical study keystone of rapid pace for a
huge market (e.g., $111B processor market†)
Innovation & Evaluation
† IMS Research, 2011
4
Prolific Artifact Production
Simulators
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Single-core X X X . . X X X X X X X X X X X X . X X X . X X X X X X X X X
Multi-core X X X . . X X X X X X . . X X . . . X X X X . X X . . X . . .
Homogen. multi-core X X . . X X X X X X . . X X . . . X . X X X X . . . X . . .
Heterogen. multi-core . . . X . . . . X . . . . . . . . . . . . . . .
SMT . . X X . X X X . X . . . . . .
Shared memory X X . . X X X X X . . X X X X . X X X X X X . . . . . .
Private memory . . . . . . . . . . . . . . . . . . . . . . .
Timing X X X . X X X X X X X X X X . X X . X X X X . X X . X X .
Cycle-accurate . X X . X X X X X X X X X X . X . X X X . X X . X X X
Functional . X . . . X X X X X X X X . . . X . X . . X X . X . X . .
Full system X X . . X X X X X . . X X . . . . X . . . X . . . . . .
Caching . . X X . X X X X X X X X X X X . X . X X X X X . X X X . X
Cache coherence . . . X X X X X . . X X X . X . X . X X . . . X . . .
In-order . . . . . X X X X X X X X . X . X . X X . . . X X . X X X
Out-of-order . . . . . X X X X X X X X X . . X . . . X X . . X . X . .
Superscalar . . . . . X X X X X X . X X X . . X . . X X X . X . X . .
Virtualization . . . . . . . . . . . . X . . . . . . . . . .
Virtual memory . X . . X X X . X X X . X X X . . X . . . . . .
VLIW . . . . . . . . X . . . . . . . . . . X . . . . . .
DRAM controller X . . . . . X X X X . X X . X . . . X X X . . . X . X . .
Scheduling . . . . . X X X X . X X . . . . X X X . . . X . X . .
Concurrency . . . . . X X X . . X X . . . . X X X . . . . . . . .
DRAM error sim . . . . . . . . . . . . . . . . . . . . . . .
On-chip network . . . . . . . . . . X . . . . . . X . . . . . .
System on chip . . . . . . . . . . . . . . . . . . X . . . . . .
Power consumption . . X X . . . X X . . . X X . . . . . . . X . . X . .
Gate-level . . . . . . . X . . . . . . . . . . . . . . . X . .
Validated . . . . . X X . X X . X . . X . . . . . . . X X .
5
I have a new idea. I want to find a
simulator, datasets, or other
artifact. What’s out there?FINDABLE
Found it! Can I download? Oh, yes,
but… it requires Windows! Or, it’s
missing something. Or…ACCESSIBLE
I downloaded it! Let me try using
it. Hmm. What does this mean?
What’s the format???INTEROPERATE
Downloaded, installed & ran. Let
me try changing it… Uh, Oh.
Where is that bug? No comments,
really??
REUSABLE
FAIR Guiding Principles
6
① Findable: It can be located.
② Accessible: It can be reached.
③ Interoperable: It can be combined.
④ Reusable: It can be reused, extended.
The FAIR principles lead to experimental
transparency, reproducibility & reusability
M.D. Wilkinson et al., “The FAIR Guiding Principles for scientific
data management and stewardship”, Scientific Data, March 2016
What we are doing…
7
Our goal: Establish and grow a community,
experiments (results) and an ecosystem for
modeling and simulation that is FAIR.
To be FAIR, we need Active Digital Objects
◦ Modifiable, Executable, Interactive
◦ Greatly simplify effort and accelerate productivity
◦ Leverage & collaborate on high quality models
◦ Interoperate & reuse models, both by user & machine
◦ May also include hardware resources
OCCAM: Open Curation for
Computer Architecture Modeling
8
Engage community to learn requirements,
needs, impediments and solutions
Research, prototype & deploy ideas for
enabling FAIR in modeling and simulation
① Build it to discover what does & doesn’t work!
② Bring value to community to attract contribution
③ Demonstrate the promise & feasibility
Educate community about the values of and
for FAIR modeling and simulation
OCCAM: Open Curation for
Computer Architecture Modeling
9
Engage community to learn requirements,
needs, impediments and solutions
Research, prototype & deploy ideas for
enabling FAIR in modeling and simulation
① Build it to discover what does & doesn’t work!
② Bring value to community to attract contribution
③ Demonstrate the promise & feasibility
Educate community about the values of and
for FAIR modeling and simulation
OCCAM: End-to-End and Lean
Digital Curation
10
Active Objects: Artifacts, Experiments & Results
Services & resources for full research lifecycle
Sharing and collaboration of content
Content is modifiable and runnable
① Innovate
② Evaluate
③ Review (AE)
④ Publication (Exp.)
⑤ Derivation (Exp.)
OCCAM: End-to-End and Lean
Digital Curation
11
Designed around users
Multiple user Roles
Exchange among users
Share & collaborate
Services for roles Develo
pers
Consumers
Experimentalists
Man
agers
OCCAM
OCCAM: Use Cases in the
Research Lifecycle
12
① Innovation and experimentation
◦ User workspace to manage & conduct research
◦ Compose to create experiments to test hypothesis
◦ Experiments are workflows
◦ Workflows generate results
◦ Analyze and visualize results
◦ Share with collaborators!
OCCAM: Use Cases in the
Research Lifecycle
13
② Peer review (or validation)
◦ Share completed experiments with reviewers
◦ Reviewers can view, modify & rerun
◦ Artifact Evaluation (http://artifact-eval.org)
③ Dissemination / sharing (post review)
◦ Provide access via OCCAM instance
◦ Groups of collaborators, or public
④ Publication & Derivation
◦ Associate experiments with peer-reviewed article
◦ Readers can derive new ideas/results
Quick Illustration of Two Uses
14
Demo of Innovation and evaluation
◦ Evaluate a continuous query (CQ) database on an
emerging memory architecture
◦ Two instances of CQ database on separate cores
◦ Database is resident in memory
◦ What is the effect of contention?
Demo of derivation from publication
◦ Peer-reviewed article published with experiments
◦ Experiments are modifiable, executable content
◦ Associated with article through digital library
Quick Illustration of Two Uses
15
Demo of Innovation and evaluation
◦ Evaluate a continuous query (CQ) database on an
emerging memory architecture
◦ Two instances of CQ database on separate cores
◦ Database is resident in memory
◦ What is the effect of contention?
Demo of derivation from publication
◦ Peer-reviewed article published with experiments
◦ Experiments are modifiable, executable content
◦ Associated with article through digital library
Flexible Deployment
16
① Private
② Cloud
③ Federated
④ Marshaled
⑤ Specialized
Demand Resources
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
OCCAMOCCAM
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
OCCAM
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
IP IP
IP IP
FPGA
Flexible Deployment
17
① Private
② Cloud
③ Federated
④ Marshaled
⑤ Specialized
⑥ Group
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
OCCAMOCCAM
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
OCCAM
consumer
web
portal
software
services FPGA
FPGA
Remote hardware
instance (e.g., emulator)
Remote hardware instance
(e.g., high-performance cluster)
Hardware ResourcesDigital Resources
experimentalist
developerdispatch
retrieve
manager
Repository
FPGA
FPGA
FPGA
FPGA
Emulator on FPGAs
Simulator on CPUs
GPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU
Simulator on GPGPUs
data
data
data
data
Database servers
artifact
benchmark
tutorial
experiment
IP IP
IP IP
FPGA
Using OCCAM –The
experimentalist’s
perspective
18
Step 1: Get a simulator
request
metadata
Obtain simulator
OCCAM
web
portal
19
Step 2: Import to Experiment
OCCAM
web
portal
build &
check
20
Step 2: Import to Experiment
OCCAM
web
portal
sim
build
provenance
build &
checkbin + env
21
Step 3: Define/Run Experiment
Experiment Editor
Configuration Editor
Form, visual, import
OCCAM
web
portal
sim
build
provenance
build &
check
define & run
experimentconfig
instantiate
dataset
exp.1exp.1exp.1exp.1
22
Step 3: Define/Run Experiment
OCCAM
web
portal
sim
build
provenance
build &
check
define & run
experiment exp.1exp.1exp.1exp.1
Pulls simulator + environment
Schedules & monitors simulations
Extracts results upon completion
Everything run in its own separate VM
Multiple job types (build, simrun, etc.)
config
instantiate
dataset
23
Step 3: Define/Run Experiment
OCCAM
web
portal
sim
build
provenance
define & run
experiment exp.1exp.1exp.1exp.1
results
config
instantiate
dataset
build &
check
24
Step 4: View Results
OCCAM
web
portal
sim
build
provenance
build &
check
resultsview
results
create graph
config
instantiate
dataset
Or data25
Containerization
o OCCAM runs simulators with Docker
• Lightweight “virtualization”
• Local & in-repository build & run
o Docker is used to create running environments
• A Docker base image
• Extended with dependences
• Docker specification generated
26
Containerization
Docker runs a daemon with root privileges.
◦ This is similar to vulnerabilities and attack surfaces in
traditional virtualization/hypervisor use cases.
◦ Only if you allow people to directly interact with the
daemon.
Only OCCAM has access to the Docker daemon
◦ Users only interact with the OCCAM software
◦ OCCAM crafts calls to Docker
does not allow arbitrary Docker commands
◦ Eliminating the known attack surfaces.
27
Containerization
28
Docker Vulnerability OCCAM Solution
Runs daemon as root. Run webserver and OCCAM daemon as a
different non-interactive user with access
to Docker.
Mounting of system files read/write OCCAM crafts the Docker commands and
sanitizes them to prevent mounting of
arbitrary directories.
Allows root escalation through the
writing of root-owned files
This is actually mitigated by using user
namespace mapping, which is a Docker
feature that isn’t on by default. Just turn
that on.
What we are doing...
Learn how to use OCCAM as an experimentalist
1. Import a simulator into OCCAM
2. Create and configure an experiment
◦ Experiment: Metadata to run a simulation
Benchmark
programSimulator
Plotting
tool
Docker containerDue to time constrains
a Docker container
with SST is already on
your system
29
What we are doing...
Learn how to use OCCAM as an experimentalist
1. Import a simulator into OCCAM
2. Create and configure an experiment
◦ Experiment: Metadata to run a simulation
3. Launch (run) experiment
◦ Schedules the simulation on a simulation engine
4. Visualise experimental results
◦ Interactive plots
30
OCCAM user interfaces
OCCAM has two means of interaction
◦ CLI – Command Line Interface
Advanced users (developers)
Using scripts
◦ WI – Web Interface (Graphical)
For everyone
Create, configure, and run experiments
We will be using
this for now
31
Accessing the web interface
Use your laptop’s web browser to access your
OCCAM instance
Follow instructions in the handout
Warning: Your browser may complain about the certificate.
When this happens, follow the browser
instructions to add an exception.
32
(a) Access to
https://<IP ADDRESS>
Home page
Login
button
Signup
buttonSearch
bar
33
Create a user
(a) Click Signup
(b) Type a username & password
(c) Click “Sign Up”
34
Creating a Workset
Worksets are like folders
◦ They hold the different OCCAM objects
Simulators, experiments, etc.
(a) Type a name for a new workset
e.g.: “Exercise 1”
(b) Click to create35
Workset
Contributors
Content
Workset settings
36
Workset
Public/Private
37
Workset
Add a bookmark
(user screen)
38
Workset
Clone the workset
39
Workset
View workset
history
40
Workset
Authors
Have permissions to
modify all contents
of a Workset.
Collaborators
Can be given have
different permissions
than authors
41
Workset
Import objects
(e.g. github)
Create new objects
(e.g. experiments)
42
Importing existing objects
Importing OCCAM objects into a Workset
Can be imported from a git repository
◦ Lets import an object from bitbucket
◦ Find the link in your handouts
(a) Choose git
repository
(b) Type the URL
(check your handouts)(c) Click import
43
Importing existing objects
You’ve just imported a simulator
Simple MIPS processor
◦ Lets use it in an experiment
You’ll see the imported
simulator here
44
Creating a new experiment
You’ve just imported a simulator
Simple MIPS processor using the SST framework
◦ Lets use it in an experiment
Lets create an experiment that
◦ Runs the simulator you imported
◦ Plots the results for inspection
45
Experiments – Workflows
First create a new experiment
(a) Select type
experiment (b) Give it a name
(c) Click add
46
Experiments
Workflow
Configurations
47
Experiments
Workflows are DAGs that represent the
execution of an experiment
This one is empty, lets add the plotting tool
Experiments are built from the end
Workflow
48
Experiments
Workflow
(a) Click the empty node
(b) Enter the type:
script
(c) Enter the name:
plotter
(d) Click attach
49
Experiment
Workflow
The plotting tool is in the
workflow!
The final block on the
workflow represents the
output of the experiment
50
Experiments
Workflow
This plotting tool can plot the output of the
simulator
The next step is to add the inputs to the plot
We’ll be adding the simulator we imported
However, we need to add an intermediate block
◦ Intermediate storage for the simulator’s output
51
Experiments
Workflow
(a) Click the plus sign
(b) Enter the type:
application/json
(c) Select the option:
{new application/json}
(d) Click attach
52
Experiments
Workflow
(a) Click the plus sign
(b) Enter the type:
simulator
(c) Enter the name:
XSim-demo
(d) Click attach
53
Experiments
Workflow
The type of the intermediate block
is the same as the output of our
simulator and the input of the
plotting tool
54
Experiments
Workflow
Finally lets add some input to the simulator
We’ve provided a program to you
(a) Click the plus sign
(b) Enter the type:
program/XSim
(c) Enter the name:
XSim-program
(d) Click attach55
Experiments
Workflow
We’ve just created a
workflow.
Now we can configure it
56
Experiments
Configurations
Xsim-demo configurations
Configuration
options
57
Experiments
Configurations
(a) Change the Xsim-demo
configurations as you see fit
(b) Click update to save any
changes
58
Experiments
Configurations
(a) These options are
configured by default
(b) DO NOT CHANGE THESE
These options select the data to
be plotted from the simulation
output
59
Experiments
Configurations
(a) You can also try to change
the plotter- Figure Options
(b) Click update to save any
changes
60
Experiments
Configurations
61
The update button saves the changes
you make in the experiment!
Whenever you change a parameter,
you must click this button
In every configuration tab
Experiments
Running an experiment is as easy as clicking a
button!
Running
(a) Click on the run tab
62
Experiments
Running an experiment is as easy as clicking a
button! (or two )
Running
(b) Click run63
Experiments
Running
If you see some errors like these, don’t worry!
Docker is just a bit verbose when checking if the
containers exist, and when they do not OCCAM
creates them!
64
Experiments
Running
(a) Once your run is complete,
you need to refresh the page to
see the results
65
Experiments
Running
(a) Click the Output tab
(b) Open the plot
66
Experiments
Running
(a) Your plot should look
like this (b) Check the provenance
tab of the plot
67
Experiments
Running
(a) You can check how
this plot was created
Inspect the experiment
that generated itThe machine where it ran
The software that created it
68
Summary
We presented OCCAM
A community-supported digital curator &
exchange for simulation, emulation,
benchmarking, experimental results
OCCAM allows to easily deploy and run
software
◦ Making reproducing results easy
◦ An fostering software reutilization
OCCAM is easy to use
◦ But how easy is it to create a simulation in OCCAM?
◦ We’ll create a simulation in the next exercise69
Acknowledgments
70
We gratefully acknowledge the support and collaboration of our
research partners.
Portions of this material is based in part upon work supported by the National Science
Foundation under Grant Numbers CCF-142331, CNS-1012070, ACI-1535232, CNS-
1305220 and CCF-1148646. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the National Science Foundation.
http://occam.cs.pitt.eduhttp://sst-simulator.org