co-funded by the european union · overview, status update andaccess. outline time title speaker(s)...
TRANSCRIPT
Co-funded bythe European Union
Co-funded bythe European Union
Fenix Research InfrastructureOverview, status update and access
OutlineTime Title Speaker(s)
17:35 - 17:45 Overview of the Fenix Infrastructure and the ICEI project Dirk Pleiter (JUELICH-JSC)
17:45 - 17:55 Implementation status update Colin McMurtrie (ETHZ-CSCS)
17:55 - 18:25 Use case presentations (5 min each)• Cerebellar simulation (SP1)• Data acquisition and analysis in the context of human
brain atlasing (SP2/SP5)• Hippocampus simulation (SP6)• Massive open online courses (MOOC) (SP6)• Fenix and the Neuromorphic Computing Platform (SP9)• Neurorobotics PaaS (SP10)
Egidio D'Angelo (UNIPV)Timo Dickscheid (JUELICH-INM1)
Michele Migliore (CNR)Jean-Denis Courcol (EPFL)Andrew Davison (CNRS) / Eric Müller (UHEI)Hossain Mahmud (FORTISS) / Susie Murphy (EPFL)
18:25 - 18:35 Access to Fenix resources Giuseppe Fiameni (CINECA)
18:35 - 18:45 Q&A All
Co-funded bythe European Union
ICEI Overview
Fenix Goals
• Establish HPC and data infrastructure services for multiple research communities• Encourage communities to build community specific platforms• Delegate resource allocation to communities
• Develop and deploy services that facilitate federation• Based on European and national resources• Science community driven approach
• Infrastructure realisation and enhancements based on co-design approach• Science communities providing resources to realise infrastructure
→ HBP SGA Interactive Computing E-Infrastructure• Resource allocation managed by community
Fenix Partners
• Currently involved centres• BSC (ES)• CEA (FR)• CINECA (IT)• CSCS (CH)• JSC (DE)
• Foreseen extensibility• Open for more partners and stakeholders
Infrastructure vs. Platform Services
Platform services
User communities
Infrastructure services
ICEI Infrastructure Services
• Computing services• Interactive Computing Services• (Elastic) Scalable Computing Services• VM Services
• Data services• Active Data Repositories• Archival Data Repositories• Data Mover Services, Data Location and Transport Services
• Service federation• Cope with variety of data sources and make data available to the wider
community• Provide access to a diversity of computing capabilities at different sites
ICEI Science and Use Cases for Co-Design# Working Title PI1 Data-driven cellular models of brain regions, Olfactory Bulb Migliore3 Learning-to-learn (LTL) in a complex spiking network on HPC and Neuromorphic hardware
interacting with NRPMaass, Meier
5 Large scale simulations of models: Cerebellum D' Angelo6 Large scale simulations of models: Hippocampus Migliore7 Elephant big data processing Grün, Denker8 Mouse Brain Atlas Pavone9 Towards a novel decoder of brain cytoarchitecture using large scale simulations Poupon, Axer
10 Multi-scale co-simulation: Connecting Arbor/Neuron, NEST and TVB to simulate the brain Morrison, Destexhe, Diesmann, Jirsa
11 Neurorobotics platform, large-scale brain simulations von Arnim, Cruz12 BBP columnar simulation Schürmann13 Ilastik as a service on the HBP Collaboratory Kreshuk14 Online visualization of multi-resolution reference atlases Amunts15 Data management and big data analytics for high throughput microscopy Dickscheid
16 Multi-area macaque NEST simulation with life visualization and interaction v. Albada, Diesmann
17 Data management and big data analytics for large cohort neuroimaging Caspers, Eickhoff
Resource Allocation Model
• Actors• Fenix Resource Providers• Fenix Communities• Fenix Users
• Role of Fenix Resource Providers• Provide fixed amount of resources for given period to Fenix Communities• Define rules for resource allocation (e.g., peer-review process)
• Fenix Users• Submit proposal for resources to relevant Fenix Community
• Fenix Community• Review proposal and award available resources to Fenix Users
Co-funded bythe European Union
ICEI - Implementation StatusBringing ICEI Infrastructure Services and Resources to the HBP
Fenix/ICEI provides the Base Infrastructure for HPAC
ICEI Resources for HBP • ICEI resources have already been made
available to the HBP and PRACE by CSCS
• There are currently 4 HBP projects with compute allocations at CSCS
• Usage of the Swift Object Storage is growing by >1% per month
• More resources are available than are being consumed so HBP users are encouraged to apply for a compute allocation
How do I use ICEI Resources? (1)Swift Object Storage:SWIFT OS can be accessed directly from your personal computer
• GUI clients e.g. CyberDuck• SP5 Python Library
• Better for mgmt. of the ACLs and Object Buckets
• https://hbp-archive.readthedocs.io/en/latest/
Reachable from inside the Collaboratory• Get/Put from Jupyter Notebooks• More capabilities coming soon
Active Data Repositories:• Comes with the compute allocation (= $SCRATCH)• Low-latency storage tier
Archival Data Repositories:• Available either as part of a computing request or
separately (proposal needed)
How do I use ICEI Resources? (2)
Pollux OpenStack IaaS:The Pollux OpenStack IaaS is available to host your platform VMs:• Accessible globally via the Horizon GUI interface• RESTful API can be used for automation
How do I use ICEI Resources? (3)Scalable Compute Resources:The Piz Daint system is available as a state-of-the-art scalable compute resource for use by HBP users• Accessible globally via Command-line Interface• Via the Unicore GUI• RESTful API offered via UNICORE for platforms
• Use of Service Accounts for Platforms is also acceptable at some sites (e.g. CSCS)
How to get HelpGeneral Contact for HPAC Platform:
• HPAC Platform: https://collab.humanbrainproject.eu/#/collab/264/nav/2378
How to apply for resources:• Send your proposals to: [email protected]
Getting help:• Send emails to: [email protected]
Examples of use case presentations
Co-funded bythe European Union
HippocampusMichele Migliore (CNR)
Motivation/Introduction
simplified net
real time
bottom-up
top-down
neuromorphic HW
clinicalimagingpharma
EXP
Join us for an EITN workshop on 28-29 January.
We would like to assemble a number of multidisciplinary HBP partners interested in working together on implementing a multilevel model of hippocampal functions and dysfunctions.
Requirements/ ICEI resources/ current problems• 450000 neurons, ~90·106 memb seg, 20 ODE/seg + synapses, ~ 2·109 ODEs
• 1” of sim time: 3-9hr on Piz Daint (CSCS) or Jureca (JSC) using 10000 cores
• ~2Tb of input, up to ~1-4Tb of output
• 50-90000 node/hours every 3 months, 20Tb of storage
Current use of ICEI resources
• 60M node/hours per year, 100Tb of storage for simulation and analysis
• offline interactive visualization
• User-friendly access from the Collaboratory (currently a blocking issue)
Planned use of ICEI resources
Co-funded bythe European Union
Co-funded by the European Union
Neurorobotics PlatformPaaS Use Case for SP7
Susie Murphy (EPFL)Hossain Mahmud (fortiss)
HBP Summit, Maastricht 16.10.2018
Co-funded by the European Union
What is the Neurorobotics Platform (NRP)?
● The Neurorobotics Platform's (NRP) purpose is to allow researchers to perform experiments on virtual robots using brain models
● In order to deal with huge simulations involving brains with millions of neurons we need access to supercomputers
● To achieve this we are working with CSCS to deploy the NRP on the Piz Daint supercomputer
Co-funded by the European Union
NRP on Piz Daint: Plan
Backend container
Frontend
Shared storage
1. Using user’s OIDC token submit request to start job(s)
Unicore
2. Job is allocated and started
3. Using a tunnel the backend sends data back to frontend and vice versa
Nest containers
Piz Daint0. HBP user has been allocated resource time on Piz Daint
Co-funded by the European Union
NRP on Piz Daint: Roadmap
SP7-SP10
Piz Daint STEP 1 Piz Daint STEP 2 Piz Daint STEP 3 Piz Daint STEP 4
2.1 October 18 2.2 March 19 2.3 October 19
Running ~10⁵ neurons in real-time ~10⁵ neurons
Run whole backend in a single container on Daint
Run whole backend AND multi-process NEST on single container on Daint
Run whole backend and spawn NEST multi-process on other nodes
Run multiple backends that spawn NEST multi-process in production
Co-funded by the European Union
NRP on Piz Daint: Accomplishments
● Having the backend container running on Piz Daint○ Docker image had to be importable into shifter
● Having OpenGL and GPU context from docker containers● Establishing bidirectional data flow to bypass the firewall○ Secure, easy and restricted method had to be set up
● NRP running on Piz Daint prototype setup
Co-funded by the European Union
NRP on Piz Daint: Challenges
● Authenticating HBP users at CSCS○ Right now HBP accounts and CSCS accounts are linked manually
● Procedure for updating docker images for users○ Currently images go to user space, and need to be updated manually○ Centralised repository for platform provider is needed
● Storing user data○ Centralised data store, which can be accessed within and outside Piz
Daint○ Need for mechanism to allow users to share data between them○ How to transport huge files
Co-funded by the European Union
Piz Daint
NRP on Piz Daint: Current Status
Backend container
Frontend
Shared storage
Piz Daint jobs can be started using OIDC, only dummy jobs have been tested
Unicore
Manual set up of a backend/frontend data flow successful
Nest containers
End solution not figured out yet. Not needed until step 3 / 4 of roadmap
At the moment Nest is inside the backend container. Planned for step 3 of roadmap
Co-funded by the European Union
NRP on Piz Daint: Roadmap status
SP7-SP10
Piz Daint STEP 1 Piz Daint STEP 2 Piz Daint STEP 3 Piz Daint STEP 4
2.1 October 18 2.2 March 19 2.3 October 19
~10⁵ ~10⁵
Run whole backend in a single container on Daint
Run whole backend AND multi-process NEST on single container on Daint
Run whole backend and spawn NEST multi-process on other nodes
Run multiple backends that spawn NEST multi-process in production
Co-funded by the European Union
http://neurorobotics.net/
/HBPNeurorobotic
@HBPNeurorobotic
Thanks for listening!
Access to Fenix resourcesGiuseppe Fiameni | Debora Testi - CINECA
Agenda■ Access mechanisms ■ Access mechanism - HBP■ Access mechanism - PRACE■ Available resources in 2018■ Available resources in 2019
Fenix: access mechanism
■ Fenix Resource Providers vs. Fenix Communities■ Fenix Resource Providers provide and operate
resources■ Fenix Communities decide
on resource allocation
Fenix: access mechanism via HBP
■ Current procedure: ■ Send 2-3 page proposal to [email protected]■ ICEI will perform a technical review■ HBP Directorate will decide on awarding the resources■ ICEI will make resources available
■ Non HBP members can apply for resources via PRACE■ First groups started to use infrastructure
■ HBP projects awarded allocations at CSCS and are already using them
ICEI PMO
HBP DIR
(1) Proposal
(6) Decision result
ICEI Sites
(4) Decision proposal
(8) Support
(7) Information about allocated resources
(5) Decision result
ICEI Experts
(2) Review request
(3) Review results
Fenix: access mechanism via PRACE■ Pilot phase in place■ Optional ICEI resources are available in call #18 of PRACE
Tier-0 resources ■ Call is now open up to the 30th of October■ PRACE performs scientific review■ ICEI provides technical feedback on the Fenix aspects
■ If successful, ICEI resources will be included in 2019 Tier-0 calls
■ A combined PRACE-ICEI DECI call will be opened in 2019
Fenix: available resources in 2018In Q4 2018, resources are available at CSCS:
Total ICEI HBP (25%) PRACE (15%) Unit
Scalable computing 186’204 116’344 69’860 node * hours
Interactive computing 297’840 186’150 111’690 node * hours
VM services 784 490 294 # VMs
Archival data repository 1’600 1’000 600 TB
Active data repository 32 20 12 TB* day
Fenix: available resources in 2019Provisional numbers (subject to change):
Total ICEI HBP (25%) PRACE (15%) Unit
Scalable computing 1’800’000 465’000 279’000 node * hours
Interactive computing 1’311’000 327’000 196’000 node * hours
VM services 9‘000 2’270 1’362 # VMs
Archival data repository 22’000 5’500 3’300 TB
Active data repository (every quarter)
600 160 100 TB* day
Fenix Collab
https://collab.humanbrainproject.eu/#/collab/28520/nav/200129
Q&A
Co-funded bythe European Union