bringing your favorite analysis applications to iplant using docker containers nirav merchant
DESCRIPTION
3 + = Simple Formula for SuccessTRANSCRIPT
![Page 1: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/1.jpg)
Bringing your favorite analysis applications to
iPlant using Docker containers
Nirav [email protected]
![Page 2: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/2.jpg)
Topic Coverage:• Which app can you bring to the Cyverse • Where can you run your app ? • Choosing the right platform to run your app at Cyverse• What is container technology• Benefits of running your container at Cyverse• Taking your container from laptop to Cyverse• Sharing your app with the world (using DE+Docker)• Hands on walk through
![Page 3: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/3.jpg)
3
+ =
Simple Formula for Success
![Page 4: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/4.jpg)
The Reality
4
+ +
• Excel, R • PERL • Python • ARCGIS• Java Ruby• Fortran C C#
C++ Matlab• etc.
• Amazon• Azure• Rackspace• Campus
HPC• XSEDE• Etc.
and lots of glue…..
![Page 5: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/5.jpg)
+ =
Simple Formula
![Page 6: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/6.jpg)
Where can you run your apps• Look at the capability of your application to use CPU,
RAM and run time needed• What happens when you run a job in:• DE (Regular and HPC)• Agave• Atmosphere• Bisque
• Pains of bringing your app to Cyverse
![Page 7: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/7.jpg)
Container technology: What is it about ?• Allows you to create a self contained package that
contains:• The specific operating system version (say Ubuntu
14.04.1)• Your application • All of the parts your application needs (such as
libraries and other dependencies)• Ability to share this with other users
• This single package can now be run on any computing system that supports Container technology (regardless of its own version of operating system)
![Page 8: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/8.jpg)
Container technology: Docker• https://docs.docker.com/engine/introduction/understa
nding-docker/
• Has many interesting features and capabilities• Parts of Docker you need to know about:• Docker client/command line (CLI)• Docker file• Docker image• Docker registry• Docker container
• Most important concept of working with large amount of data in Docker: The union file system
![Page 9: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/9.jpg)
How does it work together
![Page 10: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/10.jpg)
What happens when you run a job in DE:• Condor looks for a machine that matches your
criteria (RAM, CPU, Disk Space)• Once it find a suitable match:• Data placement container runs and brings the
data you want to operate on to that node from data store
• Your app (Docker container) runs (with the data visible to it as union file system)
• Date placement container for returning data data back to data store
![Page 11: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/11.jpg)
How do you get started• Check the step by step instructions in Wiki at:• https://goo.gl/Od6y9Z
• Get Docker setup on your local machine (win,mac,linux) or use Atmosphere
• Plan your steps i.e what you want to do• Carry out those steps and verify that things work• Create a Docker file file from those steps• Submit the request for a “new tool”• Once you hear back design your interface (and
profit)
![Page 12: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/12.jpg)
Future Directions• Ability to bring any Docker images from private
repository, Docker hub, files etc.• Share your app/container within your group• Ability to bring your own compute for containers
and attach it to Cyverse pool and manage who can send jobs using it
![Page 13: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/13.jpg)
Word of Caution• Containers are very powerful and has many bells
and whistles (only choose parts that you really need !)
• Avoid storing data inside of containers• Keep containers light and nimble, build on provided
base images from trusted source (iPlant prefers Ubuntu 14.X and CentOS 7.X from Docker hub)
• Do not trust a app without Docker file (its not easy to recreate and a blackbox, bad for reproducibility )
![Page 14: Bringing your favorite analysis applications to iPlant using Docker containers Nirav Merchant](https://reader035.vdocument.in/reader035/viewer/2022062401/5a4d1b6e7f8b9ab0599b466b/html5/thumbnails/14.jpg)
Thanks• Linux community for containers• Docker for making things more useable • Cyverse/iPlant Core SW team for integrating Docker• Eric Lyons for writing the first tutorial• Many users for building Docker files and submitting
those !!• Whole iPlant and Cyverse team and community