bde technical webinar 1 : bde platform architecture
TRANSCRIPT
1ST TECHNICAL WEBINAR
BDE Platform Architecture02.03.2016
Targets of the BDE platform
1-mars-16www.big-data-europe.eu
Easy to use
1-mars-16
Flexible
Applications running on the platform
1-mars-16www.big-data-europe.eu
Pipeline
o Communicating components
to solve a specific problem
Example
o HDFS nodes
o Spark master/workers
o Spark map-reduce algorithm
Platform architecture
1-mars-16www.big-data-europe.eu
Platform architecture
1-mars-16www.big-data-europe.eu
Platform architecture
1-mars-16www.big-data-europe.eu
1-mars-16www.big-data-europe.eu
Installation
Manual installation
o https://github.com/big-data-europe/README/wiki/Base-platform
Chef recipes
o Temporarily outdated
o https://github.com/big-data-europe/chef-base-platform-cookbook
1-mars-16www.big-data-europe.eu
1-mars-16www.big-data-europe.eu
Docker in a nutshell
Docker
o Build, ship and run any application on any platform
Docker containers
o =~ lightweight virtual machine
Docker image
o Template of a Docker container
1-mars-16www.big-data-europe.eu
Developing a component
1-mars-16www.big-data-europe.eu
Base Docker images
provided by BDE
Extend image with
custom algorithm/data
with minimal effort
Developing a pipeline
1-mars-16www.big-data-europe.eu
Pipeline = collection
of communicating
components
Describe pipeline
in 1 file using
Docker Compose
Supported components
- https://github.com/big-data-
europe/README/wiki/Components
- https://hub.docker.com/u/bde2020
1-mars-16www.big-data-europe.eu
GitHub
DockerHub
1-mars-16www.big-data-europe.eu
1-mars-16
docker-compose upCustom GUI is being built
Docker Swarm
Resources are managed by Swarm
Integrates with Docker Compose
Automatic download/install/run
1-mars-16www.big-data-europe.eu
1-mars-16www.big-data-europe.eu
Future work: Monitoring
1-mars-16www.big-data-europe.eu
1-mars-16
THANK YOU