mpack based operations model
TRANSCRIPT
![Page 1: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/1.jpg)
Jayush Luniya
Apache Ambari PMC
Principal Software Engineer @ Hortonworks
MPACK BASED OPERATIONS MODEL
![Page 2: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/2.jpg)
Agenda
Agenda
![Page 3: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/3.jpg)
Agenda
Stack Based Operations Model
Management Packs – V1
Limitations
Mpacks Based Operations Model
Summary
Q&A
![Page 4: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/4.jpg)
Stack Based Operations Model
Agenda
![Page 5: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/5.jpg)
Stack Based Operations Model
![Page 6: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/6.jpg)
Stack Based Operations Model
Ambari built on a standardized stack based operations model
Pluggable architecture; no vendor lock-in
Stacks wrap services of all shapes and sizes
Consistent definition & lifecycle control layer
Consistent approach for managing & monitoring services
Stacks embedded in Ambari
![Page 7: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/7.jpg)
Management Packs – V1
Agenda
![Page 8: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/8.jpg)
Management Packs – V1
Introduced in 2.4
V1 release based on existing stack based operations model
First take on decoupling stack releases
Bundles stacks, add-on services
Final stack definition is overlay of multiple mpacks
Stacks don’t need to be embedded in Ambari
![Page 9: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/9.jpg)
Management Packs – V1
![Page 10: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/10.jpg)
Service Level Extension Points
Extension Apache JIRA Apache Wiki
Stack Advisor Extensions AMBARI-15226 Ambari Wiki
Role Command Order Extensions AMBARI-9363 Ambari Wiki
Quick Links Extensions AMBARI-11268 Ambari Wiki
Service Repos Extensions AMBARI-15538 Ambari Wiki
Upgrade Pack Extensions AMBARI- 15388 Ambari Wiki
![Page 11: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/11.jpg)
Limitations
Agenda
![Page 12: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/12.jpg)
Limitations
Cluster attached to single stack Cannot deploy HDFS from HDP, NiFi from HDF in same cluster
Entire cluster upgrade
Stacks are monolithic
Ambari upgrade for refreshing stacks
No clear upgrade story for add-on services
Ownership of service deployment scripts
![Page 13: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/13.jpg)
Mpack Based Operations Model
Agenda
![Page 14: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/14.jpg)
Goals
Replacement for monolithic stacks
Cluster to multiple mpacks
Scenario: Deploy HDP +HDF cluster
Decouple Ambari core release
Independent release cycles
First class support for add-on services
Support dynamic installation
Modular upgrades
![Page 15: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/15.jpg)
New Concepts
Deployment artifact
Individual service ownership
Independent release cycles
Examples
HDFS-3.0.0-packlet
SPARK-2.0.0-packlet
Distribution artifact
Composition of packlets
Modular release & upgrades
Examples
ODPi-2.0.0-mpack
HDP-3.0.0-mpack
HDF-3.1.0-mpack
Packlets Management Packs (Mpacks V2)
![Page 16: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/16.jpg)
New Concepts
Catalog of management packs
Compatibility between mpacks
Search and discover mpacks
Support multiple mpack repos
Examples
ODPi Mpack Repository
Hortonworks Mpack Repository
Mpack Repository
![Page 17: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/17.jpg)
Mpack Based Operations Model
![Page 18: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/18.jpg)
Mpack Processing Engine
Mpack Repository
DistroX3.0.0
DistroY3.2.0
![Page 19: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/19.jpg)
Release Management
Completely decouple Ambari releases
Service ownership with domain experts
Independent release cadence
![Page 20: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/20.jpg)
Mpack Based Upgrades
Modular upgrades
Upgrade subset of services in a cluster that belong to mpack
Upgrade packs defined at mpack level
Download new version of mpack and kickoff upgrade
Upgrade Pack
“upgrade-from” instead of “upgrade-to”
![Page 21: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/21.jpg)
Add-on Mpacks
First class support for 3rd party services
Mpacks provide a 3rd party onboarding platform
Add-on services not overlaid into stack
Independent release management
Independent and modular upgrades
![Page 22: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/22.jpg)
Summary
Agenda
![Page 23: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/23.jpg)
Summary
Modular build & release
Independent release mgmt
Modular upgrades
3rd party onboarding platform
Multi-everything vision
Service level refactoring
Resource management library
No network connectivity
Ownership
Testing
Key Takeaways Challenges
![Page 24: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/24.jpg)
Summary
Future YARN Services
Umbrella Apache Epic
AMBARI-14714 – Multi Everything Architecture
Apache Epic
AMBARI-19621 – Mpack Based Operations Model
Dataworks Summit Talk
Future of Apache Ambari – Jaimin, Jayush
![Page 25: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/25.jpg)
Q&A
Agenda
![Page 26: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/26.jpg)
Backup Slides
Agenda
![Page 27: Mpack Based Operations Model](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64ce9b7f8b9a900f8b49db/html5/thumbnails/27.jpg)