cern it department ch-1211 genève 23 switzerland t aims2 automated installation management system

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

aims2

Automated Installation Management System

CERN IT Department

CH-1211 Genève 23


it

Aims2 – Linux Automated Installation Management System - 2

Overview of Presentation

• Give you an explanation of AIMS–Introduce you to the installation

infrastructure at CERN• My Project

–AimsRewrite–aims2

• Overview• New Features• The future

CERN IT Department

CH-1211 Genève 23


it


What is AIMS?

• Automated Installation Management System– Perform parallel installations minimising the need

for human intervention– Makes use of PXE and ELILO.– The system is based on and extends the Kickstart

software from the RedHat distribution. – Supports Linux and Windows.– Around since 2000, Perl code base.

• Used throughout CERN– Experiments– Fabric/Grid deployment– General infrastructure user

CERN IT Department

CH-1211 Genève 23


it


Installation Infrastructure

CERN IT Department

CH-1211 Genève 23


it


Architectures & Bootloaders

• Network boot loaders need to bootstrap the device, load a configuration and the kernel.

• I386, x86_64– Supported by pxelinux.0 loader– Based on syslinux

• ELILO– ElILO is the EFI Linux boot loader for IA-64(IPF),

IA-32(x86), and x86_64 EFI-based platforms.

• Each architecture has its own way of doing things.

CERN IT Department

CH-1211 Genève 23


it

Installation boot sequence

• Client makes DHCP request• DHCP service replies with NEXT_SERVER

(lxpxeboot.cern.ch) and location of the network bootstrap file to use, dependant on the client architecture

if option client-architecture = 00:00 { # Intel x86PC - in use now.

option LINUX.pxelinux-magic F1:00:74:7E;

option LINUX.pxelinux-pathprefix "aims2/";

option LINUX.pxelinux-reboottime 50;

filename "aims2/loader/pxelinux.0";

} else if option client-architecture = 00:02 { # EFI Itanium - in use now.

filename "aims2/loader/elilo64.0";

} else if option client-architecture = 00:06 { # EFI IA32 - future extension (think Intel Apple ..)

filename "aims2/loader/elilo32.0";

}

CERN IT Department

CH-1211 Genève 23


it

Boot sequence cont. (PXE)

• pxelinux.0 is invoked and starts to look for client configuration– By client system UID ( XXXXXXXX-XXXX-XXXX-XXXX-

XXXXXXXXXXXX )

– By hardware address, appending 01

– Client IP address, HEX encoded, stripping one byte and retrying

– default

• default contains info about the “kernel” to be loaded

default mainlabel main kernel /loader/vesamenu.c32 append /pxelinux.cfg/main.conf

• vesamenu.32 takes over and builds interactive boot menus from config file

CERN IT Department

CH-1211 Genève 23


it

Boot sequence cont. (PXE)

• ELILO is a little different– Boot program is elilo64.0– Does not try to load system UID– Appends ia64 to encoded IP address– “default” is elilo-ia64 or elilo.conf

image=/aims/boot/SLC4X_IA64/vmlinuz

label=slc4X

description="Install Scientific Linux CERN 4 on ia64 (graphics console)"

read-only

initrd=/aims/boot/SLC4X_IA64/initrd

append="load_ramdisk=1 maxcpus=1 network keymap=us lang=en_US.UTF-8 ip=dhcp method=http://linuxsoft.cern.ch/cern/slc4X/ia64/"

• In both cases, we use the hardware address of the device and use the architecture of the image to decide which configuration to use.

CERN IT Department

CH-1211 Genève 23


it


aimsRewrite Project

• A solution to meet the new modern requirements of CERN and its users– A rethink of what LA is providing as a remote

installation service

• Move away from AFS dependency– Kickstarts, /tftpboot/ sync'ing

• Delegate– PXE image management– Device authorisation

• Reduce maintenance/administration overhead– Improved logging and auditing

CERN IT Department

CH-1211 Genève 23


it


DHCP

• Already completed– Useful, and flexible

• DHCP/BOOTP behaviour passed back to CS– No messing around with DHCP configurations.

• Operating System in LANDB dictates which NEXT_SERVER the client is sent to.

• LINUX = lxpxeboot.cern.ch• PXELINTEST = lxpxeboottest.cern.ch

CERN IT Department

CH-1211 Genève 23


it


Alternatives

• AII– Automated Installation Infrastructure– Set of Quattor components– Supports PXE, Kickstart and Jumpstart

• Cobbler & Koan– From RedHat– Cobbler is a provissioning server. Koan is the

client installer– 'Very' feature rich (Templates, Snippets, WebUI...)– Can manage a lot (TFTP, DHCP, DNS, REPOS...)– Supports Xen, KVM and WMWare installations

CERN IT Department

CH-1211 Genève 23


it


Introduction to aims2

• A rewrite of AIMS code base making it simpler and more flexible

• Perl SOAP Client and Server• Server-side uses a modular approach• Database support provided by Oracle• Improved integration with the CERN

environment (LanDB, LDAP, e-groups, CDB)• Improved

– Customisation of a users installation– User authentication and authorisation

CERN IT Department

CH-1211 Genève 23


it


Authentication

• Identifying who the user is and whether they are allowed to use the service.

• Originally provided by a manually maintained .klogin

• Uses Kerberos 5• Defined service in the KDC• So you can get a ticket for aims2• Ticket is presented by the client• No KRB5 credentials for CERN, no access.

CERN IT Department

CH-1211 Genève 23


it


Authorisation

• Do you have permission to install X device?– Try to prevent accidental/malicious installations.

• Previously manually maintained by AFS ACLs on Kickstart directories.– Difficult to transfer ownership of a device.

• In one word - Icky!– Solution is not easy, as I found out.

• No global one-stop source of device ownership at CERN.

• Need to use multiple sources.

CERN IT Department

CH-1211 Genève 23


it

How we try to achieve this

• Is the USER listed as the OWNER or MAINUSER of the DEVICE?

• If the DEVICE is known to CDB, is the USER listed as having root permissions on the DEVICE? (CDB ACL, type=root)

• Is the USER a member of Linux Support? (They might be helping you out)

• If the device is located within Building 513 or 613, is the USER an FIO sysadmin?

• Is the owner or main user something we've been told about?

– We can map shared accounts we know about to e-groups. For example, we can map the service account "[email protected]" to "fs_installers“.

• Explicit deny at the end.

• Still not perfect, but following user feedback it is a lot better.

• Code used is very flexible so if a new source becomes available, it can be easily accommodated.


CERN IT Department

CH-1211 Genève 23


it


Kickstart Handling

• No more enforced use of AFS directories• New commands added: showks, updateks• Upload or link to your kickstart file.

– Linked Kickstart files can be easy re-sync'd if changed

– Link sources permitted include AFS and http (not https yet)

• Kickstarts rendered to Anaconda on the fly if– Correct device– Device is enabled.

• Balance between hiding the Kickstart and rendering

CERN IT Department

CH-1211 Genève 23


it


PrepareInstall

• A (large) script used by the Quattor/Elfms community

• Uses device information in CDB to generate Kickstart file for device.

• Also deals with SINDES• Only small amount of worked was needed

– Uploading of Kickstart rather than writing to AFS– Catching errors thrown from AIMS– Modification of PXE boot targets, with append

options.

• Many thanks to Jan VE for his help.

CERN IT Department

CH-1211 Genève 23


it


Kernel append options

• Previously maintained templates– bootif, eth0, eth1, with(out) serial console....

• Now allows arbitrary options to be provided with the –kopts option

• If you provide ks, it will override aims' ks– No option uses DHCP

• NEXT_SERVER/ip-address.ks

• Can now very easily deal with “icky” hardware

• Smthg new: allowwireless, essid=<essid>...

CERN IT Department

CH-1211 Genève 23


it


Arbitrary .img management

• Smart users can test and deploy their own kernels and/or intitrd.img's

• Use case within FIO– u-boot, burn-in-tests, fireware updates, hardware

utilities– SLCx– Other operating systems (unsupported)

• Client commands– addimage– showimage– remimage

• Limited right now to AFS sources

CERN IT Department

CH-1211 Genève 23


it


Master-Slave configuration

• Master-Slave configuration manually maintained in server configuration

• Should master fall over, nothing works.

CERN IT Department

CH-1211 Genève 23


it


Server independence model

CERN IT Department

CH-1211 Genève 23


it


The Role of the Database

• aims2 is Oracle driven– Used for holding device PXE states and PXE boot

media for deployment

• Centralised logging and server configurations• Service reliability

– If a server is lost, the important stuff is safe.

• Server states maintained with database daemon– Maintains connection to database, sleeping and

waking during downtime.– Polls for changes, maintain directories

CERN IT Department

CH-1211 Genève 23


it


Audit trail and Traceability

• Log “every” action– Tracer bullets everywhere

• Logging is centralised– Log the “who, what, when, where and how”– Provide information on each important step of the

installation• When the client booted• When the client pulled it's kickstart• When the client got to %POST (pxeoff)

• Helps the user understand their installation and Linux Support can follow – Still some work needed on training Linux Support

CERN IT Department

CH-1211 Genève 23


it


Improvements for the future

• Fix bugs, of course• Improve the use of Oracle

– More processing “inside” Oracle, but not everything

– Improve LOB storage. Right now Oracle is greedy

• New commands– updateimage– downloadimage– Showhistory

• As time goes on, user's needs will change

CERN IT Department

CH-1211 Genève 23


it


Evaluation

• Writing software is difficult• aims2 tries to provide a more flexible system

that can be easily adapted as the environment changes

• Benefits should be felt the users first– The installation process is improved

• and also felt by Linux Support– Improved debugging tools– Again, the benefits filter back to the user.

CERN IT Department

CH-1211 Genève 23


it


Thank You and Questions

• ALL the users of AIMS– Fantastic feedback (dev and testing)– Patient and willing to talk about new ideas (and

accept bumps on the way)– Thanks!

• Jarek as my Supervisor– LA for its support

• Jan VE for help with PrepareInstall• Many others too!• Over to you now, Question Time.

cern it department ch-1211 genève 23 switzerland t aims2 automated installation management system

Documents