cit 470: advanced network and system administrationwaldenj/classes/2011/summer/...system policy....

1

CIT 470: Advanced Network and System Administration Slide #1

CIT 470: Advanced Network and

System Administration

Administration Fundamentals


Topics

1. The Nature of System Administration

2. Organizations and Certifications

3. Change Management

4. Remote Administration

What is System Administration?

2

What is a system?

System: An organized collection of computers

interacting with a group of users.


Servers

Services

PCs

Users

Network

run o

n

run o

n

help to accomplish work

System State

System policy: specification of a system’s

configuration and its acceptable usage.

System state S(t): the current configuration (files,

kernel, memory or CPU usage) of a system.

Ideal states S*(t): states of the system that match the

system policy. Over time, the system state shifts

away from the ideal state.

System administration: modifying the system to

bring it closer to S*(t).



What do sysadmins do?

Small org: sysadmin can be entire IT staff

– Phone support

– Order and install software and hardware

– Fix anything that breaks from phones to servers

– Develop software

Large org: sysadmin is one of many IT staff

– Specialists instead of “jack of all trades”

– Database admin, Network admin, Fileserver admin, Help

desk worker, Programmers, Logistics

3


Common Activities

1. Add and remove users.

2. Add and remove hardware.

3. Perform backups.

4. Install new software systems.

5. Troubleshooting.

6. System monitoring.

7. Auditing security.

8. Help users.

9. Communicate.


User Management

Creating user accounts

– Consistency requires automation

– Startup (dot) files

Namespace management

– Usernames and UIDs

– Multiple namespaces or SSI?

Removing user accounts

– Consistency requires automation

– Many accounts across different systems


Hardware Management

Adding and removing hardware

– Configuration, cabling, etc.

Purchase

– Evaluate and purchase servers + other hardware

Capacity planning

– How many servers? How much bandwidth, storage?

Data Center management

– Power, racks, environment (cooling, fire alarm)

Virtualization

– When can virtual servers be used vs. physical?

4


Backups

Backup strategy and policies

– Scheduling: when and how often?

– Capacity planning

– Location: on-site vs. off-site.

Monitoring backups

– Checking logs

– Verifying media

Performing restores when requested


Software Installation

Automated consistent OS installs

– Desktop vs. server OS image needs.

Installation of software

– Purchase, find, or build custom software.

Managing software installations

– Distributing software to multiple hosts.

– Managing multiple versions of a software pkg.

Patching and updating software


Troubleshooting

Problem identification – By user notification

– By log files or monitoring programs

Tracking and visibility – Ensure users know you’re working on problem

– Provide an ETA if possible

Finding the root cause of problems – Provide temporary solution if necessary

– Solve the root problem to permanently eliminate

5


System Monitoring

Automatically monitor systems for

– Problems (disk full, error logs, security)

– Performance (CPU, mem, disk, network)

Provides data for capacity planning

– Determine need for resources

– Establish case to bring to management


Helping Users

Request tracking system

– Ensures that you don’t forget problems.

– Ensures users know you’re working on their

problem; reduces interruptions, status queries.

– Lets management know what you’ve done.

User documentation and training

– Policies and procedures

Schedule and communicate downtimes

Communicate

Customers

– Keep customer appraised of process.

• When you’ve started working on a request with ETA.

• When you make progress, need feedback.

• When you’re finished.

– Communicate system status.

• Uptime, scheduled downtimes, failures.

– Meet regularly with customer managers.

Managers

– Meet regularly with your manager.

– Write weekly status reports.


6


Specialized Skills Heterogeneous Environments

Integrating multiple-OSes, hardware types, or network protocols, distributed sites.

Databases SQL RDMS

Networking Complex routing, high speed networks, voice.

Security Firewalls, authentication, NIDS, cryptography.

Storage NAS, SANs, cloud storage.

Virtualization and Cloud Computing VMware, cloud architectures.

Slide #17

Qualities of a Successful Sysadmin

Customer oriented

– Ability to deal with interrupts, time pressure

– Communication skills

– Service provider, not system police

Technical knowledge

– Hardware, network, and software knowledge

– Proficiency with at least one scripting language

– Debugging and troubleshooting skills

Time management

– Automate everything possible.

– Ability to prioritize tasks: urgency and importance.


First Steps to Better SA

Use a request system.

– Customers know what you’re doing.

– You know what you’re doing.

Manage quick requests right

– Handle emergencies quickly.

– Use request system to avoid interruptions.

Policies

– How do people get help?

– What is the scope of responsibility for SA team?

– What is our definition of emergency?

Start every host in a known state.

7


Principles of SA

Simplicity – Choose the simplest solution that solves the entire problem.

– Work towards a predictable system.

Clarity – Choose a straightforward solution that’s easy to change, maintain,

debug, and explain to other SAs.

Generality – Choose reusable solutions that scale up; use open protocols.

Automation – Use software to replace human effort.

Communication – Be sure that you’re solving the right problems and that people know

what you’re doing.

Basics First – Solve basic infrastructure problems before advanced ones.

Organizations and Certifications


Organizations

USENIX: Advanced Computing Systems

Association

LISA: Large Installation System

Administration

SAGE: System Administration Guild

LOPSA: League of Professional System

Administrators

8


Types of Sites

Small

2-10 computers, 1 OS, 2-20 users.

Small staff size requires outsourcing to obtain most

specialized skills.

Midsized

11-100 computers, 1-3 OSes, 21-100 users.

Large

100+ computers, multiples OSes, 100+ users

Outsources to reduce costs, some specializations.


Certifications

• CCNA, CCNP, CCIE (Cisco)

• cSAGE (SAGE)

• MCSA (Microsoft)

• RHCE (Red Hat)

• SCSA (Sun)

• VCP (VMware)


SAGE Job Descriptions

Novice OS familiarity, help desk skills

Junior Can use OS system administration tools (370)

Intermediate Understanding of distributed computing, common servers,

automate small tasks, independent action

Senior Understanding of scaling issues, including capacity

planning, solve problems by addressing root cause, higher level programming abilities, write proposals for purchasing, data center planning, etc.

9

SA Maturity Model (SAMM)

1. Ad Hoc

Ad-hoc non-repeatable solutions, firefighting.

2. Repeatable

Some repeatable processes.

3. Defined

Documented standard processes

4. Managed

Process effectiveness measured, adapted.

5. Optimized CIT 470: Advanced Network and System Administration Slide #25

Maturity and Complexity


Constant firefighting, high

downtime

Complexity: increasing numbers of systems and/or services

Matu

rit

y

Scalable but time

lost in process.

Low downtime, high

efficiency

Works, but hard

to scale up.

Tool Maturity Levels

1. Ad Hoc

OS GUI, CLI, or web administration interfaces.

2. Repeatable

Version control (RCS, SVN, GIT), request tracker

3. Defined

Automatic monitoring (Nagios, monit, god)

4. Managed

Configuration management (AutomateIt, cfengine)

5. Optimized


10

SAGE Code of Ethics

• Professionalism

• Personal Integrity

• Privacy

• Laws and Policies

• Communication

• System Integrity

• Education

• Social Responsibility


http://www.sage.org/ethics/

Terry Childs Case

Network administrator for San Francisco

– CCIE who built city’s FiberWAN network

Terry was only person with router passwords

– IT department acknowledges knowing that

– He was on-call 24x7x365 to resolve issues

Terry refused to give passwords to boss

– Cited fears that they would be misused by

management, outside contractors.

What was the right thing for Terry to do?


Change Management

11


Change Management

Effective planning and implementation of

changes to systems.

Changes should be

1. Well documented.

2. Have a backout plan.

3. Reproducible.


Why do we need Change Management?

March 26-29, 2006: BART trains halted to

avoid running into each other when

computer systems crashed.

• Crashes on Monday/Tuesday resulted from

software maintenance upgrades.

• Crash on Wednesday resulted from installing a

backup system to avoid future crashes.

• Thousands of passengers stranded for several

hours each time.


Change Management

1. Plan change.

2. Test change on single system.

3. Test change on multiple systems.

4. File a change request.

5. Change committee approves request.

6. Schedule change.

7. Communication with users/admins.

8. Change systems at scheduled time.

9. Post-event analysis.

12


Testing Changes

• Automated checks.

– Sanity checks like Samba testparm.

– Reboot system.

• Test on one system first.

• Then test on set of systems.

– Dedicated test systems.

– System admin workstations.

– Virtual machines.


When do you need a Change Proposal?

Does the change impact critical services?

Critical machines/services

– Business critical: e-commerce server, etc.

– Essential services: routers, DNS, NFS, auth.

Non-critical machines/services

– Individual desktops

– Internal news web server


Change Proposal

1. Description of the change.

2. Systems impacted by change.

3. Why the change is being made.

4. Risks presented by the change.

5. Test procedure.

6. Backout plans.

7. How long the change will require.

13


Communication

Communicate change to impacted people.

– What change is being made (nontechnical.)

– Which services will be unavailable.

– When and how long will they be unavailable.

– What actions do they need to task (if any.)

Communication issues

– If you send too many notes, they’ll be ignored.

– Send notices only to those impacted.

– Push critical notices; use pull for non-critical.


Scheduling

Scope When Notification

Type

Routine Single host

or user.

Anytime. Personal.

Major Many hosts

or users.

Off-peak Push.

Sensitive None but

major

impact on

failure.

Off-peak. Pull.


Change Freezes

Time when only minor updates can be done.

– End of quarter or year.

– “Crunch time” for projects.

14


Backing Out

Decide back-out conditions before downtime

– Avoid the “just 5 more minutes” problem.

– Be sure that someone is keeping track of time.

Questions:

– How much time is required for back out?

– When is the latest time you can successfully back out?

– Will backing out this change prevent other changes from

being committed?


Backing Out: How to do it?

Service-level changes

Use revision control system to revert config.

Machine-level changes

Soft cutover: Old service is still running.

Hard cutover: Power up old server or restore from backups.

Snapshots

Snapshot VM before making change.

Revert to snapshot if need to backout.

Issues

Data migration and compatibility


Automatic Checks

Check integrity of critical files before use.

– Some services provide checks: LDAP, SMB.

– Check startup files by rebooting machine.

– Write your own checks for other files.

• Most people only do this after they have a problem.

15

Remote Administration

1. Network Access

2. SSH

3. Key-based Authentication

4. Console Access

5. X-Windows

6. VNC and NX

7. SSH tunneling


Network Access

Most tasks can be done from the shell.

File management.

Disk/volume management.

Troubleshooting and viewing logs.

Installing/removing software.

Start/stop network services.

Reboot/shutdown.

All we need is a way to invoke a shell across

the network.


telnet

Ubiquitous network terminal protocol

telnet hostname

Similar protocols

rlogin –l user hostname

rsh –l user hostname command

Insecure

Data, including passwords, sent in the clear.

rlogin/rsh use ~/.rhosts for access w/o

passwords.

16


ssh

Secure Shell

Replaces

telnet

ftp

rlogin

rsh

rcp


SSH Security Features


SSH: Protocols and Products

• OpenSSH

• SSH Tectia

• F-secure SSH

• Putty

• WinSCP

• SSH v1

– Insecure, obsolete.

– Do not use.

• SSH v2

– Current version.

17


SSH Features

Secure login ssh –l user host

Secure remote command execution

ssh –l user host command

Secure file transfer

sftp –l user host

scp file user@host:/tmp/myfile

Port forwarding

ssh –L 110:localhost:110 mailhost


The Problem of Passwords

1. Good passwords are hard to remember.

2. Password transferred to remote system.

3. Automating remote access with passwords

is difficult.


Public Key Cryptography

Two keys

– Private key known only to owner.

– Public key available to anyone.

Applications

– Confidentiality:

• Sender enciphers using recipient’s public key,

• Receiver deciphers using their private key.

– Integrity/authentication:

• Sender enciphers using own private key,

• Recipient deciphers using sender’s public key.

18


Key-based Authentication

SSH uses public-key authentication

Private key stored in your machine.

Public key stored on remote machines.

Public-key login protocol

1. Client sends server a login request.

2. Server issues a challenge.

3. Client responds with computation based on

challenge and private key.

4. Server checks response with public key.


Using key-based authentication

1. Generate a public/private key pair.

ssh-keygen

Encrypted key files: id_rsa, id_rsa.pub

2. Copy public key to remote host

Copy to ~/.ssh/authorized_keys.

3. Login to remote host

ssh –l user remote


Keys are more secure than Passwords

1. Need to have two items to login: key file

and passphrase.

2. Neither key nor passphrase is sent to

remote host.

3. Machine-generated cryptographic keys are

infeasible to guess, unlike passwords.

19


SSH Agent

Problem: you have to enter passphrase to decrypt the key each time you use ssh.

Solution: SSH Agent

> ssh-agent $SHELL

> ssh-add Enter passphrase for

/home/jw/.ssh/id_dsa: ********

Identity added:

/home/you/.ssh/id_dsa

(/home/jw/.ssh/id_dsa)

> ssh –l jw host


SSH Agent Features

Agent support for entire session.

Start ssh-agent on initial shell.

X: ~/.xsession (Often enabled by default.)

Multiple keys

ssh-add keyfile

ssh-add –l

Remove keys

ssh-add –d keyfile

ssh-add –D


Remote Access when Server is Down

Problem: No network access to host.

Solutions:

– Go to computer room and bring host up.

– Specialized hardware (network boot / power).

– Virtual machines.

– Console servers.

20


Console Servers

Console

– Main I/O device for computer.

– Historically: serial terminal.

– Typically: keyboard/mouse/screen.

Server allows access to multiple consoles.

– Console access: BIOS, Bootloader, Kernel

– Eliminates need for keyboards, mice, monitors.

– Serial line to each machine from server.

– One user has R/W, other users have R access.


Console Hardware

Console servers solution

– Commercial: Cisco, Cyclades, Xyplex

– Open source: Conserver + serial expander card

Hardware issues

– Connectors: DB-9, DB-25, RJ-45

– Encoding: 8N1, 7E1

– Speeds: 9600 – 230k


X-Windows

Server

– Handles user input and

graphical display.

– Runs on the machine

with display unit.

Clients (applications)

– Can run on a different

machine than server.

• Set DISPLAY env var.

• Use –display option.

http://en.wikipedia.org/wiki/File:Terminal-dec-vt100.jpg

http://en.wikipedia.org/wiki/File:Network_Computing_Devices_NCD-88k_X_terminal.jpg

http://upload.wikimedia.org/wikipedia/commons/b/b5/X_client_sever_example.svg

21


Window Manager

X client that provides features like:

– Move, resize, iconify, and kill windows.

– Window title bars.

– Popup menus.

Example window managers

– twm: Tab, primitive early window manager

– mwm: Motif, found on commercial UNIXes

– fvwm: Free, fast, very customizable.

– WindowMaker: NeXT-like, see also AfterStep.


TWM Screenshot


FVWM Screenshot

22


WindowMaker


Desktops

CDE

Common desktop env for commercial UNIXes.

Gnome

Standard Linux desktop based on GTK+.

KDE

Windows-like free desktop based on QT.

Xfce

Lightweight desktop, also based on GTK+.


VNC: Virtual Network Computing

23


Why VNC?

1. Remote desktop access.

2. Helpdesk: control a remote desktop.

3. Persistent desktop.

4. Use same desktop from multiple clients.

5. Need Linux access from Windows.

6. Need Windows access from Linux.


What is VNC?

• Open remote desktop protocol.

• Many implementations

– RealVNC: VNC from original researchers.

– TightVNC: VNC with high compression.

– VNCj: Java VNC, can run within web browser.

– PalmVNC: VNC for Palm Pilots.

– UltraVNC: enhanced VNC, only for Windows.


Using VNC

1. Start VNC server

UNIX: vncserver

Win: Start menu>Programs>RealVNC>VNCServer

2. Write down server name and display number.

It will look something like unix3:1

3. Start VNC client

UNIX: vncviewer

Win: Start menu>Programs>RealVNC>VNCViewer

4. Enter server and display to connect to (from step 2).

5. A VNC remote desktop should appear.

24


Configuring and Troubleshooting

• On UNIX, VNC stores files under ~/.vnc

• Configuration: xstartup

– Indicates which X clients to start with server.

– Typically includes vncconfig application.

• Configuration: passwd

– Contains VNC server session password.

• Log files: host:display#.log

– Any errors should appear in these logs.


Securing VNC

VNC does not provide encryption.

Use ssh tunneling to encrypt login + data: ssh –L 5901:remotehost:5901 remotehost

vncviewer localhost:1


Tunneling

Tunneling: Encapsulation of one network

protocol in another protocol

– Carrier Protocol: protocol used by network

through which the information is travelling

– Encapsulating Protocol: protocol (GRE, IPsec,

L2TP) that is wrapped around original data

– Passenger Protocol: protocol that carries original

data

25


ssh Tunneling

SSH can tunnel TCP connections

– Carrier Protocol: IP

– Encapsulating Protocol: ssh

– Passenger Protocol: TCP on a specific port

POP-3 forwarding ssh -L 110:pop3host:110 -l user pop3host

– Uses ssh to login to pop3host as user

– Creates tunnel from port 110 (leftmost port #) on localhost to port 110 (rightmost post #)of pop3host

– User configures mail client to use localhost as POP3 server, then proceeds as normal


NX

Advantages over VNC:

Speed: fast enough to use over dialup.

Built-in ssh encryption.

Disadvantages

Immature code; hard to install + set up.

GPL client/server for Linux only.

Free Windows client; commercial server.

Key Points

Principles of System Administration

– Simplicity

– Clarity

– Generality

– Automation

– Communication

– Basics First

Changes should be

– Documented

– Reproducible (usually automated)

– Have a backout plan (use VM snapshots and version control)

– Approved with a formal process

Remote Administration includes

– ssh (default)

– console (for when network is down)

– remote desktop (rarely needed)

Tunneling via ssh can secure insecure protocols.


26


References 1. Daniel J. Barrett, Robert G. Byrnes, Richard E. Silverman, SSH, The Secure Shell, 2nd edition, O’Reilly, 2005.

2. Mark Burgess, Principles of Network and System Administration, 2nd edition, Wiley, 2004.

3. Conserver, http://www.conserver.com/

4. John Fisher, “Secure X Windows,” CIAC 2316, http://www.ciac.org/ciac/documents/ciac2316.html, 1995.

5. Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly, 2002.

6. David K.Z. Harris, “Zonker’s Greater Scroll of Console Knowledge,” http://www.conserver.com/consoles/, 2005.

7. Brian Hatch, “SSH Host Key Protection,” http://www.securityfocus.com/infocus/1806, 2004.

8. Thomas A. Limoncelli and Christine Hogan, The Practice of System and Network Administration, Addison-Wesley, 2002.

9. No Machine NX, http://www.nomachine.com/

10. OpenSSH, http://www.openssh.com/

11. Real VNC, http://www.realvnc.com/

12. Daniel Robbins, “OpenSSH key management,” http://www-128.ibm.com/developerworks/library/l-keyc.html, 2001.

13. SAGE, Job Descriptions, http://www.sage.org/field/jobs-descriptions.html.

14. SAGE, SAGE Code of Ethics, http://www.sage.org/ethics/.

15. Carla Schroeder, Linux Cookbook, O’Reilly, 2004.

16. Carla Schroeder, “FreeNX ups the Remote Linux Desktop Ante,” Enterprise Networking Planet, http://www.enterprisenetworkingplanet.com/netos/print.php/3508951, 2005.

17. Glen Turner, “Linux Remote Serial Console HOWTO,” http://www.tldp.org/HOWTO/Remote-Serial-Console-HOWTO/index.html, 2003.

18. Paul Venezia, Why San Francisco's network admin went rogue, http://www.infoworld.com/d/adventures-in-it/why-san-franciscos-network-admin-went-rogue-286?page=0,1, July 18, 2008.

19. Webmin, http://www.webmin.com/

20. Window Managers for X, http://xwinman.org/

21. Todd R. Weiss, “IT upgrades slow BART trains in San Francisco,” http://www.computerworld.com/printthis/2006/0,4814,110107,00.html, ComputerWorld, March 31, 2006.

http://www.conserver.com/

http://www.ciac.org/ciac/documents/ciac2316.html

http://www.conserver.com/consoles/

http://www.securityfocus.com/infocus/1806

http://www.openssh.com/

http://www.realvnc.com/

http://www-128.ibm.com/developerworks/library/l-keyc.html





http://www.sage.org/ethics/

http://www.enterprisenetworkingplanet.com/netos/print.php/3508951

http://www.tldp.org/HOWTO/Remote-Serial-Console-HOWTO/index.html







http://www.infoworld.com/d/adventures-in-it/why-san-franciscos-network-admin-went-rogue-286?page=0,1



















http://www.webmin.com/

http://www.computerworld.com/printthis/2006/0,4814,110107,00.html

cit 470: advanced network and system administrationwaldenj/classes/2011/summer/...system policy....

Documents