Server management and security

September 10, 2002

Ko, YangWoo

yw@mrko.pe.kr

ccTLD name server training

Note

• Contents are NOT mine. Most of them are from the wonderful book “Practical Unix and Internet Security” and “Real World Linux Security”.

• Others are extracted from various good resources including;– Linux Security FAQ– Solaris Security FAQ– Sun Solaris / HP-UX / Tru64 Unix man pages

Table of contents

• Before we start

• Security basics

• Unix / Linux sever security

• System setup guide

• Detection

• Recovery

Module 1 : Before we start

Welcome to wild Internet !

• Quote from Crypto-Gram (June 15, 2001 )

A random computer on the Internet is scanned dozens of times a day. The life expectancy of a default installation of Red Hat 6.2 server, or the time before someone successfully hacks it, is less than 72 hours. A common home user setup,with Windows 98 and file sharing enabled, was hacked five times in four days. Systems are subjected to NetBIOS scans an average of 17 times a day. And the fastest time for a server being hacked: 15 minutes after plugging it into the network.

No system is ever perfectly secure.

But, still we need security.

• Any number of toolkits exist that allow total amateurs to become holy terrors.

• The good news is that if you can beat the popular intrusion toolkits, 90 percent of the bad guys will go bother somebody else who's less secure.

System security in a page

• The Seven Most Deadly Sins– Weak Passwords– Open Network Ports– Old Software Version– Poor Physical Security– Insecure CGIs– Stale and Unnecessary Accounts– Procrastination

Module 2 : Security basics

Security requirements• Confidentiality• Integrity• Authentication• Non-repudiation• Availability• Access control• Combined

– User authentication used for access control– Non-repudiation combined with authentication

Some terminologies

• System security / network security

• Passive attack / active attack– sniffing / spoofing

• Two models– Access control

• discretionary access control vs. mandatory access control

– Audit

Security policy• Simple and generic policy for system which users

can readily understand and follow.• Starting point :

– That which is not permitted is prohibited.

• Setup steps(1) Identify what you are trying to protect.(2) Determine what you are trying to protect it from.(3) Determine how likely the threats are.(4) Implement measures which will protect your assets in

a cost-effective manner.(5) Review & improve the process continuously

Security policy (continued)

• References– rfc2196 : Site Security Handbook

• Samples– ftp://coast.cs.purdue.edu/pub/doc/policy

Module 3 : Unix / Linux server security

• Password• Superuser• File system• Account• Integrity• Log and Audit• Programmed threats• TCP/IP

Module 3-1 : Password

Bad passwords• Your name, spouse’s name, partner’s name, pet’s na

me, child’s name, friends’ name, boss’s name• Operating system, hostname, username• Phone number, license plate number, birth date, soci

al security number• Words in the dictionary• Simple patterns of letters on the keyboard (qwerty)• Passwords of all the same letter• Any of above spelled backwards• Any of above followed or prepended by a single digit

Password

Good passwords

• Have both uppercase and lowercase letters.• Have digits and/or punctuation characters as

well as letters.• May include some control characters and/or

spaces.• Are easy to remember, so they do not have to

be written down.• Are seven or eight characters long.

Password

The Thompson Test• Devised by Ken Thompson• Cracking algorithm

– One to six ASCII characters– Seven or eight lowercase letters– Any word from a large dictionary such as

hangman-words, or a word spelled backward or with the digit “1” instead of the letter “l”, with the digit “0” instead of the letter “o”, or with the digit “3” instead of the letter “e”.

– Any pair of words from a large dictionary or words spelled backwards.

Password

Module 3-2 : Superuser

Who is superuser ?

• UID of 0

• Any username can be the superuser.

• Normal security checks and constraints are ignored for the superuser.

• Superuser is not for casual use.– Do not login as superuser, use ‘/bin/su’ wit

h “-” option instead.

Superuser

Simple trap to steal superuser• Premise

– Root’s PATH starts with “.”

• Contents of shell script ‘ls’#!/bin/shcp /bin/sh ./junk/.sschmod 4555 ./junk/.ssrm –f $0exec /bin/ls ${1+”$@”}

• Set a trap% cd

% chmod 700 .

% touch ./-f

• To do is just say to administrator. “I have a funny file in my directory I can’t seem to delete.”

Superuser

Several tricks for superusers• Test complex commands in a non-destructive way

before running it.– rm foo*.bar “after” echo foo*.bar

• alias rm=‘rm –i’• Only become root to do single specific task. Stay n

ormal user shell until you are sure what needs to be done by root.

• Command path– Minimum and trusted directories only– Never include “.”– No writable directories

Superuser

Several tricks for superusers (continued)

• Never use r-utilities (e.g. rlogin, rsh). Never create .rhosts for for the root.

• No login from the remote– Linux, HPUX : /etc/securetty

• file which lists ttys from which root can log in

– Solaris : /etc/default/login• CONSOLE=/dev/console

• Always be slow and deliberate running as root. Think before you type.

Superuser

Module 3-3 : File system

-rwxr--r--

File permission

File type- : plain filed : directoryc : character device (tty, printer)b : block device (disk, CD-ROM)l : symbolic links : socket=, p : FIFO

Access granted to ownerr : read / w : write / x : execute

Access granted togroup member

Access granted toothers

File system

SUID/SGID/sticky bits• SUID (set uid)

– Processes are granted access to system resources based on user who owns the file.

• SGID (set gid)– (For file) Same with SUID except group is affected.– (For directory) Files created in that directory will have th

eir group set to the directory's group.

• sticky bit– If set on a directory, then a user may only delete files th

at the he owns or for which he has explicit write permission granted, even when he has write access to the directory. (e.g. /tmp )

File system

File system tips

• Finding SUID and SGID Files# find / \( -local -o -prune \) \( -perm -004000 -o -per

m -002000 \) -type f -print

( xdev can be used in place of local/prune)

• Files without associated owner/group can be a signal of compromise.# find / -nouser –o –nogroup –print

• Users are not allowed to have .rhosts file.# find /home –name .rhosts -print

File system

File system tips (continued)

• Turning off SUID / SGID in mounted file system– use nosuid (and nodev if possible) when m

ounting remote file system or allowing users to mount floppies or CD-ROMs

• Device file can be created as a backdoor after compromise.# find / \( -local -o -prune \) \( -type c -o -type

b \) -exec ls -l {} \;

File system

Critical system files• These files should be backed up and compared with

saved version frequently.– /etc/passwd, /etc/shadow, /etc/group– /etc/rc*– /etc/ttys, /etc/ttytab, /etc/inittab– /usr/lib/crontab, /usr/spool/cron/crontabs/, /etc/crontab– /usr/lib/aliases– /etc/exports, /etc/dfs/dfstab– /etc/netgroups– /etc/fstab, /etc/vfstab– /etc/inetd.conf– UUCP related files

File system

Module 3-4 : Account

Dangerous accounts• Accounts without passwords

# cat /etc/passwd | awk -F: 'length($2)<1 {print $1}'

• Default accounts– Just remove them !

• Shared accounts– Less accountability, less security.– Create several accounts in a group.

• e-mail ID and account– Do not use e-mail ID as an account, utilized alias f

eature instead.

Account

Dormant account• Risks

– Intruder can use dormant account without being noticed.– Owner of dormant account cannot follow your policy or

order. (e.g. Dear every users, please change your passwords right now.)

• How to handle– Disabling dormant account automatically (SVR4)

• usermod –f 10 newcat (locked if no login in 10 days)

– Freeze it• Put “*” in password field• chmod 0 /home/newcat• find / -user newcat -ls

Account

Dormant account (continued)• How to find

#!/bin/shPATH=/bin:/usr/bin;export PATHumask 077THIS_MONTH=`date | awk ‘{print $2}’`/bin/last | /bin/grep $THIS_MONTH | awk ‘{print $1}’ | /

bin/sort –u > /tmp/users1$$cat-passwd | /bin/awk –F: ‘{print $1}’ | /bin/sort –u /tmp

/users2$$/bin/comm –13 /tmp/users[12]$$/bin/rm –f /tmp/users[12]$$

Account

Module 3-5 : Integrity

Simple examples

• By metadata# cat /usr/adm/filelist | xargs ls -ilds > /tmp/no

w

# diff -b /usr/adm/savelist /tmp/now

• By checksum# find `cat /usr/adm/filelist` -ls -type f -exec m

d4 {}\; > /tmp/now

# diff -b /usr/adm/savelist /tmp/now

Integrity

Tripwire• Tripwire is a tool that checks to see what has

changed on your system. The program monitors key attributes of files that should not change, including binary signature, size, expected change of size, etc.

• Where is it ?– Commercial version : http://www.tripwire.com/– For Linux user : http://www.tripwire.org/– For Unix user :

ftp://coast.cs.purdue.edu/pub/COAST/Tripwire/tripwire-1.2.tar.Z

Integrity

Tripwire tutorial in a slide• Initial setup

– download / build / install it– modify policy file (e.g. remove unnecessary files)

# vi /etc/tripwire/twpol.txt

– generate policy file# twadmin –create-polfile /etc/tripwire/twpol.txt

– build initial database# tripwire –init

• check periodically# tripwire –check– reconcile differences (e.g. software installation)

# tripwire –update –accept-all –twrfile report_file

Integrity

Module 3-6 : Log and audit

Basics• Consider remote logging to secure log data.• List of log files

– acct / pacct : Commands run by users– aculog : Dial-out modem (acu : automatic call unit)– lastlog : Most recent login success/fail times– loginlog : Bad login attempts– messages : Console / syslog facility– sulog : su command– utmp / utmpx : Each user currently logged in– wtmp / wtmpx : Login/out, shutdown/startup– xferlog : FTP access

Log and audit

Files and commands• lastlog file

– lastlog (Linux only)• Displays last login time and location.

• u/wtmp file– last

• Displays login and logout information about users and terminals

• acct/pacct file– (Solaris 5.8) /usr/lib/acct/[startup , shutacct]

• Starts or stop accounting.

– (Solaris 5.8) acctcom, lastcom• Displays the recent commands executed.

Log and audit

Monitoring logs

• logcheck (logsentry)– Extracts anything that might indicate a sec

urity violation or other abnormality, and informs via e-mail.

– http://www.psionic.com/products/logsentry.html

Log and audit

Module 3-7 : Programmed threats

Basic terms

• Bug vs. malware (or malicious software)• Kinds of malwares

– Security tools and toolkits – Back doors and trap doors – Logic bombs – Viruses – Worms – Trojan horses – Bacteria and rabbits

Programmed threats

Against programmed threats

• Back door– Do regular integrity check.– Install software only from well-known

sources.– Separate test bed and production system.

• Trojan horse– Never execute anything until you’re sure of

program or inputs to program.– Never run anything as root unless you

absolutely must.

Programmed threats

Against programmed threats (continued)

• Viruses– Use same techniques used against back do

ors and Trojan horse.– Don’t include nonstandard directories (inclu

ding .) in your PATH.– Don’t leave common binary directories unpr

otected and set permission of commands to 555 or 511.

– Make sure your own directories are writable only by you not by your group or world.

Programmed threats

Against programmed threats (continued)

• Worm– Prevention

• If an intruder can enter your machine, so can a worm program.

– If under attack,• Call computer incident response center to se if

other sites have made similar reports.• Isolate your server to prevent spread.

Programmed threats

Module 3-8 : TCP/IP

Vulnerabilities

• ftp– Passwords are sent in plain text.– /etc/ftpusers

• List of accounts that are NOT allowed to use ftp.

• telnet– Passwords are sent in plain text.– Attacker can hijack the session.

TCP/IP

Vulnerabilities (continued)

• smtp (sendmail)– Must be upgraded 8.9.3 or higher. Current

version is 8.12.6.– Check permission of /var/spool/mqueue, se

ndmail.cf, /etc/aliases*, /etc/mail/mailertable* (owned by root, writable by owner only)

TCP/IP

Vulnerabilities (continued)• Sun RPC portmapper

– Assigns the TCP/UDP ports used for RPC.– To improve security, turn it off if possible. Or,

• Replace it with Wietse Venema’s version.• Block packets on port 111.

• rexec, rsh, rlogin– Executes remote program or login.– rexec transmits plain text password and rsh/rlogin

use “trusted host/user” concept.– Disable rexec, and replace rsh/rlogin with ssh.

TCP/IP

Vulnerabilities (continued)

• web– Yet another BIG topic. See references;

• Lincoln D. Stein’s WWW Security FAQ– http://www-genome.wi.mit.edu/WWW/faqs/www-sec

urity-faq.html

• Paul Phillips CGI security FAQ– http://www.primus.com/staff/paulp/cgi-security

• NCSA’s CGI security documentation– http://hoohoo.ncsa.uiuc.edu/cgi/security.html

TCP/IP

Vulnerabilities (continued)• NFS

– Limit exported and mounted file systems– Export read-only and use root ownership– Remove group-write permission for files and director

ies– Do not export server executables and home director

ies– Do not allow users to log into server– Use fsirand and set the portmon variable– Use showmount –e– Use secure NFS

TCP/IP

Vulnerabilities (continued)

• tftp (UDP 69)– No security at all.

• finger ( 79 )– Provides user information.

• POP ( 109, 110 )– Username/password is sent in plain text.

TCP/IP

Module 4 : System setup guide

Useful links for system setup

• Solaris– Solaris/Unix Security Checklist Version 1.0

• http://www.geocities.com/losttoy2000/solarissec.rtf

– The Solaris Security FAQ• http://www.itworld.com/Comp/2377/security-faq

/

• Linux– Securing Debian Manual

• http://www.debian.org/doc/manuals/securing-debian-howto/

System setup steps (1/2)1. Disconnect system from network.2. Install a minimal Operating System.3. Install the recommended patches.4. Use BIOS/EEPROM security.5. Securing root account

– Force root to login through su.– Check environments

• default mask (027), PATH

6. Apply hardening script if available.7. Direct syslog to loghost

System setup steps (2/2)8. Create minimal accounts and disallow login.9. Let minimal services run;

– /etc/rc*, /etc/inet.d

10.Use tcpwrapper for network services.11. Install Secure Shell and encourage its use.12. Install integrity checker (e.g. Tripwire).13.Test it periodically

– e.g. Nessus, COPS, Tiger, …

14.Monitor it forever– Check logs, login/outs, commands

Module 5 : Detection

• Monitoring

• Scanning

• Handling

Monitoring (1/2)

• Log (logcheck)– Propagate it using loghost and e-mail.– Check it.

• Network port (netstat)– Trojan horse may use network ports.– http://www.glocksoft.com/trojan_port.htm

• Network (tcpdump)

Monitoring

Monitoring (2/2)

• Process (ps)– Check suspicious processes, e.g. compiler.– Record typical size of daemons and import

ant programs to detect Trojan horse.

• Load (uptime)

Monitoring

Scanning

• Find suspicious files.

• Run Tripwire.

• Detect promiscuous network interfaces.– (see next page)

Scanning

Perl script to detect sniffer#!/usr/bin/perlmy $ifconfig = “/sbin/ifconfig”;my $recips = admin@my.admin.host;my %PROMISC = ();my $interface = “”;open( IFCONFIG, “$ifconfig|” ) || die( “Error: cannot run ifconfig!” );while( <INCONFIG> ) { $interface = $1 if m/^(\S+)/; $PROMISC{$interface} = 1 if m/promisc/I;}close( IFCONFIG );if( %PROMISC ) { open( MAIL, “|Mail –s ‘Promisc mode’ $recips” ) || die( “Error: cannot send mail” ); print MAIL “Interfaces in Promisc mode: “, join( “ “, sort keys %PRMISC), “\n”; close MAIL;}

Scanning

Handling incidents

• Don’t panic– Is it really a security incident ?– Was any damage really done ?– Evidence or normal operation, that is the

question.

• Document– Write down everything you find, always

noting the date and time.

• Plan ahead !!!

Handling

Module 6 : Recovery

• Regaining control of system

• Finding and repairing the damage

• Tracing attacker

Regaining control of system• Operate as an unprivileged user.• Check integrity of commands used.• Have stealth version of crucial commands (ps / l

s / tar / …)– Build from open source. Or,– Rename from existing binary

• cd /home/larry/bin• cp /bin/ls monthly• cat text_file >> monthly• (echo ls is monthly; md5sum monthly) | lpr

• Process must be kill by –9.

TCP/IPRegaining control of system

Analyze Trojan horse

• Save suspicious executables on (removable) media.

• Analyze– strings Trojan– file Trojan

• if not stripped– nm Trojan (see function names, syscalls)– run debugger (see stack trace)

– Check files opened by Trojan• (Linux) /proc/pid/fd• (Solaris) pfile pid

Regaining control of system

Prevent further damage1. Drop connection (unplug LAN, modem)2. Shutdown abruptly

1. Close database2. Run sync (from non privileged user)3. Press reset (or power) button

3. Boot again1. Remove the system disk from the compromised system a

nd connect it as second disk to a secure system. (Or, boot from secure boot floppy.)

2. Run fsck3. Before coming up multi-user mode, check cracker generat

ed email.

Regaining control of system

Checking logs

• Log files– /var/log/*– Shell history files (esp. for root)– Mailboxes (mbox, /?/spool/mail, /?/spool/mqueue)– Firewall logs, ISP’s log– tcpwrapper log (denied log only)

• Other files– /tmp/*– Hidden directories (e.g. /home/*/.??*)– Other files started with “.”

Finding and repairing the damage

Finding cracker-altered files• Use file integrity tools (e.g. Tripwire)• Compare file system with backups.

– GNU tar “-d” option is very useful.

• Rename any Trojan horse found something obvious.– mv /mnt2/tmp/ls /mnt/tmp/ls-CRACKED– chmod 0 /mnt/tmp/ls-CRACKED

• Find normal files hidden in /dev– find /dev –type f –ls

• Find set UID programs

Finding and repairing the damage

Useful commands

• With IP address (A.B.C.D)– nslookup –type=any D.C.B.A.in-addr.arpa– dig –x A.B.C.D

• With domain name– whois

• Using ping– See the distance

• Using traceroute

Tracing hacker

Module 7 : D.I.Y.

• Requirement

• Analysis

• Plan and Do

What assets do I have ?

• Classification of assets– Hardware

• Server / PC / Storage device / Printer

– Network• Network distribution component (e.g. router, hub, switch)• Network service host (e.g. directory, NMS)• Network connection / Cabling

– Data (e.g. database, agreement, policy, guideline)– Software– Human– Environment (e.g. UPS, air conditioner, cabinet)

Requirement

How valuable they are ? (1/4)

• Review documentations– List of all servers– List of all security products in place– Operation guidelines

• Interview with operational personnel• Valuation methods

– CIA• Confidentiality / Integrity / Availability

– Cost of loss

Requirement

How valuable they are ? (2/4)

• Confidentiality– 5 : Top secret– 4 : Secret– 3 : Limited– 2 : Limited within organization

• Ordinary documents

– 1 : Open

Requirement

How valuable they are ? (3/4)

• Integrity– 5 : Critical damage to operation– 1 : No (or very least) damage to operation

• Availability– 5 : Non stop– 4 : Recovery within 4 hours– 3 : Recovery within 8 hours– 2 : Recovery within 12 hours– 1 : Recovery within 24 hours

Requirement

How valuable they are ? (4/4)

• Cost of loss– 5 : Serious loss (e.g. Bankruptcy)– 4 : Major loss (e.g. Discontinuance of some

businesses)– 3 : Significant loss (e.g. Discontinuance of

some tasks)– 2 : Loss (e.g. < U$ 10,000)– 1 : Trivial loss (e.g. < U$ 1,000)

Requirement

Define analysis areas• Network / system

security– Service daemons– Backdoors, vulnerable

files– Misuse by users– User accounts– Log management– Network configuration– Network device

management– Database security

• Physical security• Security

management– Compliance

assessment– Security policy

assessment– Contingency planning

Requirement

Analysis

• Automated analysis– e.g. Nessus

• Manual analysis– OS checklists

Analysis

Sample results

• Service daemons– Problems

• Some old-version daemons have buffer overflow vulnerabilities.

• Unnecessary daemons are running.

– To do• Remove unnecessary daemons.• Keep necessary daemon up to date.• Run security scanner periodically.

Analysis

Sample results

• Backdoors, vulnerable files– Problems

• Backdoor is not found, but there is no counter measure for future backdoors.

– To do• Install and run Tripwire periodically.

Analysis

Sample results

• Misuse by users– Problems

• Sendmail’s vulnerability can lead to root compromise.

– To do• Remove if unnecessary.• Keep it up to date if necessary.

Analysis

Sample results

• User accounts– Problems

• Super user accounts are shared by administrators and developers.

• Weak passwords are found.

– To do• Define each systems’ usages clearly.• Define each users’ role according to usage of

system.• Apply password control (including aging).

Analysis

Sample results

• Log management– Problems

• No log management.

– To do• Setup a loghost, and all logs are configured to

be sent to it.• Write a log management guideline and apply it.

Analysis

Sample results

• Network configuration– Problems

• Database servers are exposed to Internet.

– To do• Set up a DMZ.• Put external service servers at DMZ.• Put Database servers at internal network

Analysis

Categories of reaction• Configuration issue

– Issues are solved by configuring servers and network equipments properly.

– Usually done within a week.

• Infra structure issue– Issues are solved by investing on infrastructure.– Usually outsourced in long time period.

• Management issue– Several units within organization work together to

handle these issues.– Plan => Do => See cycle

Plan

Categorize To Dos• Configuration issue

– Remove unnecessary daemons.– Apply password control (including aging).

• Management issue– Write a log management guideline and apply it.– Define each systems’ usages clearly.– Define each users’ role according to usage of

system.

• Infra structure issue– Run security scanner periodically.– Set up a DMZ.

Plan