d1 02 carrier slides
TRANSCRIPT
1
An Investigator’s Guide to File System Internals
Brian [email protected]
@stake
? 2002 @stake, Inc. FIRST File System Tutorial2
Overview
• Motivation
• Disk Layout (MBR and Partitions)
• Basic File System Model
• Classical File Systems
– FAT (16-bit and 32-bit)
– UNIX (FFS and EXT2FS)
• Journaling File Systems
– NTFS
– EXT3FS
• Other File Systems
2
@stake
? 2002 @stake, Inc. FIRST File System Tutorial3
Motivation
• Currently, most evidence is found in the file system
• A thorough knowledge of the file systems allow one to:
– Make better use of forensics tools
– Recover deleted content when tools do not exist
– Allow one to describe what your automated tools are doing
• This talk is going to cover basic & advanced topics of file systems for:
– Manual file recovery
– Finding evidence of “wiping” tools
– Knowing how to most efficiently use tools
@stake
? 2002 @stake, Inc. FIRST File System Tutorial4
What is a File System?
• File Systems manage data storage.
• Organized into files, unlike memory.
• Provide an addressing scheme that is easy for humans to understand, unlike memory.
• Examples: FAT, EXT2FS, FFS, NTFS, Reiser FS,
3
@stake
? 2002 @stake, Inc. FIRST File System Tutorial5
Tools
• The @stake Sleuth Kit (TASK) will be used throughout this to show examples on real images.
• The Coroner’s Toolkit (TCT) and TCTUTILs could also be used to collect some of the UNIX file system data.
• TASK is a collection of forensics command line tools for file system analysis
• The tools parse file system structures and do not rely on kernel support
@stake
? 2002 @stake, Inc. FIRST File System Tutorial6
Disk Layout (MBR and Partitions)
4
@stake
? 2002 @stake, Inc. FIRST File System Tutorial7
The Disk
• A disk can be imagined as a long stream of millions bytes
• A disk is typically broken up into one or more groups (or partitions)
• Logical volume management systems do not directly follow this model because they can have several physical disks appear as one volume, which is then broken up into partitions.
• We will cover x86 (DOS) partitions and disk labels
@stake
? 2002 @stake, Inc. FIRST File System Tutorial8
x86: Disk Layout
• The bytes are grouped into sectors, usually 512-bytes each
• Each sector is addressed, using Logical Block Addressing (LBA) or Cylinder / Head / Sector (CHS)
• The first sector contains the Master Boot Record (MBR)and the Partition Table
• The remaining sectors are organized into Partitions, which contain file systems
5
@stake
? 2002 @stake, Inc. FIRST File System Tutorial9
x86: Master Boot Record (MBR)
• Located in first 512-bytes of disk
• The first 446 bytes are x86 assembly instructions that read the partition table to identify a bootable partition
• This is the sector that the systems reads when booting
• The Partition Table starts at offset 446 and contains four 16-byte entries (48 bytes total)
• The final 2-bytes contain a signature (0x55AA)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial10
x86: Partition Table
• Contains 4 entries, each can describe a partition
• Entries are each 16-bytes long
• Entries contain:
– Starting address of partition in CHS and LBA format
– Ending address of partition in CHS format
– Size of partition in sectors
– Flags (is it bootable?)
– Type of Partition
6
@stake
? 2002 @stake, Inc. FIRST File System Tutorial11
x86: Partition Types
• There are more than one partition type because only four entries exist in the MBR Partition Table.
• Primary Partition: Contains 1 file system and can only be described by one the four MBR partition table entries.
• Extended Partition: Contains 0 or 1 Secondary Partitions and 0 or 1 additional Extended Partitions (a linked list). Used when more than 4 partitions are needed.
• Secondary Partition: Contains 1 file system and occupies either all or part of an Extended Partition.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial12
x86: Partition Example
ExtendedSec(FAT)
Secondary(NTFS)
Primary(NTFS)
Primary(EXT2FS)
Primary(EXT2FS)
ExtendedMBR
Disk
7
@stake
? 2002 @stake, Inc. FIRST File System Tutorial13
x86: Partition Tools
• Displaying:
– fdisk
– Partition Dump (pd) <http://heat.ca.sandia.gov/projects/software.html>
– GNU Parted <http://www.gnu.org/software/parted/>
• Finding partitions when the table is gone
– gpart <http://www.stud.uni-hannover.de/user/76201/gpart/>
@stake
? 2002 @stake, Inc. FIRST File System Tutorial14
disk labels: Overview
• BSD systems use disk labels to describe a disk
• The disk label contains the disk layout (partitions), disk geometry, label, model etc. (but no boot code)
• It is located somewhere in the “beginning” of the disk
• There is not a standard format, but the contents include:
– Offset (in sectors or cylinders)
– Size
– Type
– Flags
8
@stake
? 2002 @stake, Inc. FIRST File System Tutorial15
disk labels: x86-based BSD flavors
• x86-based BSD systems (i.e. OpenBSD, FreeBSD) use both x86 partitions and disk labels
• An x86 partition is created as the primary partition for the BSD system (so that it can be booted)
• The disk label is located in the beginning of the BSD primary partition and is used by the kernel to identify where the partitions exist
• The disk label can describe partitions that do not exist in the x86 partition tables
• The labels can be viewed with the ‘disklabel’ command.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial16
Windows XP OpenBSD
a b d ef
•
disk labels: x86-based BSD flavors example
• Only two x86 partitions are defined
• The BSD disk label is located at the beginning of the OpenBSD partition (a)
• The disk label describes 5 partitions (partition f has an offset earlier than it does)
9
@stake
? 2002 @stake, Inc. FIRST File System Tutorial17
disk labels: Sun
• Sector 0 of the disk (slice 2) contains the Virtual Table Of Contents (VTOC) and partition descriptors:
– ID (boot, root, unassigned, swap, usr, backup, stand, var, home etc.)
– Permission Flag (not -mountable, Read Only)
– Starting Sector (or cylinder in some versions)
– Size of partition
• Maximum of 8 or 16 partitions
• Sectors 1-15 contain the bootblock code
• The contents can be listed with:
prtvtoc /dev/rdsk/c0t0d0s2
• The format command can also display the partition details
@stake
? 2002 @stake, Inc. FIRST File System Tutorial18
Basic File System Model
10
@stake
? 2002 @stake, Inc. FIRST File System Tutorial19
Very Basic File System Model
• In general, data within a file system can fall into 1 of 4 categories
• File System Layer
– Descriptive data about the file system (i.e. block sizes, offsets, dirty status)
• Content Layer
– File content (i.e. blocks, clusters). Most data falls into this category.
• Meta Data Layer
– Descriptive data about files (i.e. file size, permissions, time of last access)
• Name Layer
– Human addressing format (i.e. file and directory names)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial20
TASK tool names
• Each tool in TASK is named based on the file system layers:
• File System Layer:
– fsstat
• Content Layer (data):
– dls, dcalc, dstat
• Meta Data Layer (inode):
– ils, istat, icat, ifind
• Name Layer (file name)
– fls, ffind
11
@stake
? 2002 @stake, Inc. FIRST File System Tutorial21
FAT
@stake
? 2002 @stake, Inc. FIRST File System Tutorial22
FAT12, FAT16, and FAT32 File Systems
• Primitive File System – has been used since early DOS versions
• Still used in Windows 9x and on floppies
• Size limits of file system:
– FAT12: 32 megabytes (used in floppy disks)
– FAT16: 4 gigabytes
– FAT32: 8 terabytes
12
@stake
? 2002 @stake, Inc. FIRST File System Tutorial23
FAT: File System Layout
• The file system is organized into 3 areas:
• Boot Sector
– Contains file system details such as sizes and offsets
• Data Area
– Contains file and directory contents. Consecutive sectors in it are grouped into clusters.
• Primary and Backup File Allocation Tables (FAT)
– Contains tables that manage the clusters in the Data Area
@stake
? 2002 @stake, Inc. FIRST File System Tutorial24
FAT: File System Layout Diagram
Boot Sector
File Allocation Tables
Data Area
Cluster
13
@stake
? 2002 @stake, Inc. FIRST File System Tutorial25
FAT: Data Area
• All file and directory content is stored in clusters within the Data Area
• Each cluster is 1 or more (power of 2) consecutive 512-byte sectors. The size of which is saved in the Super Block.
• Each cluster has an address and the first usable one is number 2
• The File Allocation Table (FAT) manages the clusters
@stake
? 2002 @stake, Inc. FIRST File System Tutorial26
FAT: File Allocation Table
• The File Allocation Table (FAT) contains as many entries as there are clusters in the Data Area
• The table is indexed by the cluster number and the contents describe that cluster
• Table entries can be:
– An unallocated marker, used when we need to allocate new clusters
– The address of the next cluster in a file or directory
– End of File (EOF) marker if it is the last cluster in the file or directory
– Bad sector marker, so it is not reallocated
• The size of each table entry depends on the version of FAT (12-bits, 16-bits or 32-bits)
14
@stake
? 2002 @stake, Inc. FIRST File System Tutorial27
FAT: File Allocation Table Example
0 1 2 3 4 8 9
0 0 3 0 8 9 E
• Files are located in Clusters 2, 3, 4, 5 and 7, 8, 9• Cluster 6 is unallocated• E is our End-Of-File marker
4 5 E
5 6 7
@stake
? 2002 @stake, Inc. FIRST File System Tutorial28
FAT: Directory Entries
• File and directory meta data are stored in Directory Entry structures
• One directory entry is allocated for each file and directory
• Directory entries have a static size (32-bytes) and form a list within the clusters allocated by the parent directory
• The clusters for the Root Directory (C:\) are statically located in FAT12 and FAT16 at the first cluster (#2) and are dynamically located in FAT32
• The Root Directory does not have a directory entry (i.e. it has no meta data associated with it)
15
@stake
? 2002 @stake, Inc. FIRST File System Tutorial29
FAT: Directory Entry Structure
• name in ASCII (8 bytes)
• extension in ASCII (3 bytes)
• attribute flags (1 byte) [Directory, Read Only, Hidden, System, Volume ID, Archive, Long Name Entry]
• lowercase flags (1 byte)
• Create time & date (5 bytes) (optional, and is 0 on some Windows9X directories)
• Access date (2 bytes) (no time, but is required by spec)
• Written time & date (4 bytes) (optional)
• Address of the first allocated cluster (4 bytes total)
• Size of file (4 bytes)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial30
FAT: Date and Time structures
• FAT stores the actual time, not the number of seconds from a known date
• Lower 5-bits (0x001f) of time are seconds (0-29) divided by 2
• Next 6-bits (0x07e0) of time are minutes (0-59)
• Upper 5-bits (0xf800) of time are hours (0-23)
• Lower 5-bits (0x001f) of date are day (1-31)
• Next 4-bits (0x01e0) of date are month (1-12)
• Upper 7-bits (0xfe00) of date are year from 1980 (0-127)
16
@stake
? 2002 @stake, Inc. FIRST File System Tutorial31
FAT: Long File Name Directory Entries
• The Directory Entry contains the file name, but is constrained to the 8.3 format.
• Long File Name (LFN) Directory Entries are the same size as normal directory entries, but have a special value in the attribute flag (all bits are set).
• Older versions of DOS happen to ignore directory entries with this flag.
• Files with long names also have a short name directory entry after the LFN directory entries
• Each LFN can hold 13 UNICODE letters, multiple ones exist for long names.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial32
FAT: LFN Structure
• sequence (1 byte) [EOF or linear value]
• 5 UNICODE letters of name (10 bytes)
• attributes flag (1 byte)
• reserved (1 byte)
• checksum (1 byte) (equal for all LFN of a file)
• 6 UNICODE letters of name (12 bytes)
• reserved (2 bytes)
• 2 UNICODE letters of name (4 bytes)
17
@stake
? 2002 @stake, Inc. FIRST File System Tutorial33
FAT: LFN Example
abcdefghijklmnopqrstuvwxyz
1234.txt
abcdef~1.txt
LFN
LFNLFN
FILE
File is: abcdefghijklmnopqrstuvwxyz1234.txt
Name Attribute
NOTES: Stored in “reverse” order and the SFN entry is theonly one with the starting sector and other meta data
@stake
? 2002 @stake, Inc. FIRST File System Tutorial34
FAT: Example
5 6 7 8 9 13 14
6 E E 0 0 14 E9 10 E
10 11 12
file1 - 5
cluster 5
dir1 - 7foo.c - 8
bar.h - 13
Root Dir
FAT
FileContent
FileContent
FileContent
FileContent
cluster 8
cluster 13
cluster 7
cluster 6
18
@stake
? 2002 @stake, Inc. FIRST File System Tutorial35
FAT: File Lookup Example
• Examine contents of c:\dir1\file1.dat
1. Identify location of the Root Directory sectors (from the super block)
2. Read the directory entries in the root directory sectors, looking for one with the name “dir1” and a directory attribute flag
3. When the entry is found, identify the starting cluster (16 for example)
4. Read cluster 16 and process the data as directory entries, looking for the entry with “file1” as the name and “dat” as the extension.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial36
FAT: File Lookup Example
5. If an entry is not found, look in entry 16 in the FAT to identify the next cluster.
6. Repeat the lookups until the entry is found. Identify the starting cluster from the directory entry (23 for example).
7. Read the contents of cluster 23 and then lookup entry 23 in the FAT for the next cluster, 24 for example.
8. Read the contents of 24 and repeat the FAT lookups until <EOF> is reached
19
@stake
? 2002 @stake, Inc. FIRST File System Tutorial37
FAT: File System Layer
• Two structures describe the file system
• The Boot Sector structure describes file system attributes. The structure is different in FAT16 and FAT32.
– Cluster size
– Location and size of the FAT
– Start and end of the Data Area
– Magic value (0xAA55)
• The File Allocation Table (FAT) manages the clusters within the Data Area
@stake
? 2002 @stake, Inc. FIRST File System Tutorial38
FAT: TASK & sectors
• TASK uses sectors for addressing instead of clusters
• The first cluster is hundreds of sectors into the file system, after the Super Block and FATs
• Therefore, sectors are used as addresses so that we can access the contents of the FAT and Super Block
20
@stake
? 2002 @stake, Inc. FIRST File System Tutorial39
FAT: fsstat
• The fsstat tool lists the important Boot Sector values and a summary of the FAT.
# fsstat –f fat fat.dd
• Boot Sector Values:
– Volume ID
– Sector and Cluster size
– Sector of first Cluster
– Sector Range
– Locations of primary and backup FAT
• FAT Data
– Sequence of consecutive sectors that are allocated and either the next nonconsecutive sector or EOF.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial40
FAT: Content Layer
• The clusters in the Data Area contain the file and directory content data.
• We can view the contents of the clusters using the ‘dcat’ command in TASK, or using ‘dd’.
• NOTE: TASK will require the sector address and not the cluster.
# dcat –f fat fat.dd 390
• We can get the statistics of a sector using the ‘dstat’ command:
# dstat –f fat fat.dd 390
21
@stake
? 2002 @stake, Inc. FIRST File System Tutorial41
FAT: Meta Data Layer
• The Directory Entries describe the files and directories in the file system.
• We can list of details of which using the ‘ils’ command from TASK:
# ils –e –f fat fat.dd
• We can get more details of a specific one using ‘istat’:
# istat –f fat fat.dd 5
• We can view the contents of the file using that directory entry, use ‘icat’:
# icat –f fat fat.dd 5
@stake
? 2002 @stake, Inc. FIRST File System Tutorial42
FAT: TASK & Directory Entries
• Directory entries in FAT are not given addresses
• TASK treats every 32-bytes as a possible directory entry
• The Root Directory has no directory entry, so it is given an address of 2 and the first 32-bytes in the first cluster are given the address 3. The second 32-bytes are given the address 4 etc.
• Not every address will be a directory entry
22
@stake
? 2002 @stake, Inc. FIRST File System Tutorial43
FAT: Name Layer
• All names are stored in the Directory Entry structure.
• We can get a listing of the file and directory names using the ‘fls’ command:
# fls –rp –f fat fat.dd
@stake
? 2002 @stake, Inc. FIRST File System Tutorial44
FAT: File Allocation
• Results from testing a Windows 2000 system
• The new directory entry is placed at the end of the existing list. Entries from deleted files are not reallocated, this results in a LONG list of deleted files.
• The clusters are allocated by scanning the FAT for clusters that are free. I’m not 100% sure of the algorithm yet:
– If the size is known, it will look for enough consecutive clusters (i.e. it will skip over single free ones if it needs several of them)
– If the size is unknown or it is from a redirection, it allocates single free clusters (but sometimes it skips over single ones and allocates several consecutive ones - maybe due to buffering)
• In the last cluster, only the sectors that contain data are overwritten (i.e. slack space)
23
@stake
? 2002 @stake, Inc. FIRST File System Tutorial45
FAT: File Deletion
• Directory entries are unallocated by setting the first letter (which is also the first byte of the structure) to ‘0xEF’
• No other directory entry values are changed:
– The name will still exist
– The times do not reflect the deletion time
• The clusters are unallocated by setting the corresponding FAT entries to 0
@stake
? 2002 @stake, Inc. FIRST File System Tutorial46
FAT: Allocation & Deletion Tools
• We can see how files are allocated by using the following tools:
– ‘ils –e’ to list the status of all ‘inodes’
– ‘fls –rp’ to list all known directory entries
– ‘fsstat’ to get the contents of the FAT
– ‘dls -le’ to list the status of sectors
• Save the output of these tools, create or delete files, and then compare the new tool output
24
@stake
? 2002 @stake, Inc. FIRST File System Tutorial47
Forensics Notes
• Because meta data is not in a static location, it can exist for a long time before it is overwritten
• Deleted directory entries still contain names and starting cluster -but not a deleted time
• Directory entry allocation algorithm is such that deleted entries are not overwritten
• Only the Access time is required to be updated and it is only accurate to the day. Created and written times are not consistently written
• Deleted data can be found in the slack space of files
• Wiping tools will sometimes leave long file names such as “aaaaaaaaaaaaaaaaaaaaaaaaaaaaa”
@stake
? 2002 @stake, Inc. FIRST File System Tutorial48
UNIX File Systems (FFS and EXT2FS)
25
@stake
? 2002 @stake, Inc. FIRST File System Tutorial49
UNIX File Systems
• Solaris, HP, and other BSD flavors use the Berkley Fast File System (FFS)
• Linux uses the Extended 2 File System (EXT2FS) and the Extended 3 File System (EXT3FS)
• EXT2FS is based on FFS
• We will present FFS first and then show the differences with EXT2FS
@stake
? 2002 @stake, Inc. FIRST File System Tutorial50
UNIX: File System Layout
• The File System is divided into groups, called Cylinder Groups in FFS (Block Groups in EXT2FS)
• All groups are the same size
• Groups contain both meta data structures and file content
• Each group manages its own resources
26
@stake
? 2002 @stake, Inc. FIRST File System Tutorial51
UNIX: File System Layout Diagram
Group
@stake
? 2002 @stake, Inc. FIRST File System Tutorial52
UNIX FFS: Cylinder Groups
• All Cylinder Groups in a given file system are the same size, which is identified in the Super Block
• The first group starts at offset 0 (but the first 16 are reserved)
• Each group contains a copy of the Super Block, but it increments its offset (it has an offset of 16 for the first group due to the disk labels)
• Each group contains a Group Descriptor structure that describes the group. The inode and block bitmaps are stored in the group descriptor.
27
@stake
? 2002 @stake, Inc. FIRST File System Tutorial53
UNIX FFS: Blocks & Fragments
• Sequential disk sectors (within the same group) are organized into Blocks
• Blocks may also be broken into Fragments (which are usually larger than a sector)
• Each Fragment is addressable
• Example:
– Sector Size: 512-bytes
– Block Size: 4096-bytes (8 sectors)
– Fragment Size: 1024-bytes (2 sectors)
• Each group has a bit-map that specifies the allocation status of each fragment
@stake
? 2002 @stake, Inc. FIRST File System Tutorial54
UNIX FFS: Blocks & Fragments Diagram
0 2 41 3 5 6 7
512-bytesector 1024-byte
fragment
4096-byteblock
28
@stake
? 2002 @stake, Inc. FIRST File System Tutorial55
UNIX FFS: inodes
• File and directory meta data are stored in inode structures
• A static number of inode structures are allocated at file system creation time
• Each group contains the same number of inode structures, which are stored in an inode table.
• Multiple files may use the same inode (hard-links)
• Inodes have unique addresses, starting with the number 2
• The Root directory (/) is always described by inode 2
• Each group contains a bit-map that describes the allocation status of inodes in that group
@stake
? 2002 @stake, Inc. FIRST File System Tutorial56
UNIX FFS: Summary of the inode Structure
• mode (2-bytes) [ permissions & type]
• number of links (2-bytes)
• size (8-bytes)
• Last access date & time (8-bytes)
• Last modified date & time (8-bytes) – file data modifications
• Last change date & time (8-bytes) - file status (inode) change
• User ID (4-bytes) and Group ID (4-bytes)
• Partial list of allocated blocks (direct pointers) and list of blocks with additional pointers (indirect pointers)
29
@stake
? 2002 @stake, Inc. FIRST File System Tutorial57
UNIX FFS: Block Pointers
• FFS was designed on research that showed that small files were more common the large files
• An inode contains 12 direct, 1 single-indirect, 1 double-indirect, and 1 triple-indirect pointer
• Direct Pointer: Points directly to a block that contains the file or directory content
• Single-indirect Pointers: Points to a block that contains a list of direct pointers, which point to the data
• Double-indirect Pointers: Points to a block that contains a list of single-indirect pointers
• Triple-indirect Pointers: Points to a block that contains a list of double-indirect pointers
@stake
? 2002 @stake, Inc. FIRST File System Tutorial58
UNIX FFS: Block Pointer Diagram
19 20 21 22 282726252423 29 30 31
direct
19 - filecontent
...32 33 34 35
99
32 - filecontent
100 101
200 - filecontent
...
...
single
double
direct
200 201direct
single
30
@stake
? 2002 @stake, Inc. FIRST File System Tutorial59
UNIX FFS: Time & Date Format
• UNIX saves the time & date as the number of seconds since 1/1/1970 UTC.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial60
UNIX FFS: Directory Entries
• File and directory names are stored in Directory Entrystructures that are located in blocks allocated by the parent directory (similar to FAT, but different contents)
• Each structure contains a name and points to an inode structure
• Directory Entry structures have a dynamic size and form a linked list
• Multiple directory entries can point to the same inode (symbolic links)
• The ‘.’ and ‘..’ directory entries are the first two in all directories
31
@stake
? 2002 @stake, Inc. FIRST File System Tutorial61
UNIX FFS: Directory Entry Structure
• inode (4-bytes)
• record length (2-bytes)
• type (1-byte) {This does not exist in all versions}
• name length (1-byte) {This is 2-bytes if type does not exist}
• name (less than 255, NULL terminated)
• The record length is rounded up to a multiple of 4
@stake
? 2002 @stake, Inc. FIRST File System Tutorial62
UNIX FFS: Directory Entry Example
16 file1 file28357 16 8358 16 8359 file3
NOTE: The ‘.’ and ‘..’ entries are not shown
file35835916
file25835816
file15835716
namename leninoderec len
32
@stake
? 2002 @stake, Inc. FIRST File System Tutorial63
UNIX FFS: File Lookup Example
• Lookup /dir1/file1.dat
1. Locate inode 2 in the Inode Table of Group 1.
2. Read the blocks allocated to inode 2 and process them as directory entries.
3. Analyze each directory entry for the one with the name “dir1”. Save the inode value in the directory entry, 1810 for example.
4. Calculate which group inode 1810 is part of by dividing it by the number of inodes per group, group 2 for example.
5. Locate inode 1810 in the Inode Table of Group 2.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial64
UNIX FFS: File Lookup Example
6. Read the blocks allocated to inode 1810 and process as directory entries.
7. Look for an entry with the name “file1.dat” and save the associated inode value, 2020 for example.
8. Calculate which group inode 2020 is part of by dividing it by the number of inodes per group, group 3 for example.
9. Read the inode 2020 structure from the Group 3 Inode Table and the blocks allocated by it.
33
@stake
? 2002 @stake, Inc. FIRST File System Tutorial65
UNIX FFS: File System Layer
• Two types of structures describe the file system
• The Super Block structure describes file system attributes.
– Size of Blocks & Fragments
– Size of groups
– Number of blocks and inodes per group
– Offset of group descriptor within each group
• The Group Descriptor structure within each Cylinder Group describes the group attributes
– Location of inode bitmap
– Location of block bitmap
@stake
? 2002 @stake, Inc. FIRST File System Tutorial66
UNIX FFS: fsstat
• The fsstat tool lists important Super Block values and the groups layout
# fsstat –f solaris solaris.dd
• Super Block Values:
– Block and fragment sizes
– Range of block addresses
– Range of inode addresses
• Group Info:
– Block and inode range of each group
– Location of inode table, group descriptor, and Super Block in each group
34
@stake
? 2002 @stake, Inc. FIRST File System Tutorial67
UNIX FFS: Content Layer
• The blocks and fragments contain the file and directory content data.
• We can view the contents of the clusters using the ‘dcat’ command in TASK (& TCT/TCTUTILs), or using ‘dd’.
# dcat –f solaris solaris.dd 390
• We can get the statistics of a sector using the ‘dstat’ command:
# dstat –f solaris solaris.dd 390
@stake
? 2002 @stake, Inc. FIRST File System Tutorial68
UNIX FFS: Meta Data Layer
• The inode structures describe the files and directories in the file system.
• We can list of details of which using the ‘ils’ command:
# ils –e –f solaris solaris.dd
• We can get more details of a specific one using ‘istat’:
# istat –f solaris solaris.dd 200
• We can view the contents of the file using that inode with ‘icat’:
# icat –f solaris solaris.dd 200
35
@stake
? 2002 @stake, Inc. FIRST File System Tutorial69
UNIX FFS: Name Layer
• All names are stored in Directory Entry structures.
• We can get a listing of the file and directory names using the ‘fls’ command:
# fls –rp –f solaris solaris.dd
@stake
? 2002 @stake, Inc. FIRST File System Tutorial70
UNIX FFS: Block Allocation
• Blocks are allocated to the same Cylinder Group as the inode. If the Cylinder Group is full or near full, a new one ischosen based on disk rotation and percentage of free blocks.
• Solaris will not fill up a group with a large file. It will fill up half of the available blocks, then move on to the next group. OpenBSD will fill up the group before moving on.
• The remainder of the fragment is wiped (i.e. no slack space)
36
@stake
? 2002 @stake, Inc. FIRST File System Tutorial71
UNIX FFS: Inode & Directory Entry Allocation
• File inodes are allocated in the same Cylinder Group as the parent directory. If it is full, a quadratic hash search is performed.
• Directory inodes are allocated in the next Cylinder Group with a greater than average number of free inodes
• The first available inode (lowest address) in the group is used
• Directory Entries are inserted into the list at the first available location
@stake
? 2002 @stake, Inc. FIRST File System Tutorial72
UNIX FFS: File Deletion
• Blocks and Fragments: Blocks are unallocated in the bitmap (including indirect blocks).
• Inode: The inode is set to unallocated in the group’s inode bitmap. The size and block pointer fields are reset to 0. The Modified & Changed times are set to the current time.
• Directory Entry: The record length of the previous directory entry is increased by the length of the deleted entry. Solaris will also reset the inode value to 0.
37
@stake
? 2002 @stake, Inc. FIRST File System Tutorial73
UNIX FFS: Deleted File Example
32 file1 file28357 16 8358 16 8359 file3
file2 is deleted
file35835916
file25835816
file15835732
namename leninoderec len
file35835916
file25835816
file15835716
namename leninoderec len
@stake
? 2002 @stake, Inc. FIRST File System Tutorial74
UNIX FFS: Allocation & Deletion Tools
• We can see how files are allocated by using the following tools:
– ‘ils –e’ to list the status of all inodes
– ‘istat’ to get a list of all blocks and fragments that were allocated to a given inode
– ‘fls –rp’ to list all known directory entries
– ‘dls -le’ to list the status of blocks and fragments
38
@stake
? 2002 @stake, Inc. FIRST File System Tutorial75
EXT2FS
@stake
? 2002 @stake, Inc. FIRST File System Tutorial76
UNIX EXT2FS: Block Groups
• EXT2FS calls the groups Block Groups - They are not organized by disk geometry
• Each group manages itself and contains a duplicate copy of the Super Block (which is always at the same offset)
• Each group has a table of Group Descriptors (FFS only had the local one)
• Entire blocks are allocated to the inode and block bitmaps
• The Sparse Super Block option prevents the Super Block from existing in every group and the space is used for data
39
@stake
? 2002 @stake, Inc. FIRST File System Tutorial77
UNIX EXT2FS: inode
• The EXT2FS inode contains much of the same data as FFS
• They have a different layout though
• The EXT2FS inode has a deletion time
@stake
? 2002 @stake, Inc. FIRST File System Tutorial78
UNIX EXT2FS: Block Allocation
• Blocks are allocated in the same Block Group as the inode (if there is room). It tries to allocate a block where the next 7 are also free.
• Attention is not paid to one file filling up a Block Group.
40
@stake
? 2002 @stake, Inc. FIRST File System Tutorial79
UNIX EXT2FS: Inode & Dir Entry Allocation
• File inodes are allocated to the same block group as parent directory, unless there is no room. Then they are allocated to another based on a Quadratic Hash or linear search.
• Directory inodes are allocated to the block group with the most free blocks and greater than average free inodes. The search begins at group 0, so it could be the same one as the parent directory.
• In both cases, the first (lowest address) inode is used.
• Directory entries are placed in the first available location in the list.
@stake
? 2002 @stake, Inc. FIRST File System Tutorial80
EXT2FS: File Deletion
• Linux changed how they deleted files from the 2.2 kernel to 2.4
• Blocks & Fragments: Unallocated in the group bitmap.
• Inode: The inode is set to unallocated, but the block pointers and size are not reset (the size is set to 0 when a directory is deleted though).
• The Modified, Changed, and Deleted times of the inode are set to the current time.
• Directory Entries: The length of the previous entry is increased. In the 2.2 kernel the inode value was not cleared. As of 2.4 it is set to 0, as FFS does.
41
@stake
? 2002 @stake, Inc. FIRST File System Tutorial81
EXT2FS: Other Tools
• debugfs runs on Linux and will let you view and change any structure
@stake
? 2002 @stake, Inc. FIRST File System Tutorial82
UNIX: Forensics Notes
• Times still exist in unallocated inodes, but usually not the directory entry pointer (except Open & FreeBSD FFS).
• With EXT2FS we still have access to the block pointers
• We can identify meta data wiping tools because there will be zeroed inodes in between allocated inodes in a CG
• Blank directory entries in the middle of a directory are also suspect
• To find the blocks for an unallocated inode, start looking in the CG, not the beginning of the disk (use ‘fsstat’ and ‘dls’)
• By manually parsing the directory entry structure, you can sometimes tell some order of file deletions
42
@stake
? 2002 @stake, Inc. FIRST File System Tutorial83
Journaling File Systems
@stake
? 2002 @stake, Inc. FIRST File System Tutorial84
Journaling File Systems: Motivation
• When a system crashes or is not shutdown properly, the file system must be validated
• The ‘fsck’ tool scans inodes, directory entries, and bitmaps to ensure that all structures agree
• This can take a LONG time on large file systems
43
@stake
? 2002 @stake, Inc. FIRST File System Tutorial85
Journaling File Systems: Overview
• A log is kept that contains meta data (and sometimes content) changes, atomic transactions are used.
• Before the data is written, the intended updates are recorded
• If the system crashes, the log is consulted and it can easily identify which transactions did not complete. It then either retries the update or turns back to the previous state.
• No public forensic tools currently take advantage of these logs.
• Examples: NTFS, EXT3FS, IBM JFS
@stake
? 2002 @stake, Inc. FIRST File System Tutorial86
NTFS
44
@stake
? 2002 @stake, Inc. FIRST File System Tutorial87
NTFS
• Based on HPFS (designed by Microsoft and IBM)
• Used in Microsoft Windows NT, 2000, & XP
• Design is not widely published by Microsoft, but a sourceforge project exists to write a Linux driver and document it
• NTFS is object oriented and very flexible
• There are too many data structures to address in detail in this talk (refer to TASK or Linux source)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial88
NTFS: File System Layout
• 3 Major Areas (similar to FAT)
• Boot Sector
– Contains file system details such as sizes and offsets
• Master File Table (MFT)
– Manages the files and other objects
• Data Area
– Content area for files and objects
45
@stake
? 2002 @stake, Inc. FIRST File System Tutorial89
NTFS: Layout Example
Boot Sector
Master File Table
Data Area
@stake
? 2002 @stake, Inc. FIRST File System Tutorial90
NTFS: Master File Table
• The Master File Table (MFT) contains an entry for each user and system file
• MFT entries are called File structures and have a static size of 1024 bytes
• MFT entries store file and directory meta data
• Data is stored in attribute structures (objects)
• Some files will require multiple MFT entries (due to static size)
• Multiple files may use the same MFT entry (hard-links)
• MFT entries are addressed (File Reference) and have a sequence number
46
@stake
? 2002 @stake, Inc. FIRST File System Tutorial91
NTFS: MFT Entry Examples
• The MFT itself and mirror copy (0 & 1)
• LogFile (2)
• Root Directory (5)
• Volume Bitmap (3)
• Bad Clusters (8)
• Quota data (9)
• All files and directories (25+)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial92
NTFS: Attributes
• An attribute describes a specific property of the file or directory (i.e. the name or dates)
• Attributes have a header and a value.
• The header is always stored in the MFT entry
• The value is stored in either the MFT entry (resident) or in the Data Area (non-resident)
47
@stake
? 2002 @stake, Inc. FIRST File System Tutorial93
NTFS: Attribute Header
• The attribute header is always located in the MFT
• It has the following fields:
– type (4-bytes) & id (2-bytes)
– length (4-bytes)
– resident flag
– name length and name (all have a name except the primary data one)
– If resident: A pointer to attribute stream
– If non-resident: A run-list of the clusters that contain the attribute data
– Virtual Cluster Number (VCN): The address of the cluster in a run with respect to the beginning of the attribute (0 - n), not the beginning of the file system (Logial Cluster Number - LCN)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial94
NTFS: Attribute Types
• Standard Information: Read Only, Archive, MAC Times, number of links (type 16)
• File Name(s) (type 48)
• Data (type 128)
• Index Root (type 144) and Index Allocation (type 160) (for directory contents)
• Security Descriptor (type 80)
• Anything: For example, the icon on a Macintosh file is stored in an attribute so it gets moved with the file (more type 128)
• Each attribute type has its own data structure
48
@stake
? 2002 @stake, Inc. FIRST File System Tutorial95
NTFS: Standard Information Attribute
• Attribute Type 16
• Dates and Times:
– Creation
– Last MFT change
– Last write
– Last access
• Permissions in DOS format
• version
• Owner Id
• Security Id
@stake
? 2002 @stake, Inc. FIRST File System Tutorial96
NTFS: File Name Attribute
• Attribute Type 48
• MFT Reference Number
• MFT Sequence Number
• Dates and Times:
– Creation
– Last MFT change
– Last write
– Last access
• Character set
• Name length and name
49
@stake
? 2002 @stake, Inc. FIRST File System Tutorial97
NTFS: Data Attribute
• Attribute Type 128
• Data attributes just contain the raw data
• The name of the data is given in the attribute header
• The only attribute that can not have a name is the primary data attribute, which is typically given the name $Data by default.
• Directories can have data attributes in addition to their B-Tree attributes
@stake
? 2002 @stake, Inc. FIRST File System Tutorial98
TASK & NTFS Attributes
• MFT entries are addressed the same way in NTFS as inodes are with UNIX file systems
• One can give the Type and Id (if there are more than one of a given type) for a specific instance in the form of:
MFT-TYPE-ID
• For example:
# icat ntfs.dd 300
# icat ntfs.dd 300-128 (both are the same)
# icat ntfs.dd 300-128-3 (if an ADS exists)
50
@stake
? 2002 @stake, Inc. FIRST File System Tutorial99
NTFS: Data Area
• Data that can not fit in the MFT entry is stored in the Data Area
• The Data Area is organized into Clusters (groups of consecutive sectors)
• Cluster numbering starts with the first partition sector (unlike FAT)
• The clusters are managed using the Volume Bitmap MFT Entry (#6)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial100
NTFS: Directories
• Directories store the file descriptors in B-trees, which are sorted by any descriptor (file name by default)
• The B-tree nodes contain the:
– MFT reference number
– The structure that is being sorted by (file name structure for example)
• The MFT entry also stores the file name in an attribute
• Names are stored in UNICODE
• The Root Directory (\) is pointed to by a static MFT entry (number 5)
51
@stake
? 2002 @stake, Inc. FIRST File System Tutorial101
NTFS: File Lookup Example
• Lookup \dir1\file1.dat
1. Read the Boot Sector to locate the beginning of the MFT
2. Read the MFT Entry for the Root Directory (#5)
3. Find & read the Index Root and Index Allocation attributes
4. Process the B-Tree and analyze the file name structures for the one with the name “dir1”. Save the file reference value, 180 for example.
5. Read MFT entry 180
6. Find & read the Index Root and Index Allocation attributes
@stake
? 2002 @stake, Inc. FIRST File System Tutorial102
NTFS: File Lookup Example
7. Process the B-Tree and find the file name structure with “file1.dat”. Save the reference value, 181 for example.
8. Read MFT entry 181.
9. Find & read the “$Data” attribute
52
@stake
? 2002 @stake, Inc. FIRST File System Tutorial103
NTFS: File System Layer
• Several structures describe the file system
• The Super Block contains the location of the MFT and the cluster sizes
• The MFT entry in the MFT describes the location and size of itself
• The $Log MFT entry contains the atomic transaction logs
• All of these can be viewed with the ‘fsstat’ command
# fsstat –f ntfs ntfs.dd
@stake
? 2002 @stake, Inc. FIRST File System Tutorial104
NTFS: Content Layer
• Data is stored in the MFT itself if it is small enough
• Larger data is stored in clusters in the Data Area
• Clusters are numbered from the beginning of the file system
• The dcat tool will allow us to display the contents of any cluster
# dcat –f ntfs ntfs.dd 8016
• The icat tool will allow us to display the contents of attributes that are stored in the MFT:
# icat -f ntfs ntfs.dd 76-160
53
@stake
? 2002 @stake, Inc. FIRST File System Tutorial105
NTFS: Meta Data Layer
• All file and directory meta data is generally located in the MFT
• If the value of the meta data is too large, it will be located in the Data Area
• Each MFT is addressed using a File Reference Number
• We can view the details of an MFT entry using istat:
# istat –f ntfs ntfs.dd 5
• We can list them all using ils:
# ils –f ntfs –e ntfs.dd
• The contents of the file can be viewed using icat (with no type or id value):
# icat -f ntfs ntfs.dd 9901
@stake
? 2002 @stake, Inc. FIRST File System Tutorial106
NTFS: Naming Layer
• File names are stored in a node of the B-Tree in the directory’s MFT entry
• The file’s MFT entry also has all names that it uses (multiple ones if it has hard-links)
• The list of known file names can be shown with fls:
# fls –f ntfs –rp ntfs.dd
54
@stake
? 2002 @stake, Inc. FIRST File System Tutorial107
NTFS: File Allocation
• An area following the MFT is reserved for its expansion
• MFT entries are allocated linearly
• Names are allocated into directories based on their name and location in the B-tree
• Only the sectors in the last cluster that are used are overwritten (slack space)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial108
NTFS: File Deletion
• Content Blocks are unallocated by clearing the bits in the bitmap
• MFT entry is freed by setting the in-use flag. The attributes are unchanged. (link count never goes to 0).
• No times are set to the current time
• The name still exists in the MFT entry
• Parent directory Index Tree is resorted (the data may still exist in the tree depending on the other files in the directory)
55
@stake
? 2002 @stake, Inc. FIRST File System Tutorial109
Forensics Notes
• Because of MFT allocation, it is likely that unallocated entries will be reused faster than with UNIX inodes
• No times are set when an MFT is unallocated, so time lines of file activity are not as revealing
• Deleted file names will not exist as frequently as with FAT or UNIX because they will get overwritten when the tree is sorted
• We can collect slack space data
• Clean MFT entries in the middle of the MFT is likely a result ofwiping tools
• If times in Standard Info are different than File Name, wiping tools may have been used
@stake
? 2002 @stake, Inc. FIRST File System Tutorial110
NTFS: Other Tools
• NTFS File Sector Information Utility (nfi) (Microsoft)
• NTFS Info (www.sysinternals.com)
56
@stake
? 2002 @stake, Inc. FIRST File System Tutorial111
EXT3FS
@stake
? 2002 @stake, Inc. FIRST File System Tutorial112
EXT3FS: Overview
• EXT3FS is identical to EXT2FS except it interfaces with the Journal layer
• The journal is kept in a special file or a specific journaling device
• Currently, no tools publicly exist to utilize the journal for forensics
57
@stake
? 2002 @stake, Inc. FIRST File System Tutorial113
Summary of Structures
Directory Entries
B-tree Index trees
Directory Entries
Name Layer
InodesMFT EntriesDirectory Entries
Meta Data
Fragments & Blocks in groups
Clusters in the Data Area
Clusters in the Data Area
Content
Super Block, Group Desc
Boot Sector, MFT
Boo Sector, FAT
File System
UNIXNTFSFAT
@stake
? 2002 @stake, Inc. FIRST File System Tutorial114
Log-based File Systems
58
@stake
? 2002 @stake, Inc. FIRST File System Tutorial115
Basics of Log-structured File Systems
• “Classical” File Systems waste time by seeking to allocated blocks and overwriting the contents (for data or inodes)
• Log-structured File Systems work like a log file and append new data to the “end” of the data
• All data and meta data are written to the current pointer.
• An inode index file points to the current location of the inode
• The inode points to the current location of its data blocks
• Deleted Content is not overwritten for a while...
• No Forensic Tools currently take advantage of Log FS...
• Example: BSD Log File System, Linux Log-Structured FS
@stake
? 2002 @stake, Inc. FIRST File System Tutorial116
PALM OS
• Everything is a database (even applications)
• Each database stores the dates of Creation, last modification, and last backup
• When a database is deleted, it is flagged as not in use, but not deleted (until a hotsync is done)
• See Joe Grand’s talk on the pdd Palm Acquisition tool (Wednesday)
59
@stake
? 2002 @stake, Inc. FIRST File System Tutorial117
Summary
• Knowledge of file system structures will help interpreting the results of tools
• Knowledge will also help write tools when ones do not exist
• Evidence of wiping tools can be found by examining the structures and their location
• Using allocation algorithms, one can sometimes recover deleted data more quickly
@stake
? 2002 @stake, Inc. FIRST File System Tutorial118
Software References
• TASK can be found at:
http://www.atstake.com/research/tools/task
• TCT & TCTUTILs (for UNIX only) can be found at:
www.fish.com
www.porcupine.org
www.cerias.purdue.edu/homes/carrier
60
@stake
? 2002 @stake, Inc. FIRST File System Tutorial119
References
• FAT
– Microsoft FAT32 File System Specification
– Microsoft Long Filename Specification
• FFS
– The Design and Implementation of the 4.4 BSD Operating System (McKusick, Bostic , Karels , Quarterman)
• EXT2FS
– The extended-2 filesystem overview (Oxman)
– Design and Implementation of the Second Extended Filesystem (Card, Ts’o,Tweedie)
@stake
? 2002 @stake, Inc. FIRST File System Tutorial120
References
• EXT3FS
– Transcript of Stephen Tweedie talk (olstrans .sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html)
• NTFS
– Inside Windows 2000 - 3rd Edition (Solomon, Russinovich)
– Linux NTFS Project (linux-ntfs.sourceforge.net)