aff4: the new standard in forensic imaging - osdfcon · aff4: the new standard in forensic imaging...
TRANSCRIPT
AFF4:Thenewstandardinforensicimagingandwhyyoushouldcare
Dr.BradleySchatzDirector,SchatzForensic
v1.1– OSDFCon 2016©SchatzForensic2016
©2016SchatzForensic
Aboutme• BradleySchatz
– PhD,DigitalForensics(2007); BSc,ComputerScience• SchatzForensic/Evimetry(2009-)
– Practitioner,R&D,toolvendor• Researchaffiliations
– JournalofDigitalInvestigation(EditorialBoard)– DFRWSConference,Chair,TechnicalProgramCommittee(2016)
• Practicalcontributions– VolatilityMemoryForensicsFramework(Vista&Windows7support)(2010)– Autopsy(index.dat support)
• QueenslandUniversityofTechnology– Adjunctassociateprofessor,doctoralsupervision
©2016SchatzForensic
Agenda• WhyshouldIcareaboutforensicformats?• What’swrongwithcurrentforensicformats?• WhatisAFF4?• AFF4StatusUpdate
WhyshouldIcareaboutforensicformats?
©2016SchatzForensic
Limitationsinforensicformatshaveprofoundeffectsonpractice
• Theimageasalinearbitstream– Triage:relianceonlogicalimagingandacceptanceoflossofpotentiallyrelevantdata
• Theusageofheavyweightcompression– SignificantdelaysduetoCPUconsumption
• Theusageofnocompression– Significantdelaysduetohashingandcopyingsparsedata
• Inextensiblemetadatastorage– Toolinteroperabilityanongoingproblem
©2016SchatzForensic
AFF4supportsstoringmultiplestreamspercontainerPmem aff4acquire=physicalmemory+mappedfiles
Virtualaddress
#pages
Acquiredmappedfiles&pagefiles
©2016SchatzForensicTimespentwaitingforresults
Evidentiary
Completeness
Digitalforensicbestpractice
Triage&IncidentResponse
Liveforensics
TheAFF4forensicformatenablesfasterandnewapproachestoacquisition
Increasedspeed
Liveanalysiscanoccurduringevidence
preservation
Non-linearpartialimages
©2016SchatzForensic
AFF4shiftsacquisitionthroughputtobeingCPUlimited1TBacquiredin20minutes(~50GB/min)
Acquisitiontechnique Acquire+Verify
Evimetry Wirespeed 0:52:04
Xways +WinFE 2:48:00
Macquisition EWF 7:08:38
1TBNVMe (Corei7-4578U,2Cores)Macbook ProA1502(Evimetry2.2.0a)
What’swrongwithcurrentforensicformats?
©2016SchatzForensic
ForensicImagingv1.0:RAW
• Good– Universaltoolsupport– 1:1mapping– Easytoimplement
• Bad– Nostandardisedmetadatastorage– Copyingsparseregions(zero-filled)isawasteoftime
– Linearbitstreamhashisabottleneckathighspeeds
MD5
SourceHardDrive
ACMECo.C1.D1.raw
ACMECo.C1.D1.raw.txt
# LinearBitstreamHash
©2016SchatzForensic
Thelinearbitstreamhashisabottleneckathighspeeds
TargetStorage Interconnect Hash Filesystem Interconnect Evidencestorage
Algorithm AverageThroughputMB/s
SHA1 619.23MD5 745.65Blake2b 601.87
©2016SchatzForensic
Linearbitstreamhashingisabottleneckwithcurrentgenerationstorage
TargetStorage Interconnect Hash Filesystem Interconnect Evidencestorage
TargetStorage SustainedRead1TBSeagate3.5”7200rpmSATA 100MB/sCurrentgeneration 3.5”7200rpmSATA 200MB/sIntel730SSD 550MB/sMacbook Pro1TB ~1 GB/sRAID15000rpmSAS >1GB/sSamsung850NVMe 1.5– 2.5GB/s
©2016SchatzForensic
ForensicImagingv2.1:ThreadedEWF• Good
– Imagesarefasttocopy(compressed)– Nearuniversaltoolsupport
• Bad– Inextensiblemetadatastorage– Poorlydefined*– Copying&Compressingsparseregions(zero)
isawasteoftime**– Deflatecompressionisabottleneck– Linearbitstreamhashisabottleneckathigh
speeds
*DespitetheexcellentworkofMetz**RecentEWFsupportssparseregions
MD5
Deflate DeflateDeflate
SourceHardDrive
ACMECo.C1.D1.e01
# LinearBitstreamHash
©2016SchatzForensic
Thedeflatealgorithmisasignificantbottleneck
TargetStorage Interconnect Hash Compress Filesystem Interconnect Evidence
storage
Data Deflate MB/s InflateMB/s
Highentropy 40.4 439
Lowentropy 259 IObound
*Singlecoreofquadcorei7-47703.4Ghzmeasuredwithgzip
©2016SchatzForensic
8-corei7&uncontendedIO?ThreadedEWFisCPUbound
TargetStorage Interconnect Hash Compress Filesystem Interconnect Evidence
storage
SHA1600MB/s
SATA3Intel720SSD~500MB/s
SATA3600MB/s
SATA3Samsung850EVOPro~500MB/s
Acquisition 240GB@255MB/s =14m35sVerification 240GB@350MB/s =10m37sTOTAL =25m12s
Deflate31.9MB/s/core
©2016SchatzForensic
What’swrongwithAFF(v1-3)?• Good
– Welldefinedformat– Opensource– ExtensibleName/Valuepairmetadatastorage– Nichecommercialtoolsupport
• Bad– Copying&Compressingsparseregions(zero)isa
wasteoftime– Deflatecompressionisabottleneck– Largecompressedchunksizes(16Mbydefault)
sloww/NTFSMFT
WhatisAFF4?
©2016SchatzForensic
ForensicImagingv4.0:AFF4(2009)
• ZIP64basedcontainer• Storagevirtualization• Extensiblelinked-datametadata
storage• Inter-containerreference
scheme• 2-levelindexing
• Opensourceimplementation
©2016SchatzForensic
AFF4StorageVirtualisation:theMap
ACMECo.S1.RAID0.af4
ACMECo.S1.D1.af4 # LinearBitstreamHash
ACMECo.S1.D2.af4
# LinearBitstreamHash
CompressedBlockStorageStream
VirtualStorageStream(Map)
©2016SchatzForensic
ExampleusesoftheAFF4Map• ReconstructingRAIDfromimages• Rearrangingspareanddatarangesinflashimages• Zerostoragecarving• Storingdiscontiguous dataranges• Storingnon-linearimages• Representingsparseregions
©2016SchatzForensic
Linkeddatametadatastorage<aff4://fd488f0f-95ad-45e4-a948-a36afcb03a08>
aff4:contains <aff4://08b52fb6-fbae-45f3-967e-03502cefaf92> ;aff4:stored <aff4://0658d383-3984-42f5-b1aa-c39f8e0cdbae> ;aff4:systemBiosVendor "American Megatrends Inc."^^xsd:string ;aff4:systemBiosVersion "F3"^^xsd:string ;aff4:systemChassisAssetTag "To Be Filled By O.E.M."^^xsd:string ;aff4:systemChassisSerial ""^^xsd:string ;aff4:systemChassisType "3"^^xsd:string ;aff4:systemChassisVendor "Gigabyte Technology Co., Ltd."^^xsd:string ;aff4:systemChassisVersion "To Be Filled By O.E.M."^^xsd:string ;aff4:systemEthernetAddress "94:DE:80:7C:EC:6C"^^xsd:string ;aff4:systemProductName "Z87X-UD3H"^^xsd:string ;aff4:systemProductSerial ""^^xsd:string ;aff4:systemProductUUID ""^^xsd:string ;aff4:systemProductVersion "To be filled by O.E.M."^^xsd:string ;aff4:systemVendor "Gigabyte Technology Co., Ltd."^^xsd:string ;aff4:systemboardAssetTag "To be filled by O.E.M."^^xsd:string ;aff4:systemboardName "Z87X-UD3H-CF"^^xsd:string ;aff4:systemboardSerial ""^^xsd:string ;aff4:systemboardVendor "Gigabyte Technology Co., Ltd."^^xsd:string ;aff4:systemboardVersion "x.x"^^xsd:string ;a aff4:ComputeResource .
• Arbitraryinformationstorage
• Refertodatarangesandinformation
• Inter-containerreferences
©2016SchatzForensic
ForensicImagingv4.1:AFF4(2010)
• Non-linearacquisition• Hashbasedimaging
(deduplication)
©2016SchatzForensic
ForensicImagingv4.2:AFF4(2015)
• Lightweightcompression• Blockbasedhashing• Partialacquisition
– whatwedidn’tacquire– whatwecouldn’tacquire
©2016SchatzForensic
Lightweightcompression
TargetStorage Interconnect Hash Compress Interconnect Evidencestorage
Compression Algorithm ThroughputMB/s/core*
Deflate(ZIP, gzip) 31.9Snappy(GoogleBigTable) 1,400LZO(ZFS) 1,540
©2016SchatzForensic
Blockbasedhashingallowshashingtoscaleacrossallcores
Hash
Compress CompressCompress
SourceHardDrive
Hash Hash
BlockHashes
# BlockHashesHash
©2016SchatzForensic
BlockhashingshiftsthebottleneckfromfromCPUtoI/O
TargetStorage Interconnect Hash Compress Filesystem Interconnect Evidence
storage
SHA1600MB/s/core
SATA3Intel730SSD500MB/s
4xSATA32.4GB/s
RAID04xSATA32TB800MB/s
SnappyAvg1.5GB/s/core
Acquisitionapplication Linear Acquisition Verification
X-WaysForensics 14:35255MB/s(15.3 GB/min)
10:37350MB/s(21.0 GB/min)
Evimetry (linear) 7:23500MB/s(30.3GB/min)
4:12888MB/s(53.33GB/min)
©2016SchatzForensic
Generationofasinglehashw/blockbasedhashing
ACMECo.C1.D1.af4ACMECo.C1.D1.af4
BlockHashes
CompressedBlockStream
##
VirtualBlockStream(Map)
LinearBlockHash
MapHash
BlockHashesHash
##
##
©2016SchatzForensic
Implementations• 4scientificallypeerreviewedpapersover6years
• 3prototypeimplementations(2009– 2013)– 3languages,4revisionsoftheMap– Subtleimplementationdifferencesduetoambiguity– HeritagevisibleinGoogleResponseRig
• 2currentimplementations– Evimetry(Java)&Rekall/libaff4(CandPython)
• Convergenceisrequired
©2016SchatzForensic
Convergence:AFF4StandardisationEffort
• AFF4WorkingGroup– BradleySchatz(Evimetry),MichaelCohen(Google)chairing– JoeSylve (Blackbag)– Firstmeeting@DFRWS2016
• Intendedoutputs– Corporaofstandardimages[draft]– Specification(AFF4s)[draft]– Opensourceimplementations(C,Python,Java)[pre-draft]
©2016SchatzForensic
AFF4StandardisationEffort:Changes&Clarifications
• Namespacechange–Was http://afflib.org/ nowhttp://aff4.org/
• Propertynamingmadeconsistent• ImageStreamIndex(compressedblockstorage)–Was[offset0,offset1,offset2,offset3 ..offsetn]– Now[(offset0,length0),(offset1,length1)…]
• IdentificationofImageincontainer
©2016SchatzForensic
Sleuthkit AFF4support
• Draftstandardimage(producedbyEvimetry)
• Readwithlibaff4+AFF4spatches
• Patchestosleuthkit andlibaff4comingverysoon
©2016SchatzForensic
Sleuthkit AFF4support
• Draftstandardimage(producedbyEvimetry)
• Readwithlibaff4+AFF4spatches
• Patchestosleuthkit andlibaff4comingverysoon
Nextsteps
©2016SchatzForensic
Volatilememory
• Partialvolatilememoryacquisition–Multi-tenantconsiderations– Virtualmachinehostkernelmemory
• Livevolatilememoryanalysisandacquisition
©2016SchatzForensic
Front-loadedpre-processing
• Filehashingduringacquisition– Alreadyimplemented– Spinningdisk– optimizingpathacrossdiskvsRAM(lowseek)– SSD– notsuchanissue– Noneedforexpensiveprocessingforknownfiles
• Carvinglandmarkandfeatureidentification– Noneedforexpensivediskscans
©2016SchatzForensic
AFF4asaninterchangeformat
• RelationshipwithDFAXetc?
©2016SchatzForensic
Moretocome• Centralpointofcommunication– http://aff4.org/
• Updatestocome– DFSci mailinglist– sleuthkit-users
• Interestedinhelping?– Sendusanemail– [email protected] &[email protected]
ContactDrBradleySchatzhttps://evimetry.com/[email protected]@blschatz
'HardDiskDriveX-Ray'imageby JeffKubina