file systems for flash memories - politecnico di...
TRANSCRIPT
File Systems for Flash File Systems for Flash MemoriesMemories
Marcela ZuluagaMarcela ZuluagaSebastian IsazaSebastian Isaza
Dante RodriguezDante Rodriguez
OutlineOutline
Introduction to Flash MemoriesIntroduction to Flash MemoriesIntroduction to File SystemsIntroduction to File SystemsFile Systems for Flash MemoriesFile Systems for Flash MemoriesYAFFS (Yet Another Flash File System)YAFFS (Yet Another Flash File System)
Introduction to Flash MemoriesIntroduction to Flash Memories
Memories technology Memories technology classificationclassification
Flash Memories: main featuresFlash Memories: main features
NNonon--volatilevolatile: it does not need power to maintain : it does not need power to maintain the information stored in the chip.the information stored in the chip.Fast read Fast read access timesaccess times (though not as fast as (though not as fast as volatile volatile DRAMDRAM) and better shock resistance than ) and better shock resistance than hard diskshard disks..Hence, suitable for data storage in embedded Hence, suitable for data storage in embedded devices.devices.
Introduction to Flash MemoriesIntroduction to Flash Memories
……featuresfeatures
Unlike Unlike EEPROMEEPROM, it is erased and programmed , it is erased and programmed in blocks consisting of multiple locationsin blocks consisting of multiple locationsCosts far less than Costs far less than EEPROMEEPROMExamples of applications include USB Flash Examples of applications include USB Flash drives, drives, digital audio playersdigital audio players, , digital camerasdigital cameras and and mobile phonesmobile phones
Introduction to Flash MemoriesIntroduction to Flash Memories
Two typesTwo types
NOR FlashNOR FlashRandom, direct access Random, direct access interface for readingsinterface for readingsFast random readsFast random readsSlow erase and writeSlow erase and writeSuitable for code Suitable for code storagestorage
NAND FlashNAND FlashRandom access is not Random access is not supported, only sequentialsupported, only sequentialLower cost (than NOR)Lower cost (than NOR)Smaller size erase blocksSmaller size erase blocksBetter performance for Better performance for erase and writeerase and writeMore enduranceMore enduranceMainly for data storageMainly for data storage
Introduction to Flash MemoriesIntroduction to Flash Memories
DisadvantagesDisadvantages
Random access for rewrite or erase not Random access for rewrite or erase not supportedsupportedFinite number or erase/write cyclesFinite number or erase/write cyclesHigher cost per bit compared to hard disks Higher cost per bit compared to hard disks drivesdrives
Introduction to Flash MemoriesIntroduction to Flash Memories
Introduction to File SystemsIntroduction to File SystemsSubsytemSubsytem ofof an OS, an OS, whosewhose purposepurpose isis to to provideprovide longlong--termterm storagestorageSet of Set of abstract data typesabstract data types ADT (set of data and ADT (set of data and operations)operations)Storage hierarchical organization, manipulation, Storage hierarchical organization, manipulation, navigation, access, and retrieval of navigation, access, and retrieval of datadataMake use of an underlying Make use of an underlying data storage devicedata storage deviceAccess to an array of fixedAccess to an array of fixed--size size blocksblocks (sectors, (sectors, generally 512 bytes)generally 512 bytes)File system software is responsible for organizing these File system software is responsible for organizing these sectors into sectors into filesfiles and and directoriesdirectories, and keeping track of , and keeping track of which sectors belong to which file.which sectors belong to which file.
Types of File Systems Types of File Systems
Disk file systems Disk file systems Storage of Storage of filesfiles on a on a data storage device (disk drivedata storage device (disk drive).).ExpExp: : FATFAT, , NTFSNTFS, , HFSHFS, , ext2ext2, , ISO 9660ISO 9660, , ODSODS--55, and , and UDFUDF..
Database file systemsDatabase file systemsNon hierarchical structured managementNon hierarchical structured managementIdentification by their characteristics, like type of file, topiIdentification by their characteristics, like type of file, topic, c, author author ……. . metadatametadata
Introduction to File SystemsIntroduction to File Systems
PC File PC File SystemsSystemsFATFSFATFS File Allocation Table File System File Allocation Table File System
-- Developed by Developed by MicrosoftMicrosoft for for MSMS--DOSDOS-- SSupportedupported by virtually all existing by virtually all existing operating systemsoperating systems-- Main drawback Main drawback isis fragmentation.fragmentation.HPFSHPFS High Performance File System High Performance File System
-- EEfficientfficient placements of the allocation tables and root directories placements of the allocation tables and root directories -- FFilenamesilenames can be up to 254 chars longcan be up to 254 chars longNTFS New Technology File System NTFS New Technology File System -- SSeveraleveral improvements over FAT such as improved support for improvements over FAT such as improved support for metadatametadata-- AAdvanceddvanced data structures to improve performance, reliability and disk spdata structures to improve performance, reliability and disk space ace utilization utilization Ext2Ext2 second extended file systemsecond extended file system
-- Native Native filesystemfilesystem for Linux for Linux -- MMaximumaximum data size of 4 data size of 4 terabytesterabytes-- MMaximumaximum filename length of 255 charactersfilename length of 255 characters
Introduction to File SystemsIntroduction to File Systems
FATFAT’’ss File File SystemsSystems
FAT12FAT12 FAT16FAT16 FAT32FAT32
DeveloperDeveloper MicrosoftMicrosoft
File Allocation TableFile Allocation TableFull NameFull Name
(12(12--bit version)bit version) (16(16--bit version)bit version) (32(32--bit version)bit version)
IntroducedIntroduced1977 (Microsoft Disk BASIC)1977 (Microsoft Disk BASIC) July 1988 (MSJuly 1988 (MS--DOS 4.0)DOS 4.0) August 1996 (Win 95 OSR2)August 1996 (Win 95 OSR2)
0x0B, 0x0C (MBR)0x0B, 0x0C (MBR)
EBD0A0A2EBD0A0A2--B9E5B9E5--44334433
--87C087C0--68B6B72699C768B6B72699C7
Max file sizeMax file size 32 32 MiBMiB 2 2 GiBGiB 4 4 GiBGiB
Max number of filesMax number of files 4,0774,077 65,51765,517 268,435,437268,435,437
Max filename sizeMax filename size 8.3 or 255 characters when using 8.3 or 255 characters when using LFNsLFNs
2 2 GiBGiBMax volume sizeMax volume size 32 32 MiBMiB 4 4 GiBGiB with some with some
implementationsimplementations
2 2 TiBTiB
Partition identifierPartition identifier 0x01 (MBR)0x01 (MBR) 0x04, 0x06, 0x0E (MBR)0x04, 0x06, 0x0E (MBR)
Introduction to File SystemsIntroduction to File Systems
StorageStorage
Abstraction Abstraction givengiven by block by block devicedevice drivers.drivers.
LevelLevel ofof abstraction for abstraction for thethe HW HW responsibleresponsiblefor for storingstoring andand retrievingretrieving SpecifiedSpecified blocks blocks ofofdata.data.
Introduction to File SystemsIntroduction to File Systems
File ContentsFile ContentsIntroduction to File SystemsIntroduction to File Systems
Each file contents:Each file contents:File contents (data)File contents (data)File attributes (metadata)File attributes (metadata)
File sizeFile sizeOwner, access control listsOwner, access control listsCreation time, last access time, last modification Creation time, last access time, last modification time, time, ……
File nameFile name
Data Data ScatteringScattering
File system: A mapping problemFile system: A mapping problem
<filename, data, metadata> <filename, data, metadata> <a set of blocks><a set of blocks>
Introduction to File SystemsIntroduction to File Systems
File File SystemSystem ExampleExample
Ext2 file systemExt2 file system-- A diskA disk--based file system for Linuxbased file system for Linux-- Similar to UNIX Fast File System (FFS)Similar to UNIX Fast File System (FFS)-- Evolved to Ext3 File system (with journaling)Evolved to Ext3 File system (with journaling)-- Directory: pathname Directory: pathname metadata (imetadata (i--node)node)-- Direct/indirect block pointers: iDirect/indirect block pointers: i--node node data blocksdata blocks
Introduction to File SystemsIntroduction to File Systems
Unix File Unix File SystemSystem
Introduction to File SystemsIntroduction to File Systems
SuperblockSuperblockSize description Size description of the FSof the FS
InodeInodeMetadata, Metadata, pointerspointers
DatablockDatablockAllocation for Allocation for the FSthe FS
File Systems for Flash MemoriesFile Systems for Flash Memories
Due to the particular characteristics of Flash memories, Due to the particular characteristics of Flash memories, special file systems are needed to spread writes over the special file systems are needed to spread writes over the media media (wear(wear--levellinglevelling)) and deal with the long erase and deal with the long erase times of NOR flash blocks.times of NOR flash blocks.
Two approaches for FFS (Flash File System):Two approaches for FFS (Flash File System):Layered approachLayered approachNative (or crossNative (or cross--layer) approachlayer) approach
Layered ApproachLayered Approach
Uses the Fast Translation Layer Uses the Fast Translation Layer (FTL) to fully emulate a magnetic (FTL) to fully emulate a magnetic disk and includes:disk and includes:Sector mappingSector mappingGarbage collectionGarbage collectionPowerPower--off recoveryoff recoveryBad block managementBad block managementWearWear--levelinglevelingError correction code (ECC)Error correction code (ECC)Power managementPower management
File Systems for Flash MemoriesFile Systems for Flash Memories
BenefitsBenefitsEasy to deploy because no modification is required for upper Easy to deploy because no modification is required for upper layerslayersSome commercial Flash memories already come with FTL.Some commercial Flash memories already come with FTL.
LimitationsLimitationsMost Most FTLsFTLs are patentedare patentedKernel is not aware of the presence of flash memory.Kernel is not aware of the presence of flash memory.
Layered Approach (2)Layered Approach (2)File Systems for Flash MemoriesFile Systems for Flash Memories
Native ApproachNative Approach
CrossCross--layer optimizationlayer optimizationKernel manages raw flash memory directlyKernel manages raw flash memory directlyMore opportunities to optimize the performanceMore opportunities to optimize the performanceKernel is involved in some FTL functionalitiesKernel is involved in some FTL functionalities
Sector mapping, garbage collection, wearSector mapping, garbage collection, wear--leveling, powerleveling, power--off off recovery, etcrecovery, etc
Example:Example:FlashFlash--aware file systems: JFFS/JFFS2, YAFFSaware file systems: JFFS/JFFS2, YAFFS
LimitationsLimitationsNeed to change the host operating systemNeed to change the host operating systemOnly applicable FlashOnly applicable Flash--embedded devicesembedded devices
File Systems for Flash MemoriesFile Systems for Flash Memories
HistoryHistory
Toby Churchill Ltd (TCL) needed a flash filing Toby Churchill Ltd (TCL) needed a flash filing system for their devices reliable with fast boot system for their devices reliable with fast boot time. time. First option: adding NAND support to the First option: adding NAND support to the existing flash file systems, JFFS, JFFS2 existing flash file systems, JFFS, JFFS2
boot time and ram consumption were problemboot time and ram consumption were problemadding NAND support wasn't trivial adding NAND support wasn't trivial
The company Aleph One decided to design a The company Aleph One decided to design a different filing system explicitly for NAND different filing system explicitly for NAND
YAFFSYAFFS
..Progress..Progress
Decided to create YAFFS Decided to create YAFFS -- Dec 2001Dec 2001Working on RAM emulation Working on RAM emulation -- March 2002March 2002Working on real NAND (Linux) Working on real NAND (Linux) -- May 2002May 2002WinCE version WinCE version -- Aug 2002Aug 2002ucLinuxucLinux use use -- Sept 2002Sept 2002Linux Linux rootfsrootfs -- Nov 2002Nov 2002pSOSpSOS version version -- Feb 2003Feb 2003Shipping commercially Shipping commercially -- Early 2003Early 2003
YAFFS YAFFS -- HistoryHistory
Why NAND flash? Why NAND flash?
Embedded and mobile systems are increasingly using NAND Embedded and mobile systems are increasingly using NAND flash for storage because it has various advantages over other flash for storage because it has various advantages over other storage technologies storage technologies NOR flash is not very dense (i.e. not much storage per chip), isNOR flash is not very dense (i.e. not much storage per chip), iscostly and is slow to write costly and is slow to write NAND flash is low cost, dense, and writes fast; but it has otherNAND flash is low cost, dense, and writes fast; but it has otherlimitations limitations It is needed a journaling file system that is able to work arounIt is needed a journaling file system that is able to work around d the limitations of NAND and exploit it for best performance. the limitations of NAND and exploit it for best performance. Has a fast erase time as compared with NOR flash. Has a fast erase time as compared with NOR flash. The NAND physical interface is very simple The NAND physical interface is very simple The small size and low current requirements make it very The small size and low current requirements make it very suitable for embedded systems suitable for embedded systems
YAFFS YAFFS
Other NAND characteristics Other NAND characteristics
NAND is not random access, but page oriented. Thus, NAND is not random access, but page oriented. Thus, we do all reads & writes in pages we do all reads & writes in pages NAND writes will only change 1 bits to 0. The only NAND writes will only change 1 bits to 0. The only way to get 1s again is to erase the entire block. way to get 1s again is to erase the entire block. Errors are far more likely in NAND devices. Blocks of Errors are far more likely in NAND devices. Blocks of memory can be bad when the device is shipped and memory can be bad when the device is shipped and further blocks can become unusable over time. further blocks can become unusable over time. Therefore error detection and correction is highly Therefore error detection and correction is highly desirable. desirable. YAFFS handles bad blocks and uses formats which are YAFFS handles bad blocks and uses formats which are resistant to corruption.resistant to corruption.
YAFFSYAFFS
YAFFS Characteristics YAFFS Characteristics
The first NANDThe first NAND--specific flash file system specific flash file system Special reference to embedded systems Special reference to embedded systems
primarily for internal NAND rather than removable primarily for internal NAND rather than removable NAND (SM cards) NAND (SM cards) reliability is more important than compatibility. reliability is more important than compatibility.
Exploits lowExploits low--cost NAND chips and is both fast cost NAND chips and is both fast and robust and robust Highly portable Highly portable to other operating systemsto other operating systemsOpen source project Open source project
YAFFSYAFFS
It is design based in JFFS although NOR and NAND It is design based in JFFS although NOR and NAND flash have very different properties flash have very different properties
A flash file system that works with NOR incorporates various A flash file system that works with NOR incorporates various mechanisms that are not required for NAND, and NAND mechanisms that are not required for NAND, and NAND needs extra mechanisms not required for NOR needs extra mechanisms not required for NOR The design was greatly simplified by not including The design was greatly simplified by not including compression compression
Robustness through journaling strategies. Robustness through journaling strategies. Significantly reduce the RAM overheads and boot times Significantly reduce the RAM overheads and boot times associated with JFFS. associated with JFFS. YAFFS uses a physical flash format similar to YAFFS uses a physical flash format similar to SmartMediaSmartMedia. .
YAFFS Characteristics YAFFS Characteristics YAFFSYAFFS
File System DesignFile System Design
File data stored in chunks, same size as flash pages (512 File data stored in chunks, same size as flash pages (512 bytes)bytes)Block Block -- Erasable set of pages (usually 32)Erasable set of pages (usually 32)Each file has an id Each file has an id -- equivalent to equivalent to inodeinode. . Chunks numbered 1,2,3,4 etc Chunks numbered 1,2,3,4 etc -- 0 is header.0 is header.Each flash page is marked with file id and chunk Each flash page is marked with file id and chunk numbernumberThese tags are stored in the OOB These tags are stored in the OOB -- 64bits: including file 64bits: including file id, chunk number, write, serial number, tag ECC and id, chunk number, write, serial number, tag ECC and bytesbytes--inin--pagepage--usedused
YAFFSYAFFS
File System DesignFile System Design
On overwriting the relevant chunks are replaced by writing new On overwriting the relevant chunks are replaced by writing new pages with new data but same tags pages with new data but same tags -- the old page is marked the old page is marked ’’discardeddiscarded’’File headers (mode, id, length etc) get a page of their own (chuFile headers (mode, id, length etc) get a page of their own (chunk0)nk0)Pages also have a 2Pages also have a 2--bit serial number bit serial number -- incremented on writeincremented on write
Allows crashAllows crash--recovery when two pages have same tags (because old page recovery when two pages have same tags (because old page has not yet been marked has not yet been marked ’’discardeddiscarded’’).).
A block containing only discarded pages (termed a A block containing only discarded pages (termed a dirty blockdirty block) is an ) is an obvious candidate for garbage collection. obvious candidate for garbage collection. YAFFS maintains a tree structure in YAFFS maintains a tree structure in RAMRAM memory of the physical memory of the physical location of these chunks. location of these chunks.
YAFFSYAFFS
Page allocationPage allocation
Pages are allocated sequentially from the Pages are allocated sequentially from the currently selected block. When all the pages in currently selected block. When all the pages in the block are filled, another clean block is the block are filled, another clean block is selected for allocation. selected for allocation. If there are insufficient clean blocks available, If there are insufficient clean blocks available, then a dirty block is erased to free it up as a then a dirty block is erased to free it up as a clean block. If no dirty blocks are available, then clean block. If no dirty blocks are available, then the dirtiest block is selected for garbage the dirtiest block is selected for garbage collection. collection.
YAFFSYAFFS
Garbage collectionGarbage collection
Garbage collection is performed by copying the valid Garbage collection is performed by copying the valid data pages into new data pages thus marking all the data pages into new data pages thus marking all the pages in this block dirty and freeing it up for erasure. pages in this block dirty and freeing it up for erasure. Another idea: selecting a block at random some small Another idea: selecting a block at random some small percentage of the time percentage of the time -- thus reducing the chance of thus reducing the chance of wear differences. wear differences. Relative to NOR, NAND writes and erases very fast. Relative to NOR, NAND writes and erases very fast. Therefore garbage collection might be performed onTherefore garbage collection might be performed on--demand (during a write) demand (during a write)
YAFFSYAFFS
YAFFS 2 YAFFS 2
The original motivation for YAFFS 2 was to add support for the The original motivation for YAFFS 2 was to add support for the new NAND with 2kB pages instead of 512new NAND with 2kB pages instead of 512--byte pages and strictly byte pages and strictly sequential page writing order.sequential page writing order.To achieve this, a new design is used which also presents the To achieve this, a new design is used which also presents the following benefits:following benefits:
zero page rewrites means faster operation. (YAFFS1 uses a zero page rewrites means faster operation. (YAFFS1 uses a single rewrite in the spare area to delete a page).single rewrite in the spare area to delete a page).ability to exploit simultaneous page programming on some ability to exploit simultaneous page programming on some chips.chips.improves performance relative to YAFFS1 speed(write:1.5x to improves performance relative to YAFFS1 speed(write:1.5x to 5x, delete: 4x, garbage collection: 2x)5x, delete: 4x, garbage collection: 2x)lower RAM footprint (approx. 25% to 50% of YAFFS1).lower RAM footprint (approx. 25% to 50% of YAFFS1).
The main philosophical difference between YAFFS and YAFFS2 The main philosophical difference between YAFFS and YAFFS2 is how discarded status is tracked. Since reis how discarded status is tracked. Since re--writing cannot be done, writing cannot be done, "discarded" flags are not used. Now the strategy is including mo"discarded" flags are not used. Now the strategy is including more re tag information.tag information.
YAFFSYAFFS
File System LimitsFile System Limits
YAFFSYAFFS2^18 files (>260,000)2^18 files (>260,000)2^20 max file size (512MB)2^20 max file size (512MB)1GB max file system size1GB max file system size
YAFFS2YAFFS28GB max file system size8GB max file system size
YAFFSYAFFS
ReferencesReferences
www.wikipedia.comwww.wikipedia.comwww.aleph1.co.ukwww.aleph1.co.ukhttp://labs.google.com/papers/gfshttp://labs.google.com/papers/gfs--sosp2003.pdfsosp2003.pdfhttp://www.yale.edu/pclt/BOOT/IFS.HTMhttp://www.yale.edu/pclt/BOOT/IFS.HTMhttp://learnlinux.tsf.org.za/courses/build/internals/ch08s04.htmhttp://learnlinux.tsf.org.za/courses/build/internals/ch08s04.htmllhttp://hjem.get2net.dk/rune_moeller_barnkob/filesystems/http://hjem.get2net.dk/rune_moeller_barnkob/filesystems/http://linux.org.mt/article/filesystemshttp://linux.org.mt/article/filesystems