device-driven i/o for implicit paging...
TRANSCRIPT
![Page 1: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/1.jpg)
Device-driven I/O for Implicit Paging Operations
Jeremy Kerr <[email protected]>
IBM Linux Technology Center
![Page 2: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/2.jpg)
Outline
• K42 overview• The K42 I/O system• The device-driven model• Implementation & experimentation• Further work
![Page 3: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/3.jpg)
K42 Overview
• Research OS– Intended for easy experimentation
• Microkernel-style architecture• Fast user/kernel-space message passing system
– Protected Procedure Call (PPC)– Stub compiler to easily define new calls
• Currently working on 64-bit POWER machines• Compatible with Linux API & ABI
![Page 4: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/4.jpg)
How it works now...
• K42 I/O subsystem– File objects are kept in userspace– write() is performed asynchronously
• essentially a memcpy()
• Walk through a fsync...
![Page 5: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/5.jpg)
Current fsync Path
File Cache Manager receives fsync()
![Page 6: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/6.jpg)
Current fsync Path
File Representative manages reference to userspace file
![Page 7: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/7.jpg)
Current fsync Path
KernelPagingTransport passes I/O request to userspace...
![Page 8: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/8.jpg)
Current fsync Path
...over a shared ring-buffer, to the PagingTransport object
![Page 9: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/9.jpg)
Current fsync Path
ServerFile object handles write operation
![Page 10: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/10.jpg)
I/O Request Structure
• Basic unit of I/O• Shared with userspace• fileToken member
– reference to ServerFile– opaque within kernel
![Page 11: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/11.jpg)
K42 I/O Subsystem
• In summary:– FCM (File Cache Manager)
• Manages mapped regions– FR (File Representative)
• Holds references to userspace file objects– ServerFile
• Userspace file object– I/O performed by sending requests to userspace
• Requests are “pushed” to the device
![Page 12: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/12.jpg)
Device-driven I/O
• Goal: Maximise usage of device bandwidth– Perform implicit operations when device is idle
• New programming model for I/O operations– FCMs no longer asynchronously generate implicit
requests– Instead, FCMs are queried for requests
• “pull model”
![Page 13: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/13.jpg)
Current I/O Activity
![Page 14: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/14.jpg)
Desired I/O Activity
![Page 15: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/15.jpg)
Rationale
• Allows aggressive prefetch & page cleaning– Should stabilise bursty I/O– No need to be conservative
• FCMs “know best” how to optimise implicit ops• Low request latency
– Requests are only generated when the device is ready– No need for a cancellation mechanism
![Page 16: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/16.jpg)
Design
• Keep track of pageable FCMs– New class: PagingService
• Generate requests– New method: FCM::getRequests()
• Notify the transport that bandwidth is available– New PPC: KernelPagingTransport::notify()
![Page 17: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/17.jpg)
Device-driven Design
![Page 18: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/18.jpg)
Device-driven Design
![Page 19: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/19.jpg)
The PagingService
• Clustered object– One root object created at boot time– One “representative” created per processor
• FCMs register with the PagingService on creation– Only pageable FCMs– De-register on destruction
• Referenced by the KernelPagingTransport
![Page 20: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/20.jpg)
Device-driven Design
![Page 21: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/21.jpg)
FCM::getRequests()
• Method to retrieve a set of I/O requests– Up to a specified maximum
• Implicit request pattern controlled by FCM– Can be based on history, expected patterns
• Prefetch optimisation• Resident-set research at UofT
– Easy to explore new ideas• FCM inheritance
– Dynamic upgrade
![Page 22: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/22.jpg)
Device-driven Design
![Page 23: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/23.jpg)
Device-driven Design
![Page 24: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/24.jpg)
Device-driven Design
![Page 25: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/25.jpg)
Design: Current Model
![Page 26: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/26.jpg)
Flow Control
• Feedback loop provides flow control– Restricted by ring-buffer availability– Representative of device usage
![Page 27: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/27.jpg)
Initial Concerns
• Global optimisation of implicit operations– No longer possible?
• Resource usage by PagingService– Don't want to tie up other resources
• Latency introduced for explicit operations
![Page 28: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/28.jpg)
Tunables
• Implementation of FCM::getRequests()– Prefetch optimisation, page cleaning patterns– A whole area of research in itself...
• Allocation of bandwidth to FCMs– Which to query for requests?– Priority between FCMs
![Page 29: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/29.jpg)
The Implementation lib/libc/fslib/PagingTransport.C | 13 +- lib/libc/fslib/PagingTransport.H | 11 + lib/libc/io/FileLinux.C | 8 - os/kernel/ObjectRefsKern.H | 4 os/kernel/bilge/CObjGlobalsKern.H | 8 - os/kernel/init/KernelInit.C | 4 os/kernel/mem/FCM.C | 6 + os/kernel/mem/FCM.H | 2 os/kernel/mem/FCMDefaultMultiRep.C | 4 os/kernel/mem/FCMFile.C | 55 ++++++++- os/kernel/mem/FCMFile.H | 6 + os/kernel/mem/FR.H | 11 + os/kernel/mem/FRCommon.H | 6 + os/kernel/mem/FRPA.C | 30 +++++ os/kernel/mem/FRPA.H | 3 os/kernel/mem/KernelPagingTransport.C | 60 +++++++++- os/kernel/mem/KernelPagingTransport.H | 16 +- os/kernel/mem/Makefile | 7 - os/kernel/mem/PagingService.C | 204 ++++++++++++++++++++++++++++++ os/kernel/mem/PagingService.H | 166 +++++++++++++++++++++++++++ 20 files changed, 585 insertions(+), 39 deletions(-)
![Page 30: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/30.jpg)
Experimentation
• Initial testing– Copy between filesystems– Time taken to sync changes to disk
• Implementing a prefetch oracle– Advance knowledge of pattern
• Alter prefetch oracle– Use history to determine pattern
![Page 31: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/31.jpg)
Desired I/O Activity
![Page 32: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/32.jpg)
Actual I/O Activity
![Page 33: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/33.jpg)
Current Issues
• Current implementation is very aggressive– PagingService blocks other operations while paging
• sync has no pages to flush
• High use of the PPC mechanism– Potential bottleneck
![Page 34: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/34.jpg)
Further Work
• Tune scheduler to allow true background paging• Improve FCM-request allocation• Implementation of prefetch/cleaning algorithm
– FCM::getRequests()
![Page 35: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/35.jpg)
Resources
• K42 website at IBM Research– http://www.research.ibm.com/K42/
• Patches– http://ozlabs.org/~jk/projects/k42/
![Page 36: Device-driven I/O for Implicit Paging Operationsjk.ozlabs.org/projects/k42/device-driven-IO-lca06.pdfDevice-driven I/O for Implicit Paging Operations ... • Microkernel-style architecture](https://reader035.vdocument.in/reader035/viewer/2022062506/5f01e6c57e708231d4019819/html5/thumbnails/36.jpg)
Legal
• This work represents the view of the authors and does not necessarily represent the view of IBM.
• Linux is a registered trademark of Linus Torvalds.• Other company, product, and service names may
be trademarks or service marks of others.