programming emerging storage interfaces€¦ · abstract handle to your controller / namespace...
TRANSCRIPT
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 1
Programming Emerging Storage Interfaces
Simon A. F. LundSamsung / SSDR
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 2
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 3
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 4
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 5
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 6
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 7
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 8
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices User Space tools and libraries
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 9
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices User Space tools and libraries Example library and tool usage
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 10
Host ResponsibilitiesFrom Open-Channel to ZNS
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 11
Host Responsibilities
Device media represented as Physical blocks (OCSSD 1.2) Chunks (OCSSD 2.0 + Denali) Zones (Zoned Namespaces)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 12
Host Responsibilities
Device media represented as Physical blocks (OCSSD 1.2) Chunks (OCSSD 2.0 + Denali) Zones (Zoned Namespaces)
Nomenclature: zone
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 13
Host Responsibilities: Zones
Zone Layout
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 14
Host Responsibilities: Zones
Zone Layout Zone Attributes and Condition
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 15
Host Responsibilities: Zones
Zone Layout Zone Attributes and Condition Zone Constraints
1. Write contiguously (within a zone)2. Reset before write (again)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 16
Host Responsibilities: Zone Layout
Block Storage How many LBAs Size of an LBA
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 17
Host Responsibilities: Zone Layout
Block Storage How many LBAs Size of an LBA
Zoned Block Storage How many Zones Attributes and Condition
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 18
Host Responsibilities: Zone Attributes
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 19
Host Responsibilities: Zone Attributes
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 20
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 21
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 22
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 23
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 24
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 25
Host Responsibilities: Zone Write
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 26
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 27
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 28
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 29
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 30
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 31
Host Responsibilities: Zone Write QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 32
Host Responsibilities: Zone Write QD > 1
Synchronize:Wait for a write to WP finish before writing to WP + k
Submission Order != Execution / Completion Order
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 33
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 34
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 35
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 36
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 37
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 38
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 39
Host Responsibilities: Zone Append
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 40
Host Responsibilities: Append QD > 1
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 41
Host Responsibilities: Append QD > 1Where is my data?
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 42
Host Responsibilities: Append QD > 1Where is my data?Read the completion entry for location
Response (cq-entry)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 43
Host Responsibilities: Zone Reset
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 44
Host Responsibilities: Zone Reset
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 45
Host Responsibilities
How to manage this as storage developer? You need to get retrieve information on Zone Layout Zone Attributes and Condition
Read, Write, Append, and Reset commands
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 46
User Space tools and libraries
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 47
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 48
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file System Calls ioctl(), read(), write(), pread(), pwrite(), etc. aio_read(), aio_write() io_uring_prep_readv / io_uring_submit
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 49
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file System Calls ioctl(), read(), write(), pread(), pwrite(), etc. aio_read(), aio_write() io_uring_prep_readv / io_uring_submit
Wrapped in libraries libc, libaio, liburing
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 50
Open-Source Ecosystem: OS Support
Pros: General block storage infrastructure Efficient async R/W with io_uring / liburing
Cons: Syncronous ioctl() interface Limited control over command construction
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 51
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque*
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 52
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque* Driver in User Space Intel SPDK libnvme (SPDK without DPDK)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 53
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque* Driver in User Space Intel SPDK libnvme (SPDK without DPDK)
Driver in Kernel access from User Space NVMe-Direct
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 54
Open-Source Ecosystem: OS Bypass
Pros: Full control over command construction Efficient async interface for ANY command
Cons: Controller detachment from kernel Non-trivial controller sharing
Demo detach
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 55
Open-Source Ecosystem: tools
nvme-cli Built on Linux ioctl() Limited port for FreeBSD ioctl() Port built on SPDK
nvmecontrol FreeBSD base system, built on ioctl()
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 56
Virtual NVMe DevicesSetup, usage, and modification with
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 57
QEMU: The Quick Emulator
A generic machine emulator and virtualizer ia32, x86_64, mips, sparc, arm, risc-v KVM-client e.g. using Intel VT-x
Includes a huge collection of emulated devices
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 58
QEMU: The Quick Emulator
A generic machine emulator and virtualizer ia32, x86_64, mips, sparc, arm, risc-v KVM-client e.g. using Intel VT-x
Includes a huge collection of emulated devices Active community ~130 subsys maintainers ~1500 individual contributors)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 59
QEMU: Contributions
Upstream contributions (Klaus A. B. Jensen) Full NVMe 1.3 support Ongoing NVMe 1.4 support Full ZNS support Upcoming TPs
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 60
QEMU: Contributions
Upstream contributions (Klaus A. B. Jensen) Full NVMe 1.3 support Ongoing NVMe 1.4 support Full ZNS support Upcoming TPs
Extending the QEMU NVMe drive model
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 61
QEMU: NVMe Device Model
QEMU/hw/block/nvme.c
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 62
QEMU: NVMe Device Model Usage
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 63
QEMU: NVMe Usage Hands On!
QEMU configure and build:
QENV setup environment config machine config device config run and access
mkdir buildcd build../configure--target-list=x86_64-softmmu
make -j $(nproc)
QEMU build demo link
QENV setup demo link
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 65
User Space tools and libraries
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 66
Open-Source Ecosystem: an overview
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 67
Open-Source Ecosystem: an overview
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 68
Open-Source Ecosystem: an overview
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 69
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 70
xNVMeCross-platform libraries and tools for NVMe devices
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 71
xNVMe: API: xnvme_dev
Abstract handle to your controller / namespace
Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES, ) Reduce overhead of kernel retrieving handles for each IO
SPDK Backend Initialize and attach to controller spdk_env_opts_init() / spdk_env_init() / spdk_nvme_probe()
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 72
xNVMe: API: xnvme_buf
Allocate and free memory for use by the xnvme_cmd interface
Linux Backend Pagesize aligned for ioctl() posix_memalign() / free() io_uring_register(IORING_REGISTER_BUFFERS, ...)
SPDK Backend Allocate physical memory / DMA transferable spdk_dma_{malloc,realloc,free}()
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 73
xNVMe: API: xnvme_async
Context for asynchronous / non-blocking xnvme_cmd interface
Linux Backend Threadpool allocation for pseudo-async behavior via ioctl() SQ / CQ setup for io_uring
SPDK Backend NVMe QP setup spdk_nvme_ctrlr_{alloc,free}_io_qpair()
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 74
xNVMe: API: xnvme_cmd
Synchronous and asynchronous command interface
Linux Backend io_uring_{submit, peek , wait} Jobs to ioctl() threadpool
SPDK Backend spdk_nvme_ctrlr_cmd_{admin_raw, io_raw_with_md}() spdk_nvme_qpair_process_completions()
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 75
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 76
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 77
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 78
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 79
Usage of `zoned`Demo link
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 80
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 81
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 82
Usage of `ZROFS`Demo link
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 83
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 84
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 85
Usage of `xnvme`Demo link
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 86
Open-Source Ecosystem: a contribution
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 87
xNVMe
Building Open-Source ecosystem
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 88
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 89
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 90
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools Libraries
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 91
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools Libraries VALUE
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 922019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved.
Thanks
WWW https://xnvme.ioMAIL [email protected]
www.linkedin.com/in/simonlund