introduction to cache quality of service in linux kernel...introduction to cache quality of service...
TRANSCRIPT
![Page 2: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/2.jpg)
Agenda
• Problem definition • Existing techniques
• Why use Kernel QOS framework
• Intel Cache qos support
• Kernel implementation
• Challenges
• Performance improvement
• Future Work
2
![Page 3: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/3.jpg)
Without Cache QoS
High Pri apps
Low Pri apps
C2
Low pri apps may get more cache
Shared Processor Cache
C1 C3 Cores Cores
- Noisy neighbour => Degrade/inconsistency in response => QOS difficulties
- Cache Contention with multi threading 3
![Page 4: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/4.jpg)
Agenda
• Problem definition
• Existing techniques • Why use Kernel QOS framework
• Intel Cache qos support
• Kernel implementation
• Challenges
• Performance improvement
• Future Work 4
![Page 5: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/5.jpg)
Existing techniques
• Mostly heuristics on real systems
• No methodology to identify cache lines belonging to a particular thread
• Lacks configurability by OS
5
![Page 6: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/6.jpg)
Agenda
• Problem definition
• Existing techniques
• Why use Kernel QOS framework • Intel Cache qos support
• Kernel implementation
• Challenges
• Performance improvement
• Future Work 6
![Page 7: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/7.jpg)
Why use the QOS framework?
Threads
Architectural details of ID management/scheduling
• Lightweight powerful tool to manage cache
• Without a lot of architectural details
7
![Page 8: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/8.jpg)
With Cache QoS
- Help maximize performance and meet QoS requirements - In Cloud or Server Clusters - Mitigate jitter/inconsistent response times due to ‘Noisy neighbour’
High Pri apps
Low Pri apps
Kernel Cache QOS framework
Intel QOS h/w support
Controls to allocate the appropriate cache to
high pri apps
Proc Cache
User space
Kernel space
h/w
8
![Page 9: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/9.jpg)
Agenda
• Problem definition
• Existing techniques
• Why use Kernel QoS framework
• Intel Cache QoS support • Kernel implementation
• Challenges
• Performance improvement
• Future Work 9
![Page 10: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/10.jpg)
What is Cache QoS ?
• Cache Monitoring
– cache occupancy per thread
– perf interface
• Cache Allocation
– user can allocate overlapping subsets of cache to applications
– cgroup interface
10
![Page 11: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/11.jpg)
Cache lines Thread ID (Identification)
• Cache Monitoring
– RMID (Resource Monitoring ID)
• Cache Allocation
– CLOSid (Class of service ID)
11
![Page 12: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/12.jpg)
Representing cache capacity in Cache Allocation(example)
Bn B1 B0
Wk
W(k
-1)
W3 W2 W1 W0
Capacity Bitmask
Cache Ways
- Cache capacity represented using ‘Cache bitmask’ - However mappings are hardware implementation specific
12
![Page 13: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/13.jpg)
Bitmask Class of service IDs (CLOS)
B7 B6 B5 B4 B3 B2 B1 B0
CLOS0 A A A A A A A A
CLOS1 A A A A A A A A
CLOS2 A A A A A A A A
CLOS3 A A A A A A A A
B7 B6 B5 B4 B3 B2 B1 B0
CLOS0 A A A A A A A A
CLOS1 A A A A
CLOS2 A A
CLOS3 A A
Default Bitmask – All CLOS ids have all cache
Overlapping Bitmask (only contiguous bits)
13
![Page 14: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/14.jpg)
Agenda
• Problem definition
• Existing techniques
• Why use Kernel QOS framework
• Intel Cache qos support
• Kernel implementation • Challenges
• Performance improvement
• Future Work 14
![Page 15: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/15.jpg)
Kernel Implementation
Threads
Cgroup fs
/sys/fs/cgroup perf
User interface
Cache alloc cache
monitoring
Kernel QOS support
Intel Xeon QOS support
Shared L3 Cache
User Space
Kernel Space
Hardware
MSR Configure
bitmask per CLOS
Set CLOS/RMID for thread
During ctx switch
Allocation configuration
Read Event
counter
Read Monitored data
15
![Page 16: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/16.jpg)
Usage Monitoring per thread cache occupancy in bytes
Allocating Cache per thread through cache bitmask
16
Cgroup Clos : Parent.Closbitmask : Parent.bitmaskTasks : Empty
Exposed to user land
![Page 17: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/17.jpg)
Scenarios
• Units that can be allocated cache – Process/tasks
– Virtual machines (transfer all PIDs of VM to one cgroup)
– Containers (put the entire container into one cgroup)
• Restrict the noisy neighbour
• Fair cache allocation to resolve cache contention
17
![Page 18: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/18.jpg)
Agenda
• Problem definition
• Existing techniques
• Why use Kernel QOS framework
• Intel Cache qos support
• Kernel implementation
• Challenges • Performance improvement
• Future Work 18
![Page 19: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/19.jpg)
Challenges
• Openstack usage
• What if we run out of IDs ?
• What about Scheduling overhead
• Doing monitoring and allocation together
19
![Page 20: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/20.jpg)
Openstack usage
Applications
Openstack dashboard
Open Stack Services
Standard hardware
Shared L3 Cache Shared L3 Cache
Integration WIP Compute Network Storage
20
![Page 21: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/21.jpg)
Openstack usage …
Perf syscall
OpenStack
libvirt
Virt mgr ovirt . . .
KVM Xen . . . Kernel Cache QOS
Work beginning, not stable yet to add changes to Ceilometer (With Qiaowei [email protected] )
21
![Page 22: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/22.jpg)
What if we run out of IDs ?
• Group tasks together (by process?)
• Group cgroups together with same mask
• return –ENOSPC
• Postpone
22
![Page 23: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/23.jpg)
Scheduling performance
• msrread/write costs 250-300 cycles
• Keep a cache. Grouping helps !
• Don’t use till user actually creates a new cache mask
23
![Page 24: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/24.jpg)
Monitor and Allocate
• RMID(Monitoring) CLOSid(allocation) different
• Monitoring and allocate same set of tasks easily
– perf cannot monitor the cache alloc cgroup(?)
24
![Page 25: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/25.jpg)
Agenda
• Problem definition
• Existing techniques
• Why use Kernel QOS framework
• Intel Cache qos support
• Kernel implementation
• Challenges
• Performance improvement and Future Work
25
![Page 26: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/26.jpg)
Performance Measurement
• Intel Xeon based server, 16GB RAM • 30MB L3 , 24 LPs • RHEL 6.3 • With and without cache allocation comparison • Controlled experiment
– PCIe generating MSI interrupt and measure time for response
– Also run memory traffic generating workloads (noisy neighbour)
• Experiment Not using current cache alloc patch
26
![Page 27: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/27.jpg)
Performance Measurement[1]
2.8x
1.5x
1.3x
- Minimum latency : 1.3x improvement , Max latency : 1.5x improvement , Avg latency : 2.8x improvement
- Better consistency in response times and less jitter and latency with the noisy neighbour
27
![Page 28: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/28.jpg)
Patch status
Cache Monitoring Upstream 4.1 (Matt Fleming ,[email protected])
Cache Allocation Under review. (Vikas Shivappa , [email protected])
Code Data prioritization Under review. (Vikas Shivappa , [email protected])
Open stack integration (libvirt update)
Work started (Qiaowei [email protected])
28
![Page 29: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/29.jpg)
Future Work
• Performance improvement measurement
• Code and data allocation separately
– First patches shared on lkml
• Monitor and allocate same unit
• Openstack integration
• Container usage
29
![Page 30: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/30.jpg)
Acknowledgements
• Matt Fleming (cache monitoring support, Intel SSG)
• Will Auld (Architect and Principal engineer, Intel SSG)
• CSIG, Intel
30
![Page 31: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/31.jpg)
References
• [1] http://www.intel.com/content/www/us/en/communications/cache-allocation-technology-white-paper.html
31
![Page 32: Introduction to Cache Quality of service in Linux Kernel...Introduction to Cache Quality of service in Linux Kernel Vikas Shivappa (vikas.shivappa@linux.intel.com) 1](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bab334bf17857056aab94/html5/thumbnails/32.jpg)
Questions ?
32