ilkka ollakka cern library thin client system...
TRANSCRIPT
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
Ilkka Ollakka
CERN LIBRARY THIN CLIENT SYSTEMMODERNIZATION
Master’s ThesisDegree Programme in Computer Science and Engineering
April 2013
Ollakka I. (2013) Cern library Thin Client System modernization. University ofOulu, Department of Computer Science and Engineering, Master’s Thesis, 59 pages 2appendices.
ABSTRACT
The goal of this Master’s thesis is to modernize the CERN library public termi-nal system. With modernization the aim was set to minimize the administrativeburden on library staff from the new system. Modernization focuses on replacingthe currently used Linux based thin client system that suffers from hardware age-ing with a system that resides in the CERN-IT operated private cloud. Anotheraspect in modernization was to utilize the provided network infrastructure moreto simplify system complexity and the amount of remote components that requirelibrary staff maintenance and administration.
The new system differentiates from a traditional LTSP-system by dividing ev-ery thin client into a different virtual machine. By distributing every thin clientin our own virtual host, we can distribute resource allocation, like CPU-timeand physical memory, between different physical hosts. The system also containsmonitoring features that informs administrative problems that have been noted.Monitoring decreases the time required for dealing with system maintenance.
Linux-based thin client remote connection efficiencies are compared using UI-latency and bandwidth efficiency as metrics. A plain X11 connection is measuredagainst an NX-connection and an SSH-tunneled X11 connection. Results showNX bandwidth efficiency that comes from utilizing an extra caching layer. Mea-surements for overall latency and bandwidth usage are presented.
Keywords: virtualization, Linux, remote desktop, X11
Ollakka I. (2013) CERN kirjaston asiakaspäätteiden modernisointi. Oulun yli-opisto, Tietotekniikan osasto, Diplomityö, 59 sivua 2 liitettä.
TIIVISTELMÄ
Tässä diplomityössä modernisoidaan CERN:n kirjaston käytössä olevat asia-kaspäätteet. Modernisoinnin tarkoituksena on vähentää asiakaspäätteistä mah-dollisesti aiheutuvaa ylläpitokuormaa kirjaston henkilökuntaa kohtaan. Pääsuun-ta modernisaatiossa on rakentaa uusi asiakaspäätejärjestelmä vanhan LTSP jär-jestelmän tilalle, joka kärsii palvelinraudan vanhenemisesta, hyödyntäen CERN-IT:n operoimaa sisäistä pilvipalvelua sekä jo olemassa olevaa verkkoinfrastruk-tuuria.
Rakennettu järjestelmä poikkeaa perinteisestä LTSP-järjestelmästä määritte-lemällä jokaiselle päätteelle erillisen virtuaalikoneen johon kytkeytyä. Erillisil-lä virtuaalikoneilla saadaan käyttäjien varaamia resursseja kuten CPU-aikaa jafyysistä muistia hajautettua useammalle fyysiselle palvelimelle. Järjestelmä sisäl-tää myös proaktiivisen monitoroinnin joka ilmoittaa havaituista ongelmista yllä-pitovastuussa olevia henkilöitä eikä näin vaadi järjestelmän aktiivista tarkkailuaylläpitovastuussa olevilta henkilöiltä.
Linux-pohjaisten tyhmien päätteiden etäyhteys-protokollien tehokkuutta ver-taillaan käyttämällä käyttöliittymän latenssia ja kaistankulutusta mittareina. Nor-maalia X11 yhteyttä verrataan yleisesti käytettyyn SSH-tunneloituun X11 yhtey-teen sekä NX-yhteyteen. Tulokset osoittavat NX-yhteyden ylimääräisen välimuis-tin tehostavan kaistankäyttöä. Mittausjärjestelyt ja tulokset on esitelty.
Avainsanat: Tyhmät päätteet, virtualisointi, Linux, X11
TABLE OF CONTENTS
ABSTRACT
TIIVISTELMÄ
TABLE OF CONTENTS
PREFACE
LIST OF SYMBOLS AND ABBREVIATIONS
1. INTRODUCTION 9
2. VIRTUALIZATION 112.1. Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2. Hypervisor types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3. X86-architecture virtualization . . . . . . . . . . . . . . . . . . . . . 14
2.3.1. Binary translation . . . . . . . . . . . . . . . . . . . . . . . . 142.3.2. Paravirtualized CPU . . . . . . . . . . . . . . . . . . . . . . 152.3.3. Memory management . . . . . . . . . . . . . . . . . . . . . 152.3.4. X86 64-bit virtualization . . . . . . . . . . . . . . . . . . . . 172.3.5. Hardware assisted virtualization . . . . . . . . . . . . . . . . 18
2.4. Paravirtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.5. KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.6. HyperV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.7. Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.8. Cloud services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3. REMOTE CONNECTION PROTOCOLS 243.1. User interface latency limits . . . . . . . . . . . . . . . . . . . . . . 243.2. X11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.3. X11 tunneled inside SSH . . . . . . . . . . . . . . . . . . . . . . . . 263.4. NoMachine NX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.5. Spice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4. CERN NETWORK INFRASTRUCTURE 314.1. Network boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.2. Automated Installations Management Systems . . . . . . . . . . . . . 33
5. CURRENT SYSTEM 345.1. History of public terminals in CERN library . . . . . . . . . . . . . . 345.2. Requirements for public terminals in library . . . . . . . . . . . . . . 355.3. Thin client concept . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.4. Current implementation overview . . . . . . . . . . . . . . . . . . . 365.5. Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6. PROPOSED SOLUTION 386.1. Thin client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.2. Virtualization platform . . . . . . . . . . . . . . . . . . . . . . . . . 416.3. Desktop OS, Scientific Linux CERN 6 . . . . . . . . . . . . . . . . . 416.4. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.5. Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.6. Public Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7. LATENCY AND BANDWIDTH TESTING 45
8. REMOTE CONNECTION PROTOCOL TEST RESULTS 478.1. Latency test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478.2. Bandwidth usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9. DISCUSSION 529.1. Regressions in proposed system compared to current system . . . . . 539.2. Migration from current system to proposed solution . . . . . . . . . . 539.3. Test results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
10. CONCLUSIONS 54
BIBLIOGRAPHY 55
APPENDIX 59
List of Figures
1 High level architecture of thin client setup, where the virtual instanceresides in the cloud service. . . . . . . . . . . . . . . . . . . . . . . . 10
2 Virtual machine layer 0 is run in the host’s layer 2. . . . . . . . . . . 123 Type 1 hypervisor runs directly on top of hardware. . . . . . . . . . . 134 Type 2 hypervisor runs on top of OS. . . . . . . . . . . . . . . . . . . 135 Binary translator intercepts guest program counter. . . . . . . . . . . 156 Shadow page tables in virtualization. . . . . . . . . . . . . . . . . . . 167 VMM code location in memory segments. . . . . . . . . . . . . . . . 168 Ballooning driver inside guest OS. . . . . . . . . . . . . . . . . . . . 179 KVM high-level architecture. . . . . . . . . . . . . . . . . . . . . . . 1910 KVM guest execution loop. . . . . . . . . . . . . . . . . . . . . . . . 2011 HyperV-architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . 2112 Distinction between cloud service types. . . . . . . . . . . . . . . . . 23
13 X11 client/server separation. . . . . . . . . . . . . . . . . . . . . . . 2514 X11 traffic goes inside SSH connection from server to client. . . . . . 2715 NX architecture overview. . . . . . . . . . . . . . . . . . . . . . . . 2816 Spice architecture paired with QEMU. . . . . . . . . . . . . . . . . . 29
17 Network booting sequence in CERN network. . . . . . . . . . . . . . 32
18 Current system overview in CERN library. . . . . . . . . . . . . . . . 36
19 Overview of proposed public terminal setup. . . . . . . . . . . . . . . 3820 Thin client boot-up sequence. . . . . . . . . . . . . . . . . . . . . . . 40
21 Measuring setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
22 Screen update latency on local display during test run. . . . . . . . . . 4823 Screen update latency on plain X11 display during test run. . . . . . . 4824 Screen update latency on SSH tunneled display during test run. . . . . 4925 Screen update latency on NX-connection during test run. . . . . . . . 5026 Bandwidth in kbits/second in plain X11 display during testrun. . . . . 5027 Bandwidth in kbits/second in SSH tunneled display during testrun. . . 5128 Bandwidth in kbits/second in NX connection during testrun. . . . . . 51
PREFACE
This Master’s thesis is a result of my four-month technical student period at CERNduring summer 2011. CERN had the academic freedom to work with all the up anddown sides that comes with that. My work was performed as part of a joint projectbetween HIP-TEK and CERN library.
I would like to thank Jukka Kommeri and all the HIP-TEK researchers who mademe feel welcome and provided me with ideas during the project. Special thanks arealso due to Jukka for giving me so much time to provide comments on my writingand his overall guidance on the project. All the ’hippies’ and ’mummo’ also deserve aspecial mention for being part of that special summer. Thanks also to Mark for readingthis thesis and providing me with his useful observations.
Thanks to CERN Library staff for accommodatinge me in one of their offices duringthe summer and with whom I had interesting conversations about variety of things overlunch. In addition, the always helpful CERN-IT team sped up the project providedgave excellent help during the implementation phase.
Special thanks to Professor Röning, for providing supervising on the thesis, posingexcellent questions and making comments that helped me to the clarify content andreally think about what to write.
Last but not least, thanks to my family, particularly Jaana, who supported and helpedme during the writing process a great deal.
Ilkka Ollakka
LIST OF SYMBOLS AND ABBREVIATIONS
CERN The European Organization for Nuclear ResearchDaaS Desktop as an Service. The desktop runs inside a cloud
service and is used with a remote connection methodGuest mode Running mode for a virtual host, where hardware keeps
track of CPU states and allows for the running of privilegedcode without interfering with the hypervisor
Hyper-V Microsoft virtualization platformHypervisor Synonym for VMM in this thesisIaaS Infrastructure as a Service. The cloud service level provides
everything up to the operating system levelKernel mode Code executed in layer 0 on a computer systemKVM Kernel-based Virtual Machine, Linux virtualization solu-
tions for x86 hardwareLTSP Linux Terminal Server Project, thin client system built on
top of Linux and opensource softwareMMU Memory management unitNBD Network Block Device. Transfers raw disk blocks over
TCP/IPNested paging Hardware virtualized MMU, where a hypervisor controls
memory mapping with a nested page tableNX NoMachine X-protocol compressor for reduced bandwidth
and latency for X-connectionsPaaS Platform as a Service. A cloud service level that provides
everything up to the runtime environment level. The Tomcatservice platform is an example of this
PXE Preboot Execution EnvironmentSaaS Software as a Service, cloud service level where everything
is handled by a vendor and is only used via a user interfacewithout the need to access management
Shadow page table Virtualized software controlled mapping for virtual host ad-dress to physical address
Spice Simple Protocol for Independent Computing Environments.Spice is an open remote computing solution, providingclient access to remove machine display and devices
SLC Scientific Linux CERN, CERN’s own Linux distributionthat the use and support
TLB Translation Lookaside Buffer, memory mapping cache forMMU
tmpfs Common name for file system residing in volatile memoryinstead of persistent storage
User mode All other code execution that is not in Kernel modeVDI Virtual Desktop Infrastructure, another term for thin client
infrastructureVDI Virtual Device Interface, Spice name for paravirtual driver
interfaceVMCB Virtual Machine Control Block, contains control informa-
tion of virtual machine and state of CPU and execution
VMM Virtual Machine Monitor. Handles the virtual machine thatremains between hardware and virtual machines
X11 Network transparent graphic system, used mostly in UNIX-like systems i.e. Linux
X-terminal Thin-client terminal using X11-protocol to provide a desk-top from a remote server
XEN Virtualization platform.
1. INTRODUCTION
Thin client systems are making a comeback. Most services nowadays are Internet orintranet-based and are accessed via browsers. This reminds us of the time of big main-frames and dump terminals, where all the processing was performed in mainframesand terminals were used to simply connect to ’services’. Modern examples of such In-ternet based services include Microsoft Office 356 1, Facebook 2 and Google Maps 3.
Nowadays, it is safe to assume that most of the computers are connected to Internet.Smartphones are also gaining the same graphical abilities and processing power thanworkstations possessed a few years ago. These new processing power and graphicalabilities have made smartphones one of the main categories of computers. The Internetconnectivity of smartphones is also no longer unusual anymore and personal calendarsand music collections commonly reside in cloud services.
Mobile platforms are modern thin clients. Mobile phones are used as thin clients tocloud services. Mobile phones performance has increased rapidly. Performance allowsmore processing power but at the same time, the scale of processed data has movedto a "big data" scale. Moving computing resources to centralized locations is feasibleas networks are getting faster and easier to access. Good examples of mobile cloudservices are Amazon Silk 4 and Opera Mini 5 -browsers. Both utilize data optimizationin specialized computing centers to compress and speed up WWW-browsing over 3G-networks.
Centralizing data processing and storage allows the higher utilization of resources,as user computers do not need to be fitted to handle all the data or processing needs. Ahigher utilization rate enables the qcquisition of more resources, because they can beused more efficiently among multiple users.
The CERN library uses the LTSP (Linux Thin Client Server Project)-system6 tooffer public terminals. Thin clients are suitable for public terminal use being silent andsmall in physical size. Public terminal setup is presented in Kommeri’s Master’s thesis[1, appendix B]. A modern version of LTSP utilizes the cloud service and runs DaaS(Desktop as an Service) to provide LTSP-like desktops to thin clients. A simplificationof DaaS is illustrated in Figure 1.
The current thin-client solution [1, Appendix B] in the CERN-library is gettingcloser to the end of its life as hardware is getting old. An upgrade path is neededto be able to provide public terminals in the future.
The aim of this Masters’ thesis is to evaluate the feasibility of using the CERN IT-provided private cloud service as a platform for DaaS. Implementation is also designedand implemented during the project based on evaluation. Using the cloud service al-lows more computing power and removes direct problems resulting from ageing hard-ware. The DaaS system is presented based on evaluation and the implementation isdescribed.
The evaluation focuses on the possibility of DaaS setup using CERN network in-frastructure. CERN in-house cloud services are included in the network infrastructure.
1http://www.microsoft.com/office3652http://www.facebook.com/3http://maps.google.com/4http://www.amazon.com/gp/help/customer/display.html/?nodeId=
2007754405http://www.opera.com/mobile/features/6https://fedorahosted.org/k12linux/
Desktop
CERN LAN
Thin Client
Figure 1. High level architecture of thin client setup, where the virtual instance residesin the cloud service.
Remote desktop protocols are also evaluated for gaining a more efficient connectionmethod between the thin client and the desktop. Remote desktop protocol efficiency ismeasured using latency and bandwidth usage.
In this Master’s thesis three different virtualization platforms are evaluated; Xen,KVM and HyperV. HyperV and KVM were choices provided by CERN-IT.
NX is evaluated and compared with traditional X11-tunneling inside the SSH-connection,plain X11 and local X11 display. Latency and bandwidth usage are presented as met-rics to compare remote desktop protocols.
One motivation for upgrading CERN Library system is so that users and visitors canenjoy free public terminals for longer. Thin clients are a perfect solution for libraryterminals as they benefit from low noise and low space consumption.
11
2. VIRTUALIZATION
This chapter provides the basics on how virtualization works and what componentsvirtualization contains. The chapter starts by briefly presenting the virtual host con-cept. Different types of hypervisors are covered in section 2.1. Specialties relatingto x86-architecture and why it has been a hard target to be virtualized are covered insection 2.3. What paravirtualization is and why it is good thing to have are presentedin section 2.4. Then we briefly discuss three different types of hypervisor and virtu-alization software. Finally cloud services are specified and how virtualization fits intocloud services.
Virtualization in its current form was first described by Goldberg in his thesis [2].All evaluated virtualization platforms in this thesis follow those basic principles thatGoldberg discussed. [2]
In a normal server environment, OS runs directly on top of hardware running ser-vices like HTTP1/SMTP2. The majority of the services are not used all the time.Servers are running idle most of the time. Virtualization helps to utilize resourceslike physical space, CPU time and memory more efficiently. Virtualization enablesservers to run alongside each other on top of the same hardware. Servers runninginside a virtualized environment are called virtual hosts. [1, p. 16]
Virtualization as a technology is used in large mainframes to separate services or vir-tual hosts from each other. Using virtual hosts allow for better utilization of centralizedresources without sacrificing security. Technology has expanded from big mainframesto more common server platforms, which are used by cloud service providers, for ex-ample. Service separation means for example running different WWW-domains in aseparated virtual host [1, p. 16]
In virtualized environments it is vital to separate different virtual servers from eachother. The ability to separate a virtual host from hardware allows instances to movebetween physical hosts. Using hardware resources more efficiently also means betterutilization of physical server cabinet space, heating resources and electricity [1, pp.15-16]
2.1. Hypervisor
Running servers on top of the same hardware requires intermediate functionality tomonitor resource use and ensure that every server has its share of CPU time and I/O-resources. This task is usually handled by software called hypervisor. Hypervisor staysbetween the hardware and the servers to handle virtualization and manage servers, ormore commonly virtual hosts. [2, p. 3]
A VMM (Virtual machine monitor), or nowadays more commonly known as hy-pervisor, has the task of handling all virtual machines and their interaction with realhardware. This requires checking that all virtual machines have their share of real CPUtime, checking instructions virtual machines want to run, and mapping memory areasto virtual machines to use and handle I/O-calls. [2, p. 4]
Hypervisor needs to take precedence over virtual machines to be able to fulfill agiven task to manage virtual hosts. Prioritizing ability is achieved with hypervisor
1Hypertext Transfer Protocol, WWW-server2Simple Mail Transport Protocol, email
12
residing in the lowest layer (layer 0) and virtual machines running on the higher layers.Layers are computer architectures that are used to separate privileges. Layering helpsprioritize what can be run and accessed. [2, pp. 18-20]
Layers are also called rings. Ring 0 is closest to the hardware and is allowed to useall the instructions the hardware supports directly. Higher layers have the same set ofinstruction in use, but they are not allowed to use them all directly. This privilegedmachine code instruction group is handled by kernel. Kernel talks to the hardwareor, in a virtualized system, hypervisor. The OS distinction between layer 0 and otherlayers is names the layer 0 execution as ’kernel mode’ and other as the ’user mode’.[2, p. 45] [3, p. 12]
Privileged instructions are designed to run in ring 0. Running privileged instructionsin other rings is impossible because their behaviour is forbidden with hardware. Thehypervisor either uses hardware support or emulates instructions using software to runthose privileged instructions. Hypervisor checks instructions so they do not interruptother virtual machines or hypervisor itself. Most of the instructions can be run in realCPU without the need to intervene in hypervisor [2, p. 45].
In a system without virtualization, the kernel runs in layer 0 and the user-spacein layer 1 or higher. Goldberg visualizes the positions of virtualized system layersin terms of the host and the guest. The visualization in Figure 2 presents the layerpositioning of systems. In the figure hypervisor is running in layer 0. Hypervisor canbe run partly in layer 1 when the user-space is used to run code. Virtual machinesrun layer 2. On virtual machines, the kernel thinks it is running in layer 0. From thevirtual machines point of view, the virtual machine’s kernel is running in layer 0. [2,pp. 16-20]
Layer
0
Layer 1
Layer
0
Layer 1
Layer
0
Layer 1
Layer
0
Layer 1
Layer 2
Host and
hypervisor
Virtual hosts
Figure 2. Virtual machine layer 0 is run in the host’s layer 2.
13
2.2. Hypervisor types
Robert Goldberg defines two types of hypervisor-based virtualization methods [2].Hypervisor can run without relying on the OS (Type I) seen in Figure 3 or hypervisorrelies on the OS and runs on top of the OS (Type II) seen in Figure 4. The illustrationsare based on HyperV documents figures [4]. [2, pp. 22-27]
Type I hypervisors nowadays usually have micro kernel hypervisor implementation,which allows them to move a lot of functionality to be run in user space instead ofkernel space. This helps to keep kernel space small and fast, but requires a separatevirtual machine for hypervisor user-space code. HyperV implementation can be seenin Figure 11, and Xen also follows a similar implementation [5, section 2.3]. [4, pp.26-27]
Hypervisor
Hardware
VM1 VM2 VM3
Figure 3. Type 1 hypervisor runs directly on top of hardware.
Hypervisor
Hardware
VM1 VM2 VM3
Operating system
Figure 4. Type 2 hypervisor runs on top of OS.
For virtualization platforms, the evaluation focuses on platforms that are availableor in use during the project in CERN. The platforms under focus are Xen, KVM and
14
Hyper-V -platforms. All those platforms are hypervisor-based, so they require thesame guest and host architecture on operating systems.
2.3. X86-architecture virtualization
X86 architecture has four different rings. Usually only layers 0 and 3 are used (0for kernel and 3 for user-space). Supporting virtualization without hardware supportrequires porting the guest OS kernel. Porting is needed so it is able to run at least inlayer 1 in real hardware. [5, p. 165]
Emulation of machine code instructions is usually implemented by trapping. Trap-ping works in a similar way to interruptions work at the hardware level. The basicidea is: Hypervisor installs code for virtual hosts’ CPU privileged instructions traps.When the guest OS code calls instructions, the execution jumps into hypervisor em-ulation code. Hypervisor code is run instead of direct privileged instruction and theprogram continues normally after that. X86-architecture does not offer this instructiontrapping and some commands can also be executed in unprivileged mode, but they actdifferently in different modes [6, section 3.1].
In the past, running these privileged instructions led to the x86-architecture virtual-ization being hindered. Some instructions worked differently depending on the con-text they were run in, or instructions altered the running environment like interrupts orhardware clocks. Agesen et al. use popf instruction as an example of such command:popf loads into the %eflags-register set of flags from the stack. In kernel mode,popf loads all flags properly, including a set of ALU (Arithmetic and Logical Unit)and system flags. In user mode, flags are not restored from the stack and popf doesnot give any faults about incorrect behavior. This shortcoming disables the guest OSto clear interrupts as the hypervisor could not be sure which mode guest OS intended.[7, p. 4]
The problem could have been avoided if the architecture had implemented protec-tion traps with instructions that require kernel mode to work properly; just as main-frames had originally implemented virtualization. With traps, hypervisor could installtrap code for protection calls and emulate them if needed. X86 architecture does notimplement this protection trapping, so the problem needs workaround. [7, p. 4]
2.3.1. Binary translation
VMware overcame the protected mode problem in x86 through binary translation. Bi-nary translation intercepted instructions are meant to run in protected mode in run-time. Translating machine code would not require modification of guest OS. Most ofthe machine code is meant to run in user mode, and at requires no or very little mod-ification. Simplification and speed-up is gained by just intercepting code lazily; thereis no need for heuristic separateion of data and code. Figure 5 illustrates the binarytranslation operation environment. [7, p. 5]
15
%EIP
Binary translator
Emulate code CPU Execution
Memory
Retrieve instruction
Protected instruction
unprotected instruction
Intercepted instruction group
Receive code location
Guest OS registers
Hypervisor
Figure 5. Binary translator intercepts guest program counter.
2.3.2. Paravirtualized CPU
Xen solved the protected mode problem by providing paravirtualized CPU, which of-fers a subset of x86 instructions in direct execution and privileged instructions aredisabled if they are not able to be run in user mode properly. A guest OS side drivertells OS that it supports the restricted x86 instruction set. Porting of the guest OS ker-nel for this paravirtualized CPU architecture is required. A paravirtualized CPU offerssome privileged instructions as emulated commands. The speed decrease comes froma need for the execution of multiple machine code instructions instead of one. [5, pp.165-167]
Comparing Xen solution to VMware binary translator shows that using a binarytranslator allows the running of unmodified OS, but the hypervisor code needs morefeatures and is more complex.
2.3.3. Memory management
The MMU (Memory Management Unit) is a key point in virtualization performance,and needs special attention in hypervisor. The x86 architecture itself has support forvirtual memory since 80386 with hardware page table walker and MMU with TLB(Translation Lookaside Buffer). TLB is used by MMU to cache mapping from virtualmemory addresses to physical memory addresses.
Shadow page tables are used in virtualization as another indirection layer in memorymapping. Mapping handles virtual hosts memory addressing to physical memory lo-cations. This is similar to the task that TLB and segmenting addresses does normally.Figure 6 illustrates the shadow page table in virtualization.
16
If hardware does not offer support for MMU virtualization, hypervisor providesshadow page tables for guest OS. Shadow page tables point to ’physical memory’ forthe guest OS. The host sets the shadow page table as read-only, so guest OS writinginto memory triggers a trap call for the host. The hypervisor handles the guest writesto the shadow page table and transfers them to writing in real memory, updating theshadow page table. Isolation between the guest OS and real memory is also neededfor security. Hypervisor needs to keep the guest OS from writing outside the definedmemory areas, like hypervisor memory or other guest OS memory areas. [5, pp. 169-170]
Guest OS
Shadow
page table
Hypervisor
Memory%cr3
Read
Write
Memory I/O
Figure 6. Shadow page tables in virtualization.
Virtualization utilizes segmentation to protect access to hypervisor memory spacefrom virtual hosts. Xen and VMware moves their own address to the space upper partof the memory segment, and limiting memory segment size for virtual hosts to stayunder this limit. Locating the hypervisor code in the upper part of the memory addresssimplifies implementation, as nowadays most OSs set their memory base to 0 andassume flat address space. Xen locates the topmost 64MB area in the address space.This prevents TLB flush when moving between hypervisor and guest [5, p. 166].VMware moves itself in 4MB of address space. The memory location is illustrated inFigure 7, based on VMware illustration. [7, p. 8]
Guest Hypervisor
0x0 0xffffffff
base
base
4-64MB depending on hypervisor
%ds
%fs
Figure 7. VMM code location in memory segments.
17
VMware introduced ballooning to help guest OS memory management to work cor-rectly. Later, Xen utilized the same idea of ballooning driver [5, p. 170]. The bal-looning idea is to implement a balloon module into the guest OS kernel that inflatesand deflates inside the guest OS memory according to the host’s physical memory al-location. Inflating the guest OS memory allows the guest OS to page memory to swapspace before the physical memory runs out. Ballooning also prevents double paging,where the host has swapped the memory page and the guest also swaps memory page.[8, p. 3]
Guest OS
Memory
Hypevisor
Real memory
ballooning
Figure 8. Ballooning driver inside guest OS.
2.3.4. X86 64-bit virtualization
Intel and AMD introduced the first 64-bit processor families in 2003 and 2004. Regis-ter count increased in 64-bit processors and the size of registers. Size increased from32-bit to 64-bit. The amount of registers increased with eight general use registers.Those extra registers are available in what is termed "long mode", a 64-bit executionmode. Long mode changed segment registers, all limit checks were removed so mov-ing hypervisor code onto the top part of virtual memory would not be safe. Instructionsto move ALU states to and from the register were removed. ALU-flags could still bestored and restored using stack, which was slower. [7, p. 12-13]
AMD restored segment limits in later processors. Intel still lacks those segment lim-its and using software emulation in Intel 64-bit processors is not possible - VMware’semulator is an example of this [7, p. 13]. Most modern 64-bit processors include sup-port for hardware assisted virtualization support, and hyper-V, for example, requiressupport in the 2008 version [4, p. 48]. As hardware assisted virtualization is a com-mon feature in modern processors, it is relied upon in most of virtualization softwarenowadays.
18
2.3.5. Hardware assisted virtualization
Nowadays most of the x86 processors have hardware assist virtualization support,which helps and speeds up virtualization. AMD and Intel as major x86-architectureCPU creators have designed similar extensions for CPU guest mode execution; AMDSVM (Secure Virtual machine), Intel VT (Virtualization Technology). MMU virtual-ization also have similar extensions; AMD RVI (Rapid Virtualization Indexing) andIntel EPT (Extended Page Tables).
CPU guest mode execution allows virtualization software to tell CPU to run virtualhost in hardware directly. Hypervisor can control the execution and read state of avirtual machine using VMCB (Virtual Machine Control Block). Each virtual machinehas its own VMCB. In guest mode, OS is allowed to run privileged code withoutemulation and CPU keeps the registers’ state in VMCB so they can be separated fromeach other. [6, section 4.1]
Hypervisor uses guest mode by issuing a vmrun command with VMCB. Hardwareloads the guest OS state from VMCB and execute it until the guest OS reaches the statedefined in VMCB. When the state is reached, hardware issues the exit saving guestOS state in VMCB and restores the host state for hypervisor. Hypervisor can controlexit states to be I/O requests, page faults, TLB flushes or address-space switches, forexample. Control of the exit state allows hypervisor to control what it needs to checkor emulate itself and to keep the virtual host environment coherent between guest andhypervisor. [6, section 4.1]
MMU virtualization with hardware has hypervisor controlled page tables that arehardware maintained. The hardware can directly handle MMU operations within lim-its that hypervisor has defined and without the need of hypervisor to intervene. Hyper-visor maintains nested page tables for every guest OS to map guest physical addressto the host physical address. MMU then uses mapping information to complete guestTLB and handle guest memory access in hardware. Another abstraction layer increasesthe cost of TLB miss, when requested memory location is not in TLB. Decreasing TLBmiss and speed up TLB handling use of large page size helps significantly. Page sizesfrom 2MB up to 1GB are supported with modern hardware and should be used invirtualization. [7, pp. 14-15]
2.4. Paravirtualization
Hypervisor can provide fully virtualized devices (CPU, Memory, I/O) to virtual hosts.Providing semi-virtualized devices and drivers to talk to semi-virtualized devices gainsspeed in virtualization when compared with fully virtualized devices [9, p. 229].
Paravirtual drives can connect to host device drivers with little effort. Paravirtualizeddrivers do not need to implement everything that normal drivers usually implement,only those parts that are needed to communicate with hypervisor and implement driverAPI. Real devices are used via hypervisor. Usually I/O-devices (network, disk, serialport, display) are paravirtualized. [5, section 2.1.3]
Paravirtualization is usually handled by providing special drivers for every supportedOS to interact with paravirtual interface. This also leads to a requirement for a virtual-ization platform to be able to modify guest OS, mainly the ability to build a driver forcertain use a guest OS can utilize. [5, p. 165]
In the past, CPU and memory access needed to be paravirtualized too. This required
19
support from a guest OS to identify a paravirtualized memory controller and CPUand so to be able to communicate with those [5, p. 165]. Since hardware assistedvirtualization became a common feature in x86_64 processors, the virtualization ofCPU and MMU are nowadays virtualized with hardware assistance [6, Chapter 4].
2.5. KVM
A kernel-based virtual machine (KVM) works inside a Linux kernel. KVM is tiedto Linux as a host operating system. The Linux kernel is used to handle memorymanagement and device drivers. Porting of KVM to FreeBSD has started recently3,but it is not usable yet. KVM is the newest virtualization solutions of those that havebeen evaluated. Hardware support for virtualization is heavily dependent on KVM asthere are no code for instructions emulation. [10, p. 225]
KVM itself is only the core hypervisor type II implementation inside the Linuxkernel as a subsystem [10, p. 225]. Without supporting software to actually managevirtual machines KVM does not have any real practical use. Currently QEMU is theonly emulation software that is able to utilize the KVM interface. [9]
Figure 9, based on an illustration in an IBM article [9], shows a guest OS running asone process in a host process list run by QEMU binary. When using multiple virtualcores in guest OS, every core is shown as one thread in the host side. [9]
Figure 9. KVM high-level architecture.
Figure 10, based on an illustration in [10, p. 226], shows the execution loop of theguest in terms of the host. The basic idea is to run guest mode as any other process.Running guest continues until the need for I/O, the need for assistance or some faultis encountered. [10, p. 226] QEMU is used to handle all I/O and memory managing.
3http://retis.sssup.it/ fabio/freebsd/lkvm/
20
KVM offers hardware assistance for native CPU and MMU virtualization. [9]
IO or
Signal?
Handle
Exit
Execute Natively
in
Guest Mode
Enter
Guest
Mode
Issue Guest
Execution ioctl
Handle I/O Yes
No
Use Mode Kernel Mode Guest Mode
Figure 10. KVM guest execution loop.
QEMU provides a few paravirtualized drivers to work with KVM, using virtio. Thevirtio framework [11] is a common framework for most of the KVM paravirtualizeddrivers. Drivers include disk drivers (block device), a network driver and a VGA-console (VNC, SDL console or spice protocol, for spice see section 3.5). Host imple-mentation of those are inside QEMU, so the host does not need any specific drivers toprovide virtio to guests [12].
QEMU and KVM supports an overlay feature in disk images. The feature allowsvirtual instance to start with a base-image, but changes during runtime are saved else-where. This would allow for there to be one base-image to all virtual hosts. Allowingonly one image to be upgraded or changed to have changes propagate to all guestsusing a modified image. [13, section 1.4]
Every virtual CPU is run as a separate thread and the virtual machine itself is oneprocess in the Linux user space. Running a virtual host in the user space allows theLinux kernel to handle memory allocation as it sees fit. QEMU uses the KVM interfaceto run virtual host in guest mode and return control back when I/O or similar access isrequested.
2.6. HyperV
HyperV is Microsoft hypervisor-based virtualization platform for x86-64 systems (in-cluding 32bit guest OS) [4, p. 41]. HyperV requires hardware assisted virtualizationsupport in CPU [4, p. 48]. HyperV architecture can be seen in Figure 11. (source:Hyper-V wikipedia article4)
Terminology in HyperV states that the virtual machine is a “child partition” and the
4http://en.wikipedia.org/wiki/Hyper-V
21
partition hypervisor is used as a “Parent partition”. The difference between this andGoldberg’s terminology relating to layers or rings [2] is that HyperV documentationand architecture refers to "Ring -1". Ring -1 is where hypervisor operates. The defi-nition of ring -1 comes from the architectural model where the kernel is run on ring 0and user space on higher levels. Hypervisor resides below the kernel ring, and HyperVarchitecture calls it ring -1. [4, p. 27]
Figure 11. HyperV-architecture.
HyperV is able to run Linux as a guest OS, and Microsoft lists some Linux distribu-tions as officially HyperV aware and supported [4, p. 50-52]. Paravirtual solutions inHyperV use VMBus (Virtual Machine Bus). VMBus is a high-speed API between vir-tual machines and hypervisor communication used by paravirtual drivers. If the guestOS is not HyperV aware HyperV offers emulated devices which guest OS can interactwith. HyperV aware means in this case that HyperV paravirtualized drivers are presentin the guest OS. [4, p. 35]
All the devices showed as child partitions are VDevs (virtual devices) in Hyper-Vterminology. Devices are either emulated or synthesized. Emulated devices includelow-level devices like BIOS, PCI Bus, RTC, Ethernet device and IDE controller. Syn-thesized devices contains HID (Human Interface Device) devices, paravirtual storagedevices and interrupt controllers. Using synthetic devices requires paravirtual drivers,or the use of Integration Services in Hyper-V terminology. Guest OS is known asenlightened if it supports Hyper-V paravirtual drivers. [4, p. 31-34]
The ability to have one base image, which would be the base of multiple guests, doesnot seem to be possible currently with HyperV. According to the HyperV manual andCERN IT multiple instances from one image are not supported. During the project,functionality was not noted. HyperV has a feature which exports snapshots as newvirtual hosts, but exporting requires copying the snapshot and stopping the virtual hostbefore export. [4, pp. 79-80]
22
2.7. Xen
Xen was originally developed at the University of Cambridge as an open source project.It was later productized by Xensource [1]. Recently Xen changes were accepted intomainline Linux to act as dom0 with Xen hypervisor. Earlier use of Xen meant usingcustom kernels in hypervisor, but most of the modern distributions had those changesincluded out of the box. [14]
ABI (Application Binary Interface) does not need any modifications [5, p. 165].Xen can run unmodified guests if the host CPU has virtualization support5 [14]. Xenis the only platform from evaluated platforms, that does not require hardware supportfor virtualization. The general architecture of Xen is similar to that shown in Figure 11[5, p. 168].
Paravirtualization is implemented in Xen by providing kernel module-drivers forparavirtualized devices. Paravirtualized driver modules are part of the standard Linuxkernel [14]. Using Windows in a guest OS requires hardware virtualization support, asparavirtualized drivers have not been provided with open source XEN. [15, p. 2]
Instead of rings or layers, Xen terminology talks about different domains. Host andhypervisor are in dom0, which can be said to be synonymous with residing in ring 0.DomU in Xen terminology is ring 3 or a similar unprivileged ring, where virtual hostsare run [5, p. 168]. In Xen, the kernel is run from the dom0 file system instead of thedomU guest root file system. [15, p. 16]
Hypervisor is kept as small as possible moving most of the control and managementfeatures in the special dom0 management domain in user-space. [5, p. 168]
2.8. Cloud services
Cloud services include the principle of self-help, so users can create a virtual host andget services running independently. Self-service cuts down bureaucracy in obtainingcomputing resources, as usually users need just a platform with a certain OS. CERN-IT runs a private cloud service for CERN staff and users. Anyone with a CERN-ITcomputing account can create and manage their own virtual hosts via a web interface6.In cloud computing, this service would be called Infrastructure as a service (IaaS). Dif-ferent service types specify how low access each user has in a server. The distinctionbetween different cloud services are visualized in Figure 12. This illustration is basedon a Microsoft presentation [16]. [17, p. 51]
Examples of different services: Amazon EC2 is IaaS, Microsoft Azure is PaaS andGoogle Docs is a SaaS service.
5Most modern desktop processors have hardware support for virtualization6http://vmm.cern.ch
23
Applications
Data
Runtime
O/S
Virtualization
Servers
Storage
Networking
Applications
Data
Runtime
O/S
Virtualization
Servers
Storage
Networking
Applications
Data
Runtime
O/S
Virtualization
Servers
Storage
Networking
Applications
Data
Runtime
O/S
Virtualization
Servers
Storage
Networking
You m
anage
You m
anage
You m
anage
Man
aged
by v
endor
Man
aged
by v
endor
Man
aged
by v
endor
Regular server IaaS PaaS SaaS
Figure 12. Distinction between cloud service types.
24
3. REMOTE CONNECTION PROTOCOLS
Mainframe terminals use a character-based remote connection method via serial lines.As graphics user interfaces were created and network speed increased, graphical re-mote connection methods followed [18, p. 1–2]. The most notable in UNIX-world isthe X11-protocol, which is a desktop graphical system designed to be network trans-parent [19, p. 1]. The X11-protocol is still used today for most of the UNIX-likegraphic systems.
Remote desktops are used more and more for safe access to business software foremployees on the road. Remote access is used for software which cannot or is notfeasible to be moved over to browser-based interface, such as allowing an outsourcingpartner to run in-house development software. Gaming solutions are also under devel-opment in projects like games@large [20] to enable gaming over a network without theneed for a state-of-the-art PC or console. Only a sufficiently fast Internet connectionis required. [18, p. 1–2]
There are multiple remote desktop protocols, and most modern ones are focusedon providing low latency usage over WAN links, like Spice [21] or the games@largeproject [20]. Some methods to accomplish low latency add context aware proxies andagents between regular X11 connection endpoints. With proxies and agents the graph-ics that are actually needed are transferred with efficient compression and cachingapplied. The NX-protocol [22] (section 3.5) uses this method.
Remote connection protocols transfer user keyboard and mouse events over the wireto the server. The server-side processes them and sends back GUI updates. Raw pixelsare not feasible to send over the wire. Bandwidth use of sending raw pixels wouldmake it impractical in most networks. [18, p. 1–2]
Low latency gaming over WAN links is under research. The Games@large [20]project focused on developing gaming over WAN links. Their system utilized simi-lar ideas to Spice [21], using DirectX-drivers. Games@large developed a system toprovide low latency gaming to users via low power end user terminals, in their casegaming set-top boxes or mobile phones. [20]
The most noticeable differences between remote connection methods are bandwidthusage, noticed latency and connection support: can a connection be made to the con-sole directly, or can the connection be used to run separate applications or desktopswhich are shown on a local display.
3.1. User interface latency limits
Research done by Robert Miller [23], states that boundaries for different user expec-tations are related to latency of the system. Latency in UI is the speed of response touser action in our measurements. Response time here is visual feedback, for examplea print typed letter to screen, an update screen based on mouse action, etc. [23, p. 271]
In normal usage, if the latency of UI stays between 100ms and 200ms, the userhas a feeling of an instantly working system. If the response time increases to bebetween 100ms and 1s, the user usually assumes that the system handling data is to bepresented. Over 10s of response time causes the user to lose focus on their work andthe user assumes the system to be too slow to work with. [23, p. 271]
With those limits, we can assume the opening software could take longer than 1second, but visual feedback should be provided to show that the opening of software
25
has begun. It is acceptable for the startup time for software to take from 15 secondsup to a minute according to Miller [23, p. 272]. Then using the software, like menuopenings and scrolling, response should remain below 100-200ms for update speedsto be reasonable. When performing heavy user initiated computing, a UI can spendseconds on waiting results. Even when waiting for results, the UI should give updateson progress.
3.2. X11
The X11 protocol is designed to be network transparent [24, Chapter 3]. Networktransparency was a key factor for the use of X11-terminals and earlier remote connec-tion methods. The X11 display can be pointed to a remote host and can send graphicsand events unencrypted over the IP network[24, p. 87].
Graphics are built from graphic primitives. Primitives are basic shapes that are de-fined (line, rectangle, point, fonts, pixmaps etc.). Shapes can be transferred usingpre made primitives instead of creating everything from pixels. Transferring shapesinstead of pixels saves bandwidth. Instead of sending pixel message of a few bits,sending the same sized graphic request affects 10 to 100 times as many pixels on thescreen. [24, p. 92–95]
The client/server concept in the X11 system is different to a webservice. A Web-service client is the browser and the server is the HTTP-server. The X11 client is theapplication that is run on the server side and the server is the user end display. Anillustation of this client/server separation is presented in Figure 13. A separation ofclient and server allows network transparency, as communication can go through alocal socket or the network connection without a change of code. [24, p. 84]
The X11 server location is usually defined in UNIX-systems with a DISPLAY vari-able, which contains a ”hostname:displaynumber.visual“-style string. The local dis-play’s hostname is left out. Visual is also usually left out and it defaults to 0. Thedefault port for X11 connections is 6000+displaynumber. [25, section 9.3.1]
Server running software
Browser
Terminal
X11 Client
X11 Server
Figure 13. X11 client/server separation.
Mouse and keyboard are handled in a workstation on the X11 server side. A han-
26
dling cursor displayed at the user end eliminates RTT-latency from cursor movement.Mouse and keyboard actions are sent as events to X11 client applications. Mouseevents include button press or release, drag or drop and leaving or entering an applica-tion window area [24, p. 100]. Those events are usually handled by a windowmanager.Keyboard events are translated from X11 keycodes according to keymaps in the appli-cation side for example ascii characters [24, p. 99].
In X11 content drawing is left entirely to the client application. The X11 serveronly sends asynchronous notification events when the application window or part ofa window has become visible. Notify contains a rectangle area marking what partof window has become visible. Leaving all the drawing to the client can cause visiblegraphical issues, for example in menu drawing. A client can store pixmaps in a server’soff-screen memory for faster access so they do not need to be transferred every time.[24, p. 95-97]
X11 is designed to be expandable with extensions [24, section 10.6]. XDamageextension was designed in 2007 to help applications monitor screens more efficiently[26]. The XDamage extension provides events in rectangle areas where redisplay isneeded. Protocol balances between the number of rectangles reported and the size ofthe redisplayed area. The working principle is to make every primitive (line, cirle,rectangle etc.) that needs redrawing a single rectangle and then report on the unionof those rectangles. Based on the report the application can redraw only the requiredparts of the application content. [26]
Network latencies for X11 in the design phase were set to be between 5 and 50 mil-liseconds. Most X11 protocol commands are asynchronous from the start to archive afast enough latency over the network [24, p. 87]. The X11 protocol itself is bandwidthefficient and has low latency. The Xlib-library has been found to be the reason thatX11 applications were awkward to run over high latency links [27, chapter 1.2].
Xlib resided directly on top of X11 and most of the graphical toolkits like GTK+and QT utilized it as the lowest layer. An increase in graphic use from the days ofX11 design has increased bandwidth consumption of pixmap transfers. Pixmaps areuncompressed images and are the main cause of X11 bandwidth consumption. Generalgzip-based compression decreases bandwidth usage according to Packard and Gettysresearch from 2003 [28, pp. 215-216].
Xlib is a library which offers C-API to applications and toolkits to utilize X11. Itcontains optimizations for reducing latencies. Xlib is also one of the main reasons forX11 latency problems. Xlib, being an old library, has problems with multithreadinguse and it usually blocks calls until a response arrive from the server. Blocking causesround-trip latency for every call, which is not an issue with local display but it doesslow down remote applications excessively. [27, section 1.2]
XCB is developed to be more robust and re-entrant C-API to replace Xlib. The de-sign goals were to implement a simple and efficient thread-safe library to implementX11 protocol for clients. XCB uses “reply cookies” to hide latency, so it can return im-mediately from a request call and block it if a response is requested but is not availableyet. [27, section 2.1]
3.3. X11 tunneled inside SSH
The X11 protocol defined a way to transfer graphics and IO over the IP network, but atthe time network encryption was not defined into the protocol. Nowadays it is common
27
to use an X11-protocol tunneled inside an SSH-connection to get network encryptionin an IP connection. X11 tunneling is visualized in Figure 14.
Figure 14. X11 traffic goes inside SSH connection from server to client.
SSH X11 forwarding is established by creating a TCP-tunnel inside an SSH con-nection.A remote end DISPLAY-variable is pointed to a remote end hostname with adisplay to some free display. The TCP tunnel’s remote end port is 6000+found freedisplay and the local end is forwarded to the X11 server.The SSH negotiates X11 au-thentication on behalf of the client app with the X11 server. [25, section 9.3]
SSH can compress the transferred data alongside encryption. Compression can bemore efficient in therms of bandwidth than a plain X11-connection. Encryption isusually mandatory nowadays for multiple security reasons. SSH compression uses adeflate-algorithm [29] and is not content aware. Deflate compression is not as efficientas compression used in NX, for example. Compression can help the X11 connectiondue to the fact that compression is performed before encryption. The amount of datato encrypt is decreased and encryption time decreases from compression. Speed-upon encryption can help latency even if compression itself adds only small amount oflatency.
3.4. NoMachine NX
The NX protocol is derived from the DXPC-project [30]. The DXPC-project specifiedthed compression method for the X11-protocol by compressing identity and data-partsdifferently and separately. This provides more efficient compression of transferreddata, as the identity part is well known and can be compressed more effectively byutilizing the knowledge of the content. The data part also has a known structure basedon the identity part. DXPC also utilizes an X protocol messaging pattern, where itis higly probable to that the message is identical to a similar type of message sentrecently. DXPC design is based on the FHBX(Fast Higher Bandwidth X) article [31],and caching still uses the same method as described in the paper. DXPC proxy sendonly the cache location in cases where the same message is in the cache already. [30,design.html]
In a small number of cases there is only a small difference between the currentmessage and the new message. An example is the "Create Window" request, where
28
the “Window ID” value is usually the previous window ID created plus a small positiveinteger. DXCP sends a compressed difference of messages to a certain cache locationin this case. The difference can usually be encoded with fewer bits than the actual fieldvalue. [30, readme.html]
NX proxy handles the compression part in an NX connection. NX combines thecaching of messages with compression. Caching eliminates unnecessary round-tripsby simply sending the cache location reference number over the wire. Both ends keepcount on the other end’s cache status. By sending messages over the link, the cachelocation is sent to where the message is stored for future referencing. The NX agenthandles caching. The connection is tunneled via SSH X11 forwarding to provide en-cryption over the network. [22]
Only the encoding side needs to calculate checksums and store them, as the decodingside only needs to store payload in the cache location encoding side signals. It iscomparable to Huffman coding as the most frequently used messages are in front ofthe cache and need the lowest amount of bits to encode the position. Different NXcomponents and their placement are shown in Figure 15, courtesy of [1]. [22]
Figure 15. NX architecture overview.
3.5. Spice
Spice was originally developed by Qumranet, a company that was acquired by RedHat in 2008, which Spice protocol as opensource [32]. Unlike NX, Spice does nottransfer X11 protocol messages or primitives. Spice uses its own messaging protocoldefined by transfer graphics and other IO between the server and the client. Messagingprotocol has a separate message channel for every remote device (keyboard, mouse,video). [33, p. 4]
The basic components in Spice are the Spice server and client, QXL devices andguest QXL drivers. Guest QXL drivers are provided in X and GDI graphic systems.Graphic engine drivers translate drawing requests to QXL syntax and send them to aQXL device. [34, Chapter 2]
Libspice takes QXM syntax requests from the QXL device and adds them to thedisplay tree structure. The display tree structure represents the graphic commands to
29
reproduce display content. Libspice optimizes the display tree by dropping commandsthat would not produce visible change to the user. [34, Chapter 2]
Videostream is also detected by using the display tree. An optimized command treeis moved into the send queue, from where it is send to the Spice client as a Spiceprotocol message. After the command is either optimized out of the display tree orsent to the client, it is placed into the release ring. [34, Chapter 2]
The QXL guest driver uses the release ring to release resources related to the drawingcommand. Decreasing the amount of unnecessary traffic between the guest OS and theconnecting Spice-client helps bandwidth efficiency and latency. [34, Chapter 2]
Currently, Spice relies heavily on QEMU and libspice integration in the server-side.Libspice can be used in any Spice VDI(Virtual Device Interface)-compatible host ap-plication according to spice documentation [34, section 2.1]. QXL guest drivers followthe paravirtualization method, as they don’t need to really work as real hardware.
General IO-interfaces, such as mouse and keyboard, are offered via a VDI-agent.An overview of Spice architecture can be seen in Figure 16, which is based on a figurein the Spice overview page [21]. IO interfaces are implemented as paravirtualizeddrivers. Drivers contains a VDI port driver in the guest OS side, and a VDI port deviceinside QEMU. [34, Section 2.2]
The VDI port is also used to communicate any actions that Spice needs to be exe-cuted in a guest OS context, like resolution change. Messages coming from the VDIport from the guest OS are placed in an IO-ring inside the VDI port device. Libspicedetermines if a message should be forwarded to the Spice client or handled by theserver. [34, Section 2.2]
Spice
Figure 16. Spice architecture paired with QEMU.
The Spice server implements multiple interfaces between server and client. Thoseare: Red dispatcher, Agent, kbd/mouse/tablet, playback and record -interfaces. Everyinterface uses its own channel for communication. Every channel has a separate TCPsocket, and can be separately encrypted using SSL. [34, Section 2.4.1]
Red dispatcher handles the display and the cursor behind QXL. Red is the mainchannel that creates other channels if necessary. The agent interface contains config-uration messages between QEMU and the Spice server and Spice client. Keyboard,mouse and tablet channels are handled by own channel per one input device. The mainchannel also handles migration if guest is migrated to other host. [34, chapter 2.4.1]
The Spice client implements the following channels; Main, inputs, display, cursor,playback and record. The main channel acts as a main channel creating other channelswhen needed. The display channel handles drawing graphics and videostreams. Inputchannels handle the keyboard and mouse. The cursor is responsible for cursor position,
30
visibility and shape. Playback and record channels are used for audio, playback toplay audio on the Spice client side and record channel for audio recorded in the Spiceclient. Audio transfer is implemented using CELT-codec. Pixmaps are compressedusing either Quic or LZ. Quic is a Spice proprietary compression utility based onSFALIC-algorithm [34, section 3.3]. [34, section 2.3.2.1]
Video stream is detected by finding the region that is updated at a high rate. Areaupdates are sent to the client using M-JPEG lossy coded. Coding videostream usinglossy codec saves bandwidth and improves Spice’s overall performance. [34, Section3.4]
31
4. CERN NETWORK INFRASTRUCTURE
This chapter is based on a presentation made during the aims2 rewrite project [35]and CERN IT -wiki pages1. No other documentation was available during the project,which is the cause of using weak references.
CERN IT provides a network and network service in the CERN area. CERN IT-department’s main task is to provide the information technology required to fill CERN’smain mission. They handle everything from Internet exchange and grid computing tophones and user support. In this thesis, CERN-IT is referred to as an organizationunit inside CERN, which provides network infrastructure, computing resources andsupport. [36]
Using the CERN network infrastructure is one of the goals of the project. Bootingthin clients is functionality that can be done using the current infrastructure. In thischapter we go through used components to boot thin clients from the network.
All CERN network devices need to be registered on the Oracle database namedlanDB. LanDB keeps information where the device is located if it is not wifi-connected.The main responsible user and the main user are listed. The main responsible user ormain user can also be a group, so all users belonging to a defined group can administeror have access to the device. Information stored on the lanDB includes the MAC-address of the device NIC (Network Interface Card) and OS type.
4.1. Network boot
DHCP (Dynamic Host Configuration Protocol) provides a way of passing configura-tion information relating the the IP-network. The protocol is defined in RFC 2131.Information passed to a client usually contains an IP-address and netmask, defaultrouter and DNS-servers. The DHCP server can pass other options to the client, such asPXE booting. Basic client-server interaction receives network settings defined in RFC2131 section 3.1. [37]
TFTP protocol is defined by RFC 1350. The protocol uses UDP transfer to sendmessages. When the client requests a file from the TFTP-server, it sends an RRQ-request that contains the requested filename. If the server has requested file, sending isstarted using UDP datagrams. [38]
PXE (Preboot eXecution Environment) is described by Intel [39]. PXE containsDHCP-options to tell the PXE-client what file to request and where the TFTP-serveris to request it from. Specification is defined in a uniform way to provide pre-boot im-ages. The PXE server is defined using NEXT_SERVER, FILENAME options in theDHCP-message. The client initiates a TFTP-request towards next_server request-ing a file defined in the filename option. After a successful file retrieve, the receivedfile is started by PXE boot rom. PXE options in the DHCP-response are described inRFC 4578 [40].
Figure 17 illustrates a sequence of requests in the CERN network. When a devicerequests an IP-address from network, the DHCP server checks from the lanDB if it isallowed to have one. The DHCP server returns PXE-related options according to thelanDB information. PXE options include NEXT_SERVER and FILENAME-options.PXE options tell the server where the device should contact inorder to receive the PXEboot image. [35]
1https://twiki.cern.ch/twiki/bin/view/LinuxSupport/Aims2client
32
PXE Client DHCP
Power On
DHCP Discover
DHCP Offer (IP Address)
DHCP Offer (PXE server)
DHCP Request for IP Address
DHCP ACK for IP Address
PXE Boot Image
PXE Boot Image
PXE Server
LanDB
Read Images for client
Check cliet authorization and special options
Return binary images
Figure 17. Network booting sequence in CERN network.
PXE servers are handled via landDB by AIMS to serve boot images. The defaultimage contains a menu for OS installations. OS installations are driven by kickstart-scripts for Linux installation. AIMS stores PXE boot media and states in OracleDB.[35]
33
4.2. Automated Installations Management Systems
AIMS2 is used for the automated installation of operating systems in the CERN net-work. Client-server model implementation is performed using SOAP2 (Simple ObjectAccess Protocol) as a communication method. SOAP is a W3C defined XML-basedcommunication method to exchange structured information. AIMS is based on RedHat Kickstart3 and can use kickstart files as installation scripts.
AIMS2 is written in Perl. Integration into the CERN network infrastructure is high;AIMS2 utilizes LanDB for machine information, LDAP and E-groups for authoriza-tion. Kerkeros 5 is used for authentication. AIMS2 is defined as one service in KDC.
AIMS2 requires the user to be allowed to manage the device. Permission is checkedfrom lanDB information. If the user is marked as the owner, main user, other root orLinux support in lanDB, AIMS2 allows changes to boot options. Boot options includedefining custom kernel, initrd and kernel options.
The user interface is implemented using the aims2-client commandline tool. Thistool allows the adding/removing of registered devices from the custom PXE optionpool. Devices in this pool can have defined kernel images with initrd and optionalkickstart-files. A PXE filename in DHCP options is defined automatically if customkernel and initrd are defined for the device.
2http://www.w3.org/TR/soap12-part1/3http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/
html/Installation_Guide/ch-kickstart2.html
34
5. CURRENT SYSTEM
A library needs silent public terminals which do not need much administration. Thecurrent system was built as a pilot system in this Master’s thesis work [1].
Four thin clients are connected to one physical server located in a library’s techni-cal room. The thin clients installed have display, keyboard and mouse connected toprovide a physical user interface. This setup fitted library needs. [1]
The CERN library is operated by the Scientific Information Service at CERN andits main goal is to provide access to information that CERN produces or requires [41].They have been aware of the need for public Internet terminals for users without CERNuser accounts as part of the library service. Public terminals have also always beenprovided, with different technologies at different times. [42]
In the following chapter we write about the use of public terminals in the CERNlibrary. The history of public terminals in the CERN library is presented based ona personal interview [42]. After the history, the requirements for the public terminalsystem are listed. Requirements have stayed pretty much the same in all the publicterminals since the current system setup [1]. Section 5.4 provides a technical overviewof the implemented system, which is the basis of new design. Finally, problems thathave arisen during use or current system are described. These are the main drivingforce behind upgrading the system and finding a reasonable upgrade path.
5.1. History of public terminals in CERN library
According to the library, no official reports besides Kommeris’ thesis [1] are availableon public terminal usage in the library. Data on the history of public terminal use wasascertained through a personal interview with Tullio Basaglia, head of library servicesat CERN.
At first, the library had X-terminals, thin clients provided unencrypted X11 connec-tion to a remote host, but there was a need to change to other technology, as supportfor X-terminals ended. [42]
The next solution, desktop PC-installations, had multiple issues; customization tooktime (installation of cables and modifying system to work in public use), units took uplot of space and were noisy, and hardware ages. There were also some cases of misuseof user accounts meant for public accounts. Misuse could be for example leaving P2Psoftware running in order to use CERN’s fast Internet connection to download possiblyillegal content for later pickup. [42]
The library is always open, which leads to a requirement for high availability. Main-tenance requirements placed on librarians should be close to zero, as their main role isnot maintaining IT systems.
Currently, the most common use of terminals in the library is to check webmail andprint airline tickets. As people increasingly own laptops, these are used for working.Registering a laptop on the CERN WLAN requires a CERN computing account, sousing a public terminal for casual surfing is easier than getting access to WLAN. [42]
35
5.2. Requirements for public terminals in library
The following features are the main key points in the current system [1], and are fea-tures that are required from new system following upgrade:
• Available around the clock, every day
• Low maintenance burden, or as little maintenance as possible from administra-tion
• Usage without the need for a CERN-IT account
• Silent
Availability around the clock comes from the library being open and accessible atall times, not just office hours, so terminals should be always available too.
Low maintenance by staff is necessary as librarian’s role does not include adminis-tering IT systems. Systems should be built in a way that require as little maintenanceas possible.
Securing public usage without a login is quite a big issue. In this thesis contextsecure usage means usage without someone leaving key loggers or other malicioussoftware running. In addition, there is a need to make sure there is nothing left behindfrom user sessions, like bookmarks/files/processes, after user logs out, so the next userhas same pristine and unmodified desktop to use.
5.3. Thin client concept
A thin client is a PC that does not have full OS installed but runs a desktop froma remote computer. Minimal OS is loaded from a USB stick or from a network toestablish a connection to a desktop server. It can be said that a thin client is to a PCwhat a browser is to a web-service. Modern thin clients can be a mini PC which doesnot contain a hard drive or optical drives. The operating system is retrieved from anetwork when the device is started or the device can contain a small flash memorycontaining a minimal OS.
The minimal OS usually contains just a kernel and a minimal amount of user spaceto allow the remote desktop connection to the server to start. For Linux thin clientsthis means an X11 server and SSH applications with the required libraries.
The thin client handles the remote connection to the server that actually handles dataprocessing and running applications. Keyboard, mouse and display are handled by thethin client which usually just passes I/O events from those devices to the remote server.
Processing needs in the thin client side are not high allowing the use of passivecooling. Passive cooling makes thin clients less noisy than normal workstations. Thelack of hard drives or active cooling also reduces component breakdown rate. With justthe need for basic components including CPU, motherboard and memory, thin clientscan be assembled inside small cases. Small casing allows thin client attachments intovarious places, for example the rear of a monitor.
36
5.4. Current implementation overview
A system was built as a CERN library pilot in Kommeri’s thesis [1, Chapter 5]. Anoverview of the system can be seen in Figure 18, which is based on an illustration in[1]. This section is based on a description of the system from [1] and an evaluationthat was carried out during the project.
The current system is an LTSP-setup built on top of Xen-virtualization. Xen hoststhree different virtualservers for thin clients; the access server handles the thin clientsconnection and boot up sequence, the application server hosts the desktop environ-ment, and the printing server handles printer queues. All installations run LinuxUbuntu Feisty distribution. The access server is a standard LTSP installation. Theapplication server has Xfce4 desktop environment installed with basic software like aWWW-browser and office suite. [1]
The access server handles the thin clients bootup by providing DHCP services andTFTP images. Clients get a small image by request via the network card PXE fea-ture when they start. This image is used as the OS for the thin client and the imagecontains a Linux kernel and X11 server with minimal configuration to establish theSSH-tunneled X11 connection to the application server. Images are also configuredso that they mount a root-partition via a network block device (NBD) from the accessserver. [1]
Current thin clients are connected using Ethernet on a private VLAN (Virtual LAN)with a server, in contrast to using a wireless link as stated in Kommeris’ thesis [1].The switch has VLAN configuration made by CERN-IT as a one-time favor. Theswitch is unsupported and it is stated during evaluation that if the switch configurationis corrupted or erased, there is no guarantee that it will be restored. Figure 18 showsthe network structure in that section, and the switch handling VLANs where the thinclient is connected to a private VLAN and the server is connected to both networks.
Figure 18. Current system overview in CERN library.
37
5.5. Problems
The current system suffers from hardware aging, as the server’s hardware is becomingunstable. Hardware instability dramatically increases the need for maintenance andhinders the usability of the whole system. Server performance is slow by modernstandards. Hardware slowness by current standards does not seem to be an issue interms of daily use.
A special network configuration was installed for the system [1], which is not offi-cially supported by CERN-IT. Mr. Tullio described CERN as a fast-paced community,where the staff turnover is high [42]; there is a high risk that everything that is not sup-ported by regular staff is forgotten, and anyone who knows about the network setupcould have already left CERN when a problem appears. This makes the network setupharder to resolve if there is problem.
The lack of an appointed administrator for the system is also a problem in the cur-rent system. The system itself does not require a large amount of maintenance, butall systems should have an administrator who understands them and is available whenproblems occur. Problems with system are not automatically noticed and they are usu-ally reported by users to librarians, or by library staff to anybody who could possiblyfix the identified problem.
38
6. PROPOSED SOLUTION
The CERN Library requires quiet public terminals with anonymous access. The LTSPsystem currently in use suffers from aging hardware and needs an upgrade path. Theprivate network that has been developed is also identified as a potential risk becauseCERN-IT doesn’t officially support setup. A solution for these issues is to upgrade thethin client system in the following way:
• Move thin client servers into CERN-IT’s private cloud as virtual desktops,
• Change the desktops to CERN Scientific Linux,
• Utilize the CERN network infrastructure to provide a startup mechanism for thinclients.
With these changes we can eliminate aging hardware and network risk. Changingthe desktop OS to CERN’s own Linux-distribution eases up upgrading and providesbetter support for OS inside CERN.
The proposed solution uses the HyperV-platform provided by CERN-IT to run vir-tual hosts. Public terminals act as thin clients and connect to virtual hosts as a desktop.The virtual hosts run Linux desktops to which public terminals connect from the librarypremises. Public terminals use modified tinycorelinux -Linux distribution, which theyreceive from CERN-IT’s PXE server during start up. Public terminals use an NX con-nection to provide a desktop from the virtual machine to the user. All virtual hosts sendsystem messages to a separate syslog-server. The syslog server is dedicated to moni-toring and collecting log-messages. An overview is presented below in Figure 19.
Thin Client
DHCP/PXE
Virtual Desktops
CERN LAN
DHCP/PXE
NX
Connectio
n
Syslog server
syslog
Figure 19. Overview of proposed public terminal setup.
Each thin client has its own virtual machine to use. By using separated virtualmachines we can reduce the competition of physical resources between thin clients.
39
Resources can be allocated from different physical hosts. A separated virtual machineis installed to collect syslog-messages and monitor virtual desktops using Nagios1.Syslog is defined in RFC 31642 as a standard way of defining system messages forlogging. The UDP transport of syslog messages is defined in RFC 54263 to transfersyslog messages to a remote machine in order to log events. Nagios is a widely usedde facto monitoring tool and it allows defining complex monitoring rules per device ordevice-group. We use Nagios to check if the virtual hosts SSH daemon responds, andsend a warning message if it does not.
Desktops themselves are configured to allow a passwordless login to a desktop onlywith an account named "library". An account is restored and cleared when the accountis logged out. The account’s home directory resides under automounter control andautomounter mounts tmpfs-volume as a home directory when it is accessed. Auto-mounter detaches the home directory if it has not been accessed in a defined timeoutperiod. The account is automatically logged out when the desktop has been idle forfive minutes (without mouse or keyboard movement) and all user processes are killedon logout.
6.1. Thin client
Thin clients are used to connect users and virtual desktops, and they use modifiedtinycorelinux images. The boot image is received from a network using the PXE pro-tocol when the thin client is started. The boot image contains a kernel and initrd toget a basic system running and a connection established. The image contains a basictinycorelinux4 mini Linux distribution with extra X11 packages. NX client binariesare installed in images with dependencies to enable an NX connection to virtual desk-tops. NX client configuration contains virtual machines and a configuration to open aconnection to virtual host KDM for a login screen instead of directly opening a usersession. The initrd image contains SSH keys from virtual machines with configura-tions to connect them. Virtual servers’ public SSH keys are pre-loaded into thin clientimages so the NX connection do not need confirmation from a user about unknownkey.
The thin client system does not need any NFS or any other network mounting itstarts an NX connection to one of the virtual machines with minimal root. The thinclient and virtual desktops are paired using DNS names. The thin client resolves itsDNS name during startup and identified the correct virtual host to connect to from thatname. The system has a fallback virtual desktop to use if a name cannot be resolvedaccording to the simple configuration that is defined.
The thin client boot image is constructed from the TinyCore-version of tinycore-linux. NX client and Xorg requires additional libraries to be included into its image.Default X was changed to Xorg to get better hardware support and faster 2D perfor-mance. Startup was modified to skip the checking of restore and resume informationand to use the NX client with a certain profile as the only application that starts in X11startup.
The desktop is stored in the /etc/sysconfig/desktop-file. The content of
1http://www.nagios.org2http://tools.ietf.org/html/rfc31643http://tools.ietf.org/html/rfc54264http://distro.ibiblio.org/tinycorelinux/welcome.html
40
this file contains only the starting of the nxclient with a pre-configure session. TinyCore starts the content of this file as a desktop when it is started. The NX client in afullscreen session is started when X comes up without windowmanager or any othersoftware. The NX session configuration contains parameters on what server to connectto, an encrypted password and user name, and what desktop to start on the server side.Each profile is a separate file.
Thin clients need to be registered in a CERN network database. The group or persondefined in the LANDB can administer images. An image needs to define the PXE boottarget for the thin client. Defining is performed using the aims2-client tool.
When the thin client boots up, a small basic operating system is loaded via networkbooting. The loaded OS contains an NX client. The OS configuration contains theopening of a graphical interface and connection to one of the virtual machines. Thevirtual machine to connect to is defined based on the thin clients DNS name, and thereis a fall back virtual machine if mapping does not work. The boot-up sequence andinteraction with DHCP/PXE servers and the virtual machine is illustrated in Figure 20.
Thin Client DHCP Server Virtual Machine
Power On
DHCP Discover
DHCP Offer (IP Address)
DHCP Offer (PXE server)
DHCP Request for IP Address
DHCP ACK for IP Address
PXE Boot Image request
PXE Boot Image
PXE Server
Start loaded image
Initiate SSH connection
Request Authentication
Provide Authentication
Start NX Client
NX Traffic inside SSH
Figure 20. Thin client boot-up sequence.
When a connection is made, the login display is shown. The user can login with a“library”-account. In addition, when the user is logged out, the connection is restarted,so both ’dummy’ and ’library’ accounts are cleared and returned to a clean state aftereach session.
41
6.2. Virtualization platform
HyperV was chosen as the virtualization platform. It is integrated into CERN-IT ac-count management and the virtual host control is easy to move from person to person.Currently, over 2000 virtual hosts5 are running inside the HyperV-cloud in CERN soit can be said to be widely used. HyperV installation had a SLC6 installation targetavailable, which was used as the basis of virtual desktop hosts.
6.3. Desktop OS, Scientific Linux CERN 6
The choice was made to use SLC6 (Scientific Linux CERN) Linux distribution. SLCis supported by CERN Linux support and is recommended for use in CERN comput-ers as Linux distribution. SLC is based on Scientific Linux6 and is based on Red HatEnterprise Linux distribution. SLC contains an SELinux framework and enables au-tomatic upgrade while a major version is supported. Support for SLC6 is due for afeature freeze in 2016 and will only provide security upgrades after 2017. [43]
The system takes a daily HTTP connection to the CERN-IT Linux software reposi-tory to check if any software in the system has an updated version available. If updatesare available, they are upgraded automatically.
User home directories reside in the tmpfs file system. Unionfs was considered,but it did not prove to be usable or stable with tmpfs during tests. Tmpfs mounting iscontrolled using automounter to handle home directories. Automounter mounts a freshtmpfs volume when a home directory is accessed and unmounts the volume when thehome directory has not been accessed for a defined time. The automounter timeoutwas set to 2 seconds. Timeout is small enough to have a fresh home directory when auser is logged out and logs back in.
Tmpfs is a file system residing in the virtual memory. The virtual memory it usesis cleared when the file system is unmounted. When mounted, clear memory area isallocated for tmpfs7.
Unionfs8 is a project to allow the mount union of multiple file systems, for exampleDVD-ROM in read-only mode and tmpfs with it as writing storage. It is used in liveCDLinux-distributions and similar setups.
Automounter9 is a system divided in a user space daemon and a file system modulein the kernel. The kernel mounts the directory as an automounter-type, and kernelautomounter module then informs the user space daemon if the defined directory isaccessed and is not yet mounted. The user space daemon maps the directory to thenormal mount point (for example NFS share) which is mounted. The automounterdaemon also keeps track of when the directory is accessed, so it can unmount thedirectory if a timeout is defined in the mapping.
KDM (KDE display manager) is used as a display manager to provide a login screento thin clients and start user sessions after login. KDM was chosen as it allows pass-wordless account login. GDM (Gnome display manager) has a feature that denieslogin if an account does not have a password, even if PAM is configured to allow a
5https://vmm.cern.ch/vmm/6http://www.scientificlinux.org/7http://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt8http://unionfs.filesystems.org/9http://www.am-utils.org/
42
passwordless account in displaymanager.
6.4. Security
Dummy and library accounts reside in the memory, which is mounted for use whenthe account is used (dummy when the thin client starts and connects to a virtual ma-chine, and library when an account is used to login) and unmounted when the accountlogs out. Library account login has been disabled, excluding NX desktop connection.When a library account is logged out, the system performs a few actions to make surethere is nothing left running before the next session starts. Library logout will kill alllibrary account processes. Killing all account processes makes it a little bit harder fora malicious user to leave unwanted processes running on a virtual machine. Killing isdone with xautolock.
Xautolock monitors X11 root window activity and enables dialog to be shown andrunning commands after the defined idle time for mouse or keyboard activity is reached.Xautolock sends dialog to the user 1 minute before killing all the users processes andlogging the user account out. Xautolock is run as the same UID so the user is able tokill it, but for its original purpose it is meant to automatically logout people out whohave forgotten to log themselves out and not as a securing and authentication tool.Running xautolock as a root would be beneficial to prevent it from being killed by auser.
Virtual desktops have a default SLC6 SELinux profile in enforced mode. SELinux isan NSA-developed kernel-level security framework to allow different security contextsand finer security control than normal POSIX ACL usually allows. Rootkit-hunter10 isinstalled on virtual images to check the filesystem periodically and report any abnormalfindings. Rootkit-hunter is a Linux tool which is designed to check the file system forknown rootkits by scanning known files that rootkits infect.
6.5. Availability
A library account does not have a password to ease up login from the display man-ager. Library and dummy accounts also have a minimum password age set to 100000days, i.e. minimum number of days until the password can be changed. This preventssomeone for changing the dummy-user password and hindering the system or remotelyexploiting a user account.
Virtual machines use a dummy user account (named “dummy”) for an NX connec-tion, and a library user account for login without the need to type a password from thinclients (named ’library’).
Thin clients connect separate virtual machines. The reasoning for this decision wasto minimize the distruption of resource hogging from one user. An example of resourcehogging would be flash running in a tight loop and taking up the whole of the CPUcore. Separate virtual machines also decrease the possibility of eavesdropping on otherthin client users.
Virtual desktops should be available in all normal daily cases. CERN-IT itself runsa basic service infrastructure on top of the HyperV platform. When the virtualiza-tion platform goes down, we assumed there would be critical problems in the whole
10http://rkhunter.sourceforge.net/
43
network. We cannot prepare a system that is usable in those cases.The syslog-server has Nagios11 daemon running. Nagios is configured to check
the virtual machine alive state by checking SSH-daemon responses. If the responsefails, Nagios sends an email about the failure to a defined admin-person/group. UsingNagios decreases the need for active monitoring of the server and a more proactiveapproach to administration.
6.6. Public Cloud
In comparison to in-house cloud use, we compare running costs of a DaaS (Desktop asa Service) system using a public cloud. The amazon EC2 IaaS-system was chosen forcomparison. We only checked the running costs of desktop images. The system stillneeds a network infrastructure to boot up thin clients and connect to the cloud.
Amazon EC2 bills the running time of a virtual machine (time from creation untiltime of termination) starting from USD 0.095 per hour. We only looked at the hostingcosts and did not try to estimate the whole cost of system. Latencies coming from thedistance between the hosted third party data center and thin clients are not consideredhere. [44]
Table 2 lists two systems used in comparison. The default system is the Amazondefault setup. The LTSP-system is based on the LTSP manual [45] to handle 40 simul-taneous users. Costs [44] for those systems are presented in Table 3.
Table 2. Two possible Amazon EC2 setups
default LTSPCore count 1 virtual 2 virtualRAM 1.7G 7.5GLocal storage 160G 850GArchitecture 32-bit 64-bit
Table 3 shows that the system needs to be implemented on a large scale to be costeffective. Buying virtualization from a third party would not save any money directlycomparing buying the PC hardware as majority of the cost of a computer is related tosoftware, support and maintenance costs. [46]
Table 3. Cost of using Amazon EC2-cloud as LTSP platform
1 year price (USD) 3 year contract (USD)Default 577.90 1401LTSP - 1400 setup + 4204
11http://www.nagios.org/
44
With virtualized servers savings can be achieved from electricity/cooling/locationcosts. Virtual desktops have centralized maintenance, but the system cannot survivenetwork outages between user and server. As the Microsoft VDI TCO whitepaperstates, there is a cost shift when using thin client and virtualization solutions, and itdepends overall setup if it saves money and is a viable solution. [46]
In the scope of this project, the use of a third party public cloud would not be fea-sible. CERN-IT services are used by the CERN library for normal desktop use andservers anyway. There is no cost saving for the CERN library in not using infrastruc-ture provided by CERN-IT.
46
Bandwidth is measured using Linux iptable’s firewall. The script reads bytecountersand resets the defined firewall rules at 1 second intervals. An interval is not preciselyone second; iptables reading and resetting takes 0.012-0.017 seconds and shell sleepcommand is not fully precise either. However, the level of precision is good enoughas the results are shown in second scale and variation in scale of ten milliseconds doesnot affect plots significantly.
Latency measurement is performed by measuring XDamage-events on redraws. Sec-tion 3.2 descries the principle of XDamage-events. XDamage states which parts of thescreen are redrawn and ready to be shown, so it suits to measure screen update laten-cies.
Xresponse-software is used to drive the test sequence and browser. Latencies aremeasured from the xresponse generated XDamage event log. The event log describesXDamage latency regarding given input events. Input events are for example mousedragging. Test driving is done by xresponse generating mouse drag events to a scrollbrowsed window from top to bottom and back.
Measurement data is processed with python script listed in appendix 1.4. The scriptparses the xresponse output log and calculates a few basic metrics on latency; aver-age and median latency with variance and test sequence speed with variance. Twoseparate scripts are used to plot latency values (appendix 1.2) and bandwidth usage(appendix 1.3): both scripts use gnuplot to generate EPS and PNG plots of inputdata.
Browsers have Flash and animated images disabled. Disabling animations and Flashprovides more reliable measurements instead of constant updates because of animatedads on the page.
47
8. REMOTE CONNECTION PROTOCOL TEST RESULTS
Different remote connection protocols were examined and reviewed for bandwidthefficiency and latency shown to user. The test results are divided into latency measure-ments and bandwidth use.
The test setup contained a 10/10 Mbps network connection. RTT (Roundtrip time)between the server and the client tested with ICMP ECHO messages was 2.6±0.5ms.The test was run on AMD Phenom II X4 955 Linux PC. The relative difference inresults should not be remarkable when using passive thin client CPU as the client.
The basic test run was to scroll a browser window ten times down and back up ina loaded webpage. The webpage used was http://www.hs.fi. It was loaded atthe same time for all browsers to get the same content. Measuring the window up-date latencies via XDamage events provides a good metric to see how fast the screenis updated from the user’s point of view. The test sequence contained ten repeats ofscrolling. The sequences can be seen in the result figures in all test cases. Testing wasperformed in a local display for reference latency, NX and SSH tunnelling for band-width and latency. A comparison with a local display is made based on measurements.
XDamage-event measurements were performed using xresponse from a maemo-repository1. Xresponse generates input events and measures response time using XDamage-events from X-server.
8.1. Latency test
Measurements were made with the browser already open. The browser had the html-page loaded and the page was scrolled during measurement. Flash and animated graph-ics were disabled on the browser so we did not need to filter out update events fromanimations. The script run test-sequence is presented in appendix 1.1.
Table 4 summarizes latencies between different connection methods. Latencies inthe NX connection were closer to local values than SSH-connection values. Latencyvalues remain under 100ms, so none of the connection protocol itself is too slow.Latencies are stable, which is expected in a short test run.
Table 4. Latency values for different connection methods
mean (ms) variance (ms) amount of sampleslocal 21.50 27.43 3095X11 32.67 25.09 2641
ssh+X11 32.76 24.35 1928NX 25.17 32.09 2643
Figure 22 shows the latency in the local display and Figure 24 shows latencychanges over a normal SSH connection. Figure 25 illustrates latency in the NX connec-tion during the test run. According to [23] all the methods are usable when measuringupdate speed of display.
1http://gitorious.org/+maemo-tools-developers/maemo-tools/xresponse
48
Figure 22. Screen update latency on local display during test run.
1
10
100
1000
0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Late
ncy (
ms)
Time (ms)
Update latency during testrun
Latency
Figure 23. Screen update latency on plain X11 display during test run.
Latencies between the plain X11 connection in Figure 23 and the SSH tunneled inFigure 24 are not notable. The difference is shown as plain X11 connection takeslonger to send XDamage-events than other connections. The long event sending timeindicates that not all the data has updated before the new test run starts. A closer lookat the raw xresponse log data shows that XDamage events are generated in a single
49
area of 16x17 located in a window bar which is updated regularly once every 50 ms.This update is related to a window-bar loading icon.
1
10
100
1000
0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Late
ncy (
ms)
Time (ms)
Update latency during testrun
Latency
Figure 24. Screen update latency on SSH tunneled display during test run.
NX and local display did not differentiate from each other in a remarkable way in themeasurements. The only fluctuation in latencies occurs in the NX connection on thefirst run, compared to the following runs shown in Figure 25. During the first test run,the NX connection consumes more bandwidth than the SSH connection. Latencies stillremain the same or below the SSH connection latencies in the first run. Comparisonscan be performed by comparing latency levels in figures 24 and 25.
8.2. Bandwidth usage
For bandwidth measurement, we used the iptables byte counters. Counters were polledevery second to measure the amount of transferred traffic. The script is listed in ap-pendix 1.1. Input/Output is from the X11 server, which is where physical display,keyboard and mouse are attached. Input is incoming traffic and Output is outgoingtraffic.
By comparing plain X11 and SSH-tunneled connections in figures 26 and 27, we cansee that normal SSH connection saves bandwidth. The reason for bandwidth saving islikely to be encryption which compresses traffic in this case.
NX bandwidth peaks higher in the beginning when comparing figures 27 and 28.After the first run, bandwidth usage drops in favor of NX connection. NX connectiontakes about 400kbps at the most in input, whereas SSH connection takes 1000kbps andpeaks at around 5000kbps.
Figure 28 visualizes how caching works in NX. During the first run the cache fills up.An adjacent runs utilizes the cache to increase performance. Compression is not used
50
1
10
100
1000
0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Late
ncy (
ms)
Time (ms)
Update latency during testrun
Latency
Figure 25. Screen update latency on NX-connection during test run.
0
500
1000
1500
2000
2500
3000
3500
4000
4500
00:00 00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50
kbps
Time (min:sec)
Bitrate on testrun
InputOutput
Figure 26. Bandwidth in kbits/second in plain X11 display during testrun.
fully in the LAN NX connection profile. The LAN profile in NX disabled protocolcompression and uses low compression for data encoding to decrease latency [22].
51
0
1000
2000
3000
4000
5000
00:00 00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50
kbps
Time (min:sec)
Bitrate on testrun
InputOutput
Figure 27. Bandwidth in kbits/second in SSH tunneled display during testrun.
0
500
1000
1500
2000
2500
3000
3500
00:00 00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50
kbps
Time (min:sec)
Bitrate on testrun
InputOutput
Figure 28. Bandwidth in kbits/second in NX connection during testrun.
52
9. DISCUSSION
The possibility of simply booting mini OS using PXE and including browsers and of-fice tools into the initrd image was not feasible; a browser, for example, requires tensof megabytes disk space, office tools require hundreds of megabytes. Graphical toolk-its like GTK increase disk space requirements even more. The initrd image would beunreasonably large to be feasible to load it over a network to boot into the ram disk.There were no mini distributions like that available with similar levels of administra-tion and update support that SLC6 has. Lack of in-house support would increase theadministrative burden for library staff in keeping the mini OS up-to-date with applica-tion security upgrades.
During the project, three different remote desktop protocols were evaluated. Spicewas left out in middle of the project. The reason for dropping Spice was that CERN-ITdid not yet offer KVM virtualization as a normal service and they did not have Spicesupport in KVM virtualization.
Remote desktop protocols have also evolved from basic X11 network transparencyto clever agents in NX or totally new protocols like Spice. Combining an efficient,low latency remote connection method with an efficient and inexpensive virtualizationsolution are key components in the DaaS solution. Nowadays, both inexpensive vir-tualization and efficient remote connection protocols are readily available. Networkspeeds are also up to speed with connection requirements and latencies in normal useshould no longer be major issue.
If the same project were to be started again, KVM and Spice may be more readyfor evaluation. Even if thinstation thin-client OS looked dead during evaluation, newversion was released just before Christmas and would also be interesting to evaluate.KVM as a virtualization platform was not available for deployment testing, so Spiceand KVM were left out of the evaluation.
An alternative solution would have been upgrading the current server hardware. Thiswould have been the easiest migration. The same problems would have come up againin a few years, when the upgraded hardware will have aged. Calculations were madeto compare how much it would cost to host a desktop in a public cloud. Public cloudcosts are presented in section 6.6.
Another option for the thin client image could have been the use of thinstation distri-bution1. At the time of design thinstation distribution did not look very active and wasnot chosen for further evaluation. LTSP PXE images were not used, as in the designwe wanted to separate desktops and did not want to use network based root-partitions.
Read-only disk images were evaluated at OS-level. Read-only would have beenpossible to circumvent if an attacker could gain a root in the system. The auto-updatingprocess would be hindered with a read-only root, and would have required a patch inthe auto-updating process itself. For these reasons, read-only images were not enabled.KVM would have offered a feature for using overlay images, where only changeswould be stored in a tmpfs file, for example. We could have had the option to use justone basic image for all virtual desktops, which would help when making changes inthe system.
1http://thinstation.org/
53
9.1. Regressions in proposed system compared to current system
The proposed system contains some problems and regressions comparing to currentsystems, being are mainly decentralized administration and slowness of adding newthin clients.
Decentralized administration comes from the use of the organization’s own virtualmachine for every thin client. We reduced the need for administration by using SLCautomatic updates. Automatic updates are checked daily from the CERN-IT Linuxrepository and upgraded automatically. Library staff, having administration responsi-bility, can contact CERN’s Linux support for most problems regarding basic OS. Theonly excluded, unofficially supported component is NX usage. CERN-IT does notprovide support for NX use or the thin client system as a whole.
If there is a need for changes to be made, all virtual machines need to be changedindividually. This also relates to the fact that the Hyper-V platform does not supportrunning clone-instances of one image, which would have allowed us to setup a systemthat would only need to be restarted to be fully updated.
Adding new thin clients is slower than the current system, as adding requires reg-istering the thin client to the CERN-IT network instead of just plugging them into aswitch. In addition, a virtual host should be configured for a new thin client.
9.2. Migration from current system to proposed solution
Taking the proposed system in use requires moving thin clients from the private net-work directly into the CERN LAN. Moving a thin client to a different network requireschanging the network configuration. A minimal change to accomplish this would beto move it to take out the current server between the thin client network and the CERNLAN.
Thin clients also require registration in the CERN network database. The librarye-group should be defined as the main user. The individual looking after the thin clientsystem should be defined as the responsible person in the database.
9.3. Test results
The NX connection proved to be a more efficient connection method than the plainSSH tunnelled X11 connection when measuring bandwidth and latency. The differ-ences in tests were not large, and testing video performance, for example, would prob-ably have shown larger differences.
According to out measurements, NX connection has the highest variance in latency.Variance can be explained by looking at the latency visualization in Figure 25. Thefirst test run has higher latencies than later runs. The SSH and local display does notshow a similar latency change. Increased latency occurs at the same time as the NXfills up the cache. A similar peak can be seen in bandwidth during the first run inFigure 28.
54
10. CONCLUSIONS
In this master’s thesis, a new thin client system is proposed for used for public termi-nals in the CERN library. The thin client system design was based on an earlier LTSPsystem that was in use and needed upgrading. The design selection included a choiceof virtualization platform, and the available choices were presented as virtualizationbasics. KVM and HyperV were presented briefly.
Remote connection protocols and latency were presented as background informationon how the proposed solution was measured and how latency and bandwidth weredefined as metrics.
The current system was presented for comparison and the starting point for migra-tion. The chapter on the current system provides context on the evolution of publicterminals in the CERN library and the requirements they have for terminals. The prob-lems with the current system were specified so that they could be addressed in the newsystem design.
The benefits of using the NX protocol was also within the scope of the research, andmeasurements were made to compare local display latencies with the NX connectionand the SSH tunnelled X11 connection.
Upgrading the thin client system to use more provided network infrastructure waspossible and feasible. The new system was designed and created and it is currentlyunder migration. In the proposed solution, the designed thin client system is described.
The goal of this Master’s thesis was to find a feasible upgrade path for the currentsystem, and that goal was successfully fulfilled.
55
Bibliography
[1] Kommeri J. (2009) System Management in Server Based Computing with Vir-tualization. Master’s thesis, Helsinki University of Technology, Department ofElectrical and Communications Engineering, Networking Laboratory.
[2] Goldberg R.P. (1973) Architectural principles for virtual computer systems.Ph.D. thesis, Harvard University.
[3] Bovet D. & Cesati M. (2002) Understanding the Linux Kernel, Second Edition.O’Reilly & Associates, Inc., Sebastopol, CA, USA, 2 ed.
[4] Tulloch M. (2010) Understanding Microsoft Virtualization Solutions, From theDesktop to the Datacenter. Microsoft Press, 2nd ed.
[5] Barham P., Dragovic B., Fraser K., Hand S., Harris T., Ho A., Neugebauer R.,Pratt I. & Warfield A. (2003) Xen and the art of virtualization. SIGOPS Oper.Syst. Rev. 37, pp. 164–177.
[6] Adams K. & Agesen O. (2006) A comparison of software and hardware tech-niques for x86 virtualization. In: Proceedings of the 12th international confer-ence on Architectural support for programming languages and operating systems,ASPLOS-XII, pp. 2–13.
[7] Agesen O., Garthwaite A., Sheldon J. & Subrahmanyam P. (2010) The evolutionof an x86 virtual machine monitor. SIGOPS Oper. Syst. Rev. 44, pp. 3–18.
[8] Waldspurger C.A. (2002) Memory resource management in vmware esx server.SIGOPS Oper. Syst. Rev. 36, pp. 181–194.
[9] Jones M. (2007), Discover the linux kernel virtual machine. URL: http://www.ibm.com/developerworks/linux/library/l-linux-kvm/,referenced 26.01.2012.
[10] Kivity A., Kamay Y., Laor D., Lublin U. & Liquori A. (2007) kvm: the linuxvirtual machine monitor. In: OLS ’07: The 2007 Ottawa Linux Symposium, pp.225–230.
[11] Russell R. (2008) virtio: towards a de-facto standard for virtual i/o devices.SIGOPS Oper. Syst. Rev. 42, pp. 95–103.
[12] KVM (2011), Virtio - KVM. URL: http://www.linux-kvm.org/page/Virtio, referenced 11.10.2011.
[13] Warnke R. (2010) qemu-kvm & libvirt. Books on Demand GmbH, Norderstedt,4th ed.
[14] Stabellini S. (2011), Linux 2.6.37: first upstream linux kernel to workas dom0. URL: http://blog.xen.org/index.php/2011/01/14/linux-2-6-37-first-upstream-linux-kernel-to-work-as-dom0/,referenced 2012.01.17.
56
[15] Xen (2008) Xen Users’ Manual v3.3. URL: http://bits.xensource.com/Xen/docs/user.pdf.
[16] Vuori J. (2011) Windows azure platform overview. In: Pilvipalveluiden lisäarvoPK-sektorin yrityksille, p. 14.
[17] Vaquero L.M., Rodero-Merino L., Caceres J. & Lindner M. (2008) A break inthe clouds: towards a cloud definition. SIGCOMM Comput. Commun. Rev. 39,pp. 50–55.
[18] Nieh J., Yang S.J. & Novik N. (2000) Comparison of thin-client computing ar-chitectures. Tech. rep., Network Computing Laboratory, Columbia University.
[19] Gettys J., Karlton P.L. & McGregor S. (1990) The X Window System, Version 11.Tech. rep., Digital Equipment Corporation, Silicon Graphics Computer Systems.
[20] Nave I., David H., Shani A., Laikari A., Eisert P. & Fechteler P. (2008)Games@Large Graphics Streaming Architecture. In: Proceedings of the 12th In-ternational Symposium on Consumer Electronics (ISCE2008)), Algarve, Spain,pp. 1–4. ISCE 2008.
[21] Red Hat Inc., Spice - home page. URL: http://www.spice-space.org/,referenced 22.11.2011.
[22] NoMachine, NoMachine NX - Support:Documentation. URL: http://www.nomachine.com/documentation/NX-XProtocolCompression.php, referenced 14.11.2011.
[23] Miller R.B. (1968) Response time in man-computer conversational transactions.In: Proceedings of the December 9-11, 1968, fall joint computer conference, partI, AFIPS ’68 (Fall, part I), pp. 267–277.
[24] Scheifler R.W. & Gettys J. (1986) The x window system. ACM Trans. Graph. 5,pp. 79–109.
[25] Barret D. & Silverman R. (2005) SSH, the secure shell: the definitive guide.O’Reilly.
[26] Packard K. & Anholt E. (2007) The DAMAGE Extension, Protocol Version 1.1.
[27] Massey B. & Sharp J. (2001) Xcb: an x protocol c binding. In: Proceedings ofthe 5th annual Linux Showcase & Conference - Volume 5, USENIX Association,Berkeley, CA, USA, pp. 24–24. URL: http://dl.acm.org/citation.cfm?id=1268488.1268512.
[28] Packard K. & Gettys J. (2003) X window system network perfor-mance. In: USENIX Annual Technical Conference, FREENIX Track,pp. 207–218. URL: http://www.usenix.org/events/usenix03/tech/freenix03/packard.html.
[29] Ylönen T. & Lonvick C. (2006), The Secure Shell (SSH) Transport Layer Proto-col. RFC 4253. URL: http://www.ietf.org/rfc/rfc4253.txt.
57
[30] Pane B. (1995), The DXPC - Differential X Protocol Compression project. URL:http://www.vigor.nu/dxpc/, referenced 14.11.2011.
[31] Danskin J.M., Zhang Q. & Abrahams-Gessel D.M. (1995) Fast higher bandwidthx. In: ACM Multimedia 94, Second ACM International Conference on Multime-dia, ACM, pp. 15–20.
[32] Red Hat Inc. (2009), Red hat makes move that will allow open collaboration withpartners to drive virtualization innovation. Press release. URL: http://www.redhat.com/about/news/prarchive/2009/spice-os.html, ref-erenced 23.11.2011.
[33] Spice Project (2009) Spice remote computing protocol definition v1.0. Draft 1ed. URL: http://spice-space.org/docs/spice_protocol.pdf.
[34] Spice Project (2009) Spice for Newbies. Red Hat Inc., draft 2 ed. URL: http://www.spice-space.org/docs/spice_for_newbies.pdf.
[35] Dengate D. (2008), Aims2: Automated installation management system. presen-tation held for CERN IT. URL: https://twiki.cern.ch/twiki/pub/LinuxSupport/Aims2/aims2.ppt.
[36] CERN, Cern | it department | about us. URL: http://it-dep.web.cern.ch/it-dep/what-we-do/, referenced 19.01.2012.
[37] Droms R. (1997), Dynamic Host Configuration Protocol. RFC 2131. URL:http://www.ietf.org/rfc/rfc2131.txt.
[38] Sollins K. (1992), THE TFTP PROTOCOL (revision 2). RFC 1350. URL:http://www.ietf.org/rfc/rfc1350.txt.
[39] Henry, M and Johnston, M. (1999) Preboot eXecution Environment (PXE) Spec-ification version 2.1. Intel.
[40] Johnston M. & Venaas S. (2006), Dynamic Host Configuration Protocol (DHCP)Options for the Intel Preboot eXecution Environment (PXE). RFC 4578. URL:http://www.ietf.org/rfc/rfc4578.txt.
[41] CERN, Cern | scientific information service | mission statement. URL: http://library.web.cern.ch/library/WhoWeAre/Mission.html, refer-enced 7.12.2011.
[42] Basaglia T. (2011). Personal interview.
[43] Linux-support C. (2012), SLC6: Scientific Linux CERN 6. URL: http://linux.web.cern.ch/linux/scientific6/, referenced 22.02.2012.
[44] Amazon (2011), Amazon Elastic Compute Cloud (Amazon EC2). URL: http://aws.amazon.com/ec2/, referenced 18.10.2011.
[45] Balneaves S., Erickson J., Giraldeau F., Johnson R., Johnston D., Liebow C.,McQuillan J., Mueller J., Romm G., Sass J., Shepheard R., Stewart S., Tilma B.,Assche D.V. & Wiebe C. (2009) Linux Terminal Server Project Administrator’sReference: A Guide to LTSP Networks. 0.99 edition.
58
[46] Barna B. (2010) VDI TCO Analysis for Office Worker Environments.Whitepaper, Microsoft. URL: http://download.microsoft.com/download/7/9/A/79AAA903-25B4-4D76-8580-BC47D5700433/MicrosoftVDITCOwhitepapercustomerreadyv12.pdf.
59
APPENDIXAppendix 1 Scripts used in latency and bandwidth measurementAppendix 2 Serverside modifications for ltsp-setup
APPENDIX 1/1
Scripts used in latency and bandwidth measurement
Listing 1.1. mittaukset/mittaa.sh# ! / u s r / b i n / env bash
i f [ [ $# −ne 2 | | ! " $1 " −g t 0 ] ]then
echo " u s a g e s $0 <number o r r e p e a t s > <name of measurement >"e x i t 1
f i
# i p t a b l e s b inary , i p t a b l e s or i p 6 t a b l e sIPTABLES=" / u s r / s b i n / i p 6 t a b l e s "# I n p u t r u l e l o c a t i o n , c h a i n name and row numberINPUT="INPUT 1 "# Outpu t r u l e l o c a t i o n , c h a i n name and row numberOUTPUT="OUTPUT 1 "
echo " K l i k k a a t e s t i −i k k u n a a "windowdata=$ ( x w i n i n f o )
g e o m e t r i a =$ ( echo " $windowdata " | g r ep geomet ry | c u t −c 12−)s i z e =$ ( echo $ g e o m e t r i a | c u t −f1 −d "+" )wid th =$ ( echo " $windowdata " | g r ep " Width " | awk ’{ p r i n t $2 } ’ )h e i g h t =$ ( echo " $windowdata " | g r ep " He i gh t " | awk ’{ p r i n t $2 } ’ )x p o s i t i o n =$ ( echo " $windowdata " | g r ep " A b s o l u t e upper− l e f t X" | awk ’{ p r i n t $4 } ’ )y p o s i t i o n =$ ( echo " $windowdata " | g r ep " A b s o l u t e upper− l e f t Y" | awk ’{ p r i n t $4 } ’ )windowid=$ ( echo " $windowdata " | g r ep " i d : " | c u t −f4 −d " " )
l e t x s c r o l l b a r =" $ x p o s i t i o n +$width−6"l e t y s c r o l l b a r =" $ y p o s i t i o n +137 "l e t y b o t t o m p o s i t i o n =" $ y p o s i t i o n + $ h e i g h t −35"t e s t s e q u e n c e ="−d ${ x s c r o l l b a r } x$ { y s c r o l l b a r }−${ x s c r o l l b a r } x$ { y b o t t o m p o s i t i o n }∗30+100
−d ${ x s c r o l l b a r } x$ { y b o t t o m p o s i t i o n }−${ x s c r o l l b a r } x$ { y s c r o l l b a r }∗30+100 "
# Take t i m e s t a m p t o v a r i a b l et imes t amp = ‘ d a t e +"%Y.%m.%d−%H:%M" ‘
# Zero i n p u t c o u n t e rsudo $IPTABLES −L $INPUT −Z
# c r e a t e empty f i l e t o i n f o r m l o o p s when t o s t o pt o u c h $ {2} _ r u n n i n g
# run measurement loop u n t i l $ {2} _ r u n n in g f i l e i s removedwhi le [ −f $ {2} _ r u n n i n g ]do
echo −n $ ( d a t e +"%s .%N" ) " " ; sudo $IPTABLES −L $INPUT −x −v −n −Z | head −1 | awk ’{p r i n t $1 , $2 ∗8} ’
s l e e p 1done > $ {2} _$ { t imes t amp } _ i n p u t . t x t &
# Zero o u t p u t c o u n t e rsudo $IPTABLES −L $OUTPUT −Z
# run measurement loop u n t i l $ {2} _ r u n n in g f i l e i s removedwhi le [ −f $ {2} _ r u n n i n g ]do
echo −n $ ( d a t e +"%s .%N" ) " " ; sudo $IPTABLES −L $OUTPUT −x −v −n −Z | head −1 | awk ’{p r i n t $1 , $2 ∗8} ’
s l e e p 1done > $ {2} _$ { t imes t amp } _ o u t p u t . t x t &
APPENDIX 1/2
x r e s p o n s e −c ${ x s c r o l l b a r } x$ { y s c r o l l b a r } −w 1 −i d ${ windowid }f o r i i n ‘ seq $1 ‘ ; do
x r e s p o n s e −w 10 −i d ${ windowid } ${ t e s t s e q u e n c e }done > $ {2} _$ { t imes t amp } _ v i i v e e t . t x t ;
rm $ {2} _ r u n n i n g
# c a l c u l a t e mean , v a r i a n c e and measurement s i z e w i t h py th on s c r i p tpy thon . / t e s t i l a s k u r i . py < $ {2} _$ { t imes t amp } _ v i i v e e t . t x t
sh . / p l o t t a a . sh $ {2} _$ { t imes t amp } _ v i i v e e t . t x t $ {2} _$ { t imes t amp } _ l a t e n c ysh . / p l o t t a a _ b w . sh $ {2} _$ { t imes t amp } $ {2} _$ { t imes t amp } _bandwid th
#show r e s u l t sd i s p l a y $ {2} _$ { t imes t amp } _ l a t e n c y . png &d i s p l a y $ {2} _$ { t imes t amp } _bandwid th . png &
Listing 1.2. mittaukset/plottaa.sh# ! / u s r / b i n / env bash
g n u p l o t <<EOFs e t y l a b e l " La tency ( ms ) "s e t x l a b e l " Time ( ms ) "s e t l o g ys e t t i t l e " Update l a t e n c y d u r i n g t e s t r u n "p l o t "< g rep damage $ {1} " u 1 : 3 w i m p u l s e s t i t l e " La t ency "xmin = GPVAL_DATA_X_MINs e t t e rm p o s t s c r i p t eps c o l o u r enhanceds e t o u t p u t " $ { 2 } . eps "p l o t "< g rep damage $ {1} " u ( \ $1−xmin ) : 3 w i m p u l s e s t i t l e " La t ency "s e t t e rm png t r u e c o l o r s i z e 1280 ,1024s e t o u t p u t " $ { 2 } . png "p l o t "< g rep damage $ {1} " u ( \ $1−xmin ) : 3 w i m p u l s e s t i t l e " La t ency "EOF
Listing 1.3. mittaukset/plottaa_bw.sh# ! / u s r / b i n / env bash
g n u p l o t <<EOFs e t y l a b e l " kbps "s e t x l a b e l " Time ( min : s e c ) "s e t t i t l e " B i t r a t e on t e s t r u n "s e t x d a t a t imes e t t i m e f m t "%s .%N"p l o t " $ {1} _ i n p u t . t x t " u 1 : ( \ $3 / 1 0 0 0 ) w l i n e s t i t l e " I n p u t " , " $ {1} _ o u t p u t . t x t " u 1 : ( \
$3 / 1 0 0 0 ) w l i n e s t i t l e " Outpu t "xmin = GPVAL_DATA_X_MINs e t t e rm p o s t s c r i p t eps c o l o u r enhanceds e t o u t p u t " $ { 2 } . eps "p l o t " $ {1} _ i n p u t . t x t " u ( \ $1−xmin ) : ( \ $3 / 1 0 0 0 ) w l i n e s t i t l e " I n p u t " , " $ {1} _ o u t p u t . t x t
" u ( \ $1−xmin ) : ( \ $3 / 1 0 0 0 ) w l i n e s t i t l e " Outpu t "s e t t e rm png t r u e c o l o r s i z e 1280 ,1024s e t o u t p u t " $ { 2 } . png "p l o t " $ {1} _ i n p u t . t x t " u ( \ $1−xmin ) : ( \ $3 / 1 0 0 0 ) w l i n e s t i t l e " I n p u t " , " $ {1} _ o u t p u t . t x t
" u ( \ $1−xmin ) : ( \ $3 / 1 0 0 0 ) w l i n e s t i t l e " Outpu t "EOF
Listing 1.4. mittaukset/testilaskuri.py# ! / b i n / env py t ho n# −∗− e n c o d i n g : u t f −8 −∗−
import s y s
def median ( l i s t a ) :
APPENDIX 1/3
l i s t a = s o r t e d ( l i s t a )p i t u u s = l e n ( l i s t a )i f not p i t u u s :
re turn 0i f not p i t u u s %2:
re turn ( l i s t a [ ( p i t u u s / 2 ) −1]+ l i s t a [ ( p i t u u s / 2 ) ] ) / 2e l s e :
re turn l i s t a [ p i t u u s / 2 ]
def e x p e c t a t i o n v a l u e ( l i s t a ) :i f l e n ( l a t e n c i e s ) :
re turn sum ( l i s t a , 0 . 0 ) / l e n ( l a t e n c i e s )re turn 0
def v a r i a n c e ( l i s t a ) :i f l e n ( l i s t a ) <= 1 :
re turn 0 ,0k e s k i a r v o = e x p e c t a t i o n v a l u e ( l i s t a )v a r i a n c e = 0 . 0f o r l uku in l i s t a :
v a r i a n c e += ( ( l uku − k e s k i a r v o ) ∗∗2) / ( l e n ( l i s t a ) −1)v a r i a n c e = v a r i a n c e ∗∗ 0 . 5re turn v a r i a n c e , k e s k i a r v o
l a t e n c i e s = [ ]sums = [ ]
i n t e r m e d i a t e s u m s = [ ]f o r l i n e in s y s . s t d i n . r e a d l i n e s ( ) :
i f l i n e . f i n d ( ’ S t a r t u p ’ ) != −1 and l e n ( i n t e r m e d i a t e s u m s ) :sums . append ( sum ( i n t e r m e d i a t e s u m s ) )i n t e r m e d i a t e s u m s = [ ]
i f l i n e . f i n d ( ’ damage ’ ) == −1:c o n t in u e
( t imes tamp , l a t e n c y , e v e n t ) = l i n e . s p l i t ( ’ : ’ )l a t e n c y = i n t ( s t r . s t r i p ( l a t e n c y ) [ : −2 ] )i f l a t e n c y > 0 :
l a t e n c i e s . append ( l a t e n c y )i n t e r m e d i a t e s u m s . append ( l a t e n c y )
sums . append ( sum ( i n t e r m e d i a t e s u m s ) )i n t e r m e d i a t e s u m s = [ ]
k e s k i h a j o n t a , k e s k i a r v o = v a r i a n c e ( l a t e n c i e s )p r i n t " u p d a t e l a t e n c y %f±%fms median :%fms sample s i z e :%d " %( k e s k i a r v o , k e s k i h a j o n t a ,
median ( l a t e n c i e s ) , l e n ( l a t e n c i e s ) )p r i n t " s e q u e n c e speed %f±%fms median :%fms sample s i z e :%d " %( v a r i a n c e ( sums ) [ 0 ] ,
v a r i a n c e ( sums ) [ 1 ] , median ( sums ) , l e n ( sums ) )
APPENDIX 2/1
Serverside modifications for ltsp-setup
Following changes were made on default SLC6 installation on virtual image, so secureit up and setup ltsp login for nx.
Listing 2.1. mittaukset/ltsp–terminal–serverside
B a s i c s e t u p i s from h t t p : / / vmm. c e r n . ch SLC6 v i r t u a l machine wi th 2G ram and 2 cpuc o r e s . S e r v e r s a r e named i n s t y l e l i n u x−l t s p − { 1 . . . . } , c u r r e n t l y up t o 5 . As t h i n−c l i e n t s a r e named i n s t y l e o f l i n u x−t e r m i n a l − { 1 . . . } and b o o t i n s c r i p t w i l l mapsame number t e r m i n a l t o c o r r e c t l t s p .
i n s t a l l e d p a c k e t s a d d i t i o n a l t o b a s i c i n s t a l l a t i o nf l a s h −p l u g i nc h k r o o t k i tr k h u n t e rkdm@xfcex a u t o l o c k ( doesn ’ t come i n r e g u l a r r e p o s i t o r i e s , you can use c e n t o s / s c i e n t i f i c l i n u x
package w i t h o u t p rob lems )n x s e r v e r ( n x c l i e n t , nxproxy , n x s e r v e r f r e e v e r s i o n from h t t p : / / www. nomachine . com )
p a c k e t s t h a t I removed : ( sudo yum remove . . . )l i b v i r t −c l i e n tr advdw p a _ s u p p l i c a n tNetworkManagergdmf p r i n t d
For dummy and l i b r a r y −use rnames t h e r e s a u t o f s h a n d l i n g t h e i r homed i r s i n tmpfs :copy p r o v i d e d a u t o . home and a u t o . m a s t e r t o / e t cu s e r a d d dummy −> s e t passwd as ∗∗∗∗∗∗∗u s e r a d d l i b r a r y −> passwd −d l i b r a r y , a l s o passwd −n 100000 l i b r a r ymeans t h a t l i b r a r y passwd can ’ t be changed i n 100000 days , so nobody c o u l d changet h a t password t o some th ing e l s e
copy kdmrc m o d i f i c a t i o n t o kdmkdmrc −> / e t c / kde / kdm / kdmrc
Enab le xdmcp from l o c a l h o s t−−−echo 1 2 7 . 0 . 0 . 1 >> / e t c / kde / kdm / Xaccess−−−
S e t d e f a u l t d e s k t o p t o x f c e 4−−−echo PREFERRED= s t a r t x f c e 4 > / e t c / s y s c o n f i g / d e s k t o p−−−
Copy g e n e r a t e d nx−keys i n p l a c e ( n o t needed i f u s i n g d e f a u l t keys , b u t t h e y a r e n tt h a t good i d e a t o use ) .
cp node . l o c a l h o s t .∗ / u s r /NX/ e t c / keys /
Enab le r e q u e s t i n g xdmcp i n nx−s i d e :−−−echo EnableDirec tXdmQuery=" 1 " >> / u s r /NX/ e t c / s e r v e r . c f g−−−
Se tup l o g s t o g o t i n t o l t s p −s y s l o g s e r v e r−−−echo −e " ∗ . ∗ \ t@l t sp−s y s l o g . c e r n . ch " >> / e t c / r s y s l o g . con f−−−
APPENDIX 2/2
So you can l o g i n t o l t s p −s y s l o g−machine and l o g s from a l l s e r v e r s a r e i n / v a r / l i b /messages and / v a r / l i b / s e c u r e
Change tmp and / v a r / tmp f i l e s t o be i n tmpfs and noexec / nodev / n o s u i d f l a g s on , sou s e r s c a n t run any b i n a r y from them or c r e a t e / use d e v i c e nodes o r su id−b i t s :
−−−echo −e " tmpfs / tmp tmpfs d e f a u l t s , noexec , nodev
, n o s u i d 0 0 " >> / e t c / f s t a becho −e " tmpfs / v a r / tmp tmpfs d e f a u l t s ,
noexec , nodev , n o s u i d 0 0 " >> / e t c / f s t a b−−−
Deny a t and c ron usage f o r l i b r a r y and dummy u s e r s . Th i s way someone c o u l d bea b l e t o l e a v e some s c r i p t r u n n i n g on background o t h e r w i s e . Also d i s a b l e dummyu s e r l o g i n from s s h .
−−−echo " l i b r a r y " >> / e t c / c ron . denyecho "dummy" >> / e t c / c ron . denyecho " l i b r a r y " >> / e t c / a t . denyecho "dummy" >> / e t c / a t . denyecho " DenyUsers dummy" >> / e t c / s s h / s s h d _ c o n f i g−−−
Change r o o t k i t h u n t e r c o n f i g t o c o n t a i n p a s s w o r d l e s s l i b r a r y u s e r and . hmac f i l e a ssshd−c o n f i g .
cp r k h u n t e r . con f / e t c / r k h u n t e r . con f
/ e t c / kde / kdm / X s e s s i o n m o d i f i c a t i o n t o s t a r t x a u t o l o c k so we k i c k o u t i d l e−s e s s i o n s (add on s t a r t o f Xsess ion− f i l e ) :
−−−/ u s r / X11R6 / b i n / x a u t o l o c k −s e c u r e −t ime 5 −l o c k e r " / u s r / b i n / p k i l l −9 −u $USER" −n o t i f y
60 −n o t i f i e r " l o g g e r \ " N o t i f y i n g u s e r \ " && xmessage −c e n t e r −t i m e o u t 60 ’ IDLEWARNING: Auto l o g o u t i n 1 minu te . Do a n y t h i n g t o c a n c e l au to−l o g o u t . ’ " &
−−−remember t h e l a s t &
/ e t c / kde / kdm / X r e s e t i s run when u s e r l o g o u t s , so p u t t h e s e t h e r e so we c l e a n u pl i b r a r y −u s e r l e f t o v e r f i l e s and k i l l a l l t h e p r o c e s s e s :
−−−f i n d / tmp −u s e r l i b r a r y −exec rm −Rf {} \ ;−−−