z/architecture overview and z/vm 64-bit support -

IBM eServer™

© 2003 IBM Corporationz/VM 64-bit InternalsJune 3, 2003

Damian OsisekIBM z/VM [email protected]

z/Architecture Overview andz/VM 64-bit Support Internals

BayBunchMay 30, 2003

IBM eServer™

© 2003 IBM Corporation

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

The following are trademarks or registered trademarks of other companies.* Registered trademarks of IBM Corporation

* All other products may be trademarks or registered trademarks of their respective companies.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or bothPenguin (Tux) compliments of Larry EwingJava and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countriesUNIX is a registered trademark of The Open Group in the United States and other countries.Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.SET and Secure Electronic Transaction are trademarks owned by SET Secure Electronic Transaction LLC.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experiencevary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without noticConsult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibilitor any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Trademarks

z/VM 64-bit InternalsJune 3, 2003 64BIT_TM1

e-business logo*Enterprise Systems Architecure/390ESCON*FICON

IBM*IBM eServerIBM logo*Multiprise*S/390*

VM/ESA*z/Architecturez/OSz/VM*zSeries*

IBM eServer™


Agenda

Introduction64-bit ArchitectureImplications for VM

VM 64-bit SupportDesign GoalsConstraintsImplementationRestrictionsRelated Changes

Summary

64BIT_0010

IBM eServer™


64-bit Architecture

Architecture mode selected by softwareESA/390z/Architecture

z/Architecture major characteristics64-bit registers64-bit addresses and addressing modeNew instructionsFive-level DAT structure128-bit PSWsTwo-page prefix area

64BIT_0020

IBM eServer™


64-bit Registers

16 64-bit GRs64-bit addressing64-bit integer arithmetic

16 64-bit CRsFew architectural changes from ESA/390Some 64-bit exploitation

1, 7, 10, 11, 12, 13 and 15 support 64-bit addresses

64BIT_0030

IBM eServer™


64-bit Addresses

Larger real and virtual storage sizesGigabytes (GB) = 2**30Terabytes (TB) = 2**40Petabytes (PB) = 2**50Exabytes (EB) = 2**60

Additional addressing mode (64-bit)

64BIT_0040

IBM eServer™


Instruction set

New instructionsTo deal with new architectural components

AGR, LPSWETo convert between old and new formats

LLGTR, SLGF, STMHTo do things we've always wanted

RLLG, OILLESA/390 instructions continue to operate

Some are sensitive to new 64-bit addressing mode LA

Some new 32-bit instructionsRLL

64BIT_0050

IBM eServer™


Sample new instructions

Instruction Syntax Result

Load (32) L 1,FIELD uuuuuuuu FEDCBA98

Load (64) LG 1,FIELD FEDCBA98 76543210

Load (64 from 32) LGF 1,FIELD FFFFFFFF FEDCBA98

Load Logical (64 from 32) LLGF 1,FIELD 00000000 FEDCBA98

Load Logical Thirty-one Bits LLGT 1,FIELD 00000000 7EDCBA98

Insert Characters under Mask (high)

ICMH 1,X'F',FIELD FEDCBA98 uuuuuuuu

Load Multiple (32) LM 1,2,FIELD uuuuuuuu FEDCBA98uuuuuuuu 76543210

Load Multiple (64) LMG 1,1,FIELD FEDCBA98 76543210

Load Multiple High LMH 1,2,FIELD FEDCBA98 uuuuuuuu76543210 uuuuuuuu

uuuuuuuu = unchanged FIELD = FEDCBA98 76543210

64BIT_0060

IBM eServer™


Five-Level DAT Structure

Region Table Entry (RTE)64-bit entry holding Region-Table Origin or Segment-Table Origin

Region TableOne to four pages (512 to 2048 entries)Region First Table

Each entry points to Region Second TableRegion Second Table

Each entry points to Region Third TableRegion Third Table

Each entry points to Segment Table

64BIT_0070

IBM eServer™


Five-Level DAT Structure (continued)

Segment-Table Entry (STE)64-bit entry holding Page-Table OriginPage Table is 256 entries (maps 1M)

Page-Table Entry (PTE) 64-bit entry holding Page-Frame Real Address

64-bit virtual address structure:

SX PX BXRFX RSX RTX(11) (11) (11) (11) (8) (12)

11 524433220 63

64BIT_0080

IBM eServer™


Five-Level DAT Structure (continued)

Address-Space-Control Element (ASCE)Generic name for 64-bit designator; may be:

Segment-Table Designation (STD)Region-Table Designation (RTD) -- First, Second, or ThirdReal-Space Designation (RSD)

Segment Table maps 2048 1M segments (one region, 2G)Region Third Table maps 2048 regions (2K x 2G = 4T)Region Second Table maps 2048 RTTs (4M x 2G = 8P)Region First Table maps 2048 RSTs (8G x 2G = 16E)Self-distinguishing (type) and self-defining (length)

64BIT_0090

IBM eServer™


128-bit PSWs

High-order word largely unchanged from ESA/390Bit 12: must be 0 (cf. must be 1 in ESA/390)Bit 31: Extended-Addressing mode bit

Second word contains Basic-Addressing mode bitThird and fourth words contain 64-bit instruction address

Addressing modes:

PSW.31(EA)

PSW.32(BA)

Addressing mode

0 0 24-bit0 1 31-bit1 0 invalid1 1 64-bit

64BIT_0100

IBM eServer™


IE Prog E0R000TOX Key 0MWPASCCMask0000000A0 5 8 12 16 18 20 24 31

BA000000000000000000000000000000032 63

Instruction Address 64 95

128-Bit Program Status Word

Instruction Address (continued)

96 127

64BIT_0110

IBM eServer™


Two-Page Prefix Area

Prefix Area doubled in sizeTwo frames (8K) on 8K boundaryLogout area moved to second frameMade room for expanded interruption old/new PSWs

Separate locations for ESA/390 and z/Architecture PSWsNote: Software needs base register to address second frame

64BIT_0120

IBM eServer™


Implications for VM

Guest can switch architecture dynamicallyNo MACHINE statement supportPrepare in advance for potential guest z/Architecture switch

Guests need 64-bit real and 64-bit virtual storageMajor upheaval in storage management

Architectural concepts must be reflected in VM command languageConcern for compatibility (e.g., of scripted use)

64BIT_0130

IBM eServer™


Implications for VM (continued)

A single image could be huge256 system-owned devices probably inadequate

A single guest could be hugeScheduler algorithms may require some attention

Need to support more than 14 CPUsNeed to continue to deliver support for ESA/390 architecture processors

64BIT_0140

IBM eServer™


VM 64-bit Support

Design GoalsConstraintsImplementationRestrictionsRelated Changes

64BIT_0150

IBM eServer™


Design Goals

Add support for z/Architecture-mode guestsMinimize change to CPRun on both ESA/390 and z/Architecture hardwareMaintain compatibility where appropriateMaintain performance parity where feasibleEliminate deprecated function where feasibleDeliver with hardware

64BIT_0160

IBM eServer™


Constraints

Small teamSome large real exploitation deferred

Other features and functions requiredAggressive schedule

Tied to hardwareNeeded early hardware to test

64BIT_0170

IBM eServer™


Implementation

Significant changes inStorage managementGuest simulationVirtual SIE (vSIE)Minidisk CacheTRACEDISPLAY/DUMP/STOREI/ODump formatMajor control blocks

64BIT_0180

IBM eServer™


Implementation (continued)

Little or no change inSpoolXC-mode supportMost console functionsDevice supportInter-machine communicationUser DirectoryMost CP APIs (still 31-bit)

64BIT_0190

IBM eServer™


Implementation (continued)

Most CP code relies only on ESA/390 functionalitySelected modules exploit z/Architecture capabilities

64-bit registers and instructionsStorage above 2G64-bit addressing mode

Linkage changed to facilitate different register use stylesTwo builds, one source base

Conditional assemblyVariable assemblyMacro encapsulationModule clonesBuild-specific modules

64BIT_0200

IBM eServer™


Major Control Block Changes

Significant changes toPFXPGVMDBKPGMBKSIEBKSAVBKStorage-related architectural structures

64BIT_0210

IBM eServer™


z/Architecture Exploitation

Storage above 2G used forMinidisk cachePageable guest pagesVDISK pages (as of z/VM 4.4.0)

Not used forCP free storageCP pagesPreferred guest pages

65BIT_0220

IBM eServer™


z/Architecture Exploitation (continued)

Some use of 64-bit addressing mode64-bit registers used fairly extensivelyArchitectural level set allows use of Immediate and Relative instructions in both builds

zSeries9672 G5-G6Multiprise 3000NOT 3090, 4381, 9221, 9121, 9021, 9672 R1/R2/R3/G3/G4, MP2000,P/390, R/390

64BIT_0230

IBM eServer™


Above-2G Storage Management

Page faults resolved using above-2G frames if availableCP translation services (HCPTRANS) move page below 2G line to return 31-bit addressAll locked pages must reside below 2GAll CP data structures (including DAT tables, PGMBKs) must reside below 2G"Low" and "high" storage managed separately based on demandWhen storage below 2G constrained:

Existing mechanisms would page out to expanded storage first (if available), then to DASDIf Xstore constrained or not available, pages are migrated from below to above 2G (new in z/VM 4.3.0)If PGMBKs consume excessive storage below 2G, above-2G steal task driven to increase chance that PGMBKs can be paged out (new in 4.3.0)

64BIT_0240

IBM eServer™


Module Linkage

Module register use style may be of three types:SHORTREG - 32-bit onlyLONGREG - 32-bit and 64-bitFULLREG - build-dependent

Caller's registers must be saved/restored across callCallee has access to caller's registers in SAVBKLinkage must preserve caller's registers in callee-sensitive fashion

64BIT_0250

IBM eServer™


Module Linkage (continued)

Module generally uses one register styleDeclared in MDLATENT macro for module or entry pointMost modules are SHORTREG (default)

Might have entry points that use other stylesE.g., HCPSTK is SHORTREG, but entry points HCPSTKCG and HCPSTKFG are LONGREGIn this case

Must declare module register styleAssume majority of module runs in declared register styleNeed to exercise some care

64BIT_0260

IBM eServer™


Module Linkage (continued)

64-bit build SAVBK doubled in size to accommodate changesFlags in SAVBK define its formatNew SVGBK maps save area for FULLREG and LONGREG modulesLinkage macros require SAVBK/SVGBKThree cases to consider (in 64-bit build):

SHORTREG => SHORTREGLONGREG => SHORTREGanyREG => LONGREG (or FULLREG)

64BIT_0270

IBM eServer™


SHORTREG => SHORTREG

Header

Low-orderRegisters

Workarea

Reserved

SAVELARG

X'100'

X'80'

SAVBK

64BIT_0280

IBM eServer™


LONGREG => SHORTREG

Header

Low-orderRegisters

Workarea

Reserved

High-order Registers

SAVELARG+SAVERG64

X'100'

X'80'

X'C0'

SAVBK

64BIT_0290

IBM eServer™


AnyREG => LONGREG

Header

64-bitRegisters

Workarea

Reserved

Reserved

SVGLARG+SVGCREG+SVGRG64

X'100'

SVGBK

64BIT_0300

IBM eServer™


Conditional Assembly

HCPAIF (&HCPESAME) STMG R0,R5,PFXTRCSV Save registers used by trace HCPAELSE (&HCPES390) STM R0,R5,PFXTRCSV Save registers used by trace HCPAEND

New HCPAIF, HCPAELSE, HCPAEND macrosGlobal HCPOPTNS symbols define target build

&HCPES390 - 1 for 32-bit, 0 for 64-bit&HCPESAME - 0 for 32-bit, 1 for 64-bit

New maclib holds 64-bit build HCPOPTNS

64BIT_0310

IBM eServer™


15038 HCPAIF (&HCPESAME) 0000D4 EB05 0C28 0024 00C28 15039 STMG R0,R5,PFXTRCSV Save 15040 HCPAELSE (&HCPES390) 15041+* Start of Excluded Block ------------ 15042- STM R0,R5,PFXTRCSV Save 15045- HCPAEND 15048+* End of Excluded Block --------------

Conditional Assembly (continued)

Listing for either build shows all source code

64BIT_0320

IBM eServer™


Variable Assembly

Designed primarily for FULLREG modulesSmall set of variable symbols defined in HCPFULSY COPYAllow build-dependent symbols to be created

&G SETC 'G' (64-bit) - or - &G SETC '' (32-bit)

L&G R1,WORKADDR

64BIT_0330

IBM eServer™


Variable Assembly (continued)

Some use of &HCPES390/&HCPESAME in open code and macros

TM SAVEFORM,SAVELARG BC 1*&HCPES390+8*&HCPESAME,DIE

LA R5,GPRSIZE(,R3) Next reg ... GPRSIZE EQU 4*&HCPES390+8*&HCPESAME

AIF (NOT &HCPESAME).NOTME ... .NOTME ANOP ,

64BIT_0340

IBM eServer™


Macro Encapsulation

Many macros generate build-dependent codeOften parameter-drivenNew parameter notations introduced (e.g., for pointer to 64-bit value)

Some new macros introduced to ease build dependenciesE.g., HCPGREG

General-purpose macro to locate, extract, or set virtual machine general register contentsAccommodates guests with 32-bit or 64-bit registersUsed extensively in Diagnose support

64BIT_0350

IBM eServer™


Module Clones

Some functions are architecturally different in fundamental ways between two builds

E.g., address translationRather than use conditional assembly, created parallel modules ("clones")Modules included in Nucleus build determined by MDLATENT NUCLEUS= parameter

MDLATENT HCPPAS, MODATTR=(RES,MP,DYN), NUCLEUS=ESA390

64BIT_0360

IBM eServer™


Build-Specific Modules

Some functions required in only one buildE.g., >2G storage initialization

Target nucleus specified using MDLATENT macro

MDLATENT HCPISG, MODATTR=(RES,MP,STAT,LONGREG), NUCLEUS=ESAME

64BIT_0370

IBM eServer™


Other CP Changes for 64-bit Support

New STORAGE/MAXSTORAGE statements in User Directory (and in profiles)Indirect addressing constructs on DISPLAYPage Table Invalidation Lock use improvedNew DEFINE STORAGE CONFIGURATION command addedEliminated IPLable CP NucleusEliminated RIO370 areaExtensions to HCPUSING and HCPDROP

Allow use of named, dependent, and ranged USINGsExit-related variable symbols in System Config file

Allow build-dependent exit file selectionMonitor record extensions

64BIT_0380

IBM eServer™


HCPUSING/HCPDROP

CP's versions of USING and DROPExtended to allow dependent, labeled, and ranged USINGsAlso allow "named" DROPsOriginally because of SIEBK changes but generally useful

64BIT_0390

IBM eServer™


Dependent USINGs

HCPUSING VMDBK,R11 HCPUSING SIEBK,VMDSDSC,VMDBK TM SIEPSW1,PSWWAIT HCPDROP SIEBK...SIEBK DSECT...VMDBK DSECT...VMDSDSC DS CL512...

64BIT_0400

IBM eServer™


Labeled USINGs

HCPUSING VMDBK,R11NEW HCPUSING VMDBK,R7... CLC VMDUSER,NEW.VMDUSER... L R7,NEW.VMDCYCLE HCPDROP NEW

64BIT_0410

IBM eServer™


Ranged USINGs

HCPUSING (CACXSIDE,CACYSIDE),R7...CACBK DSECT...CACXSIDE DS 0X...CACYSIDE DS 0X...

64BIT_0420

IBM eServer™


Restrictions

No DCSS or NSS above 2GMost Diagnose functions not supported in guest 64-bit addressing mode; operands must be below 2G in guest storage. Exceptions:

Diagnose 8, 44, 4C, DC (as of z/VM 4.3.0)New 64-bit version of page-fault handshaking (PFAULT), exploited by LinuxDiagnose 4 supports host addresses above 2G

CP and preferred guests must fit in 2GPreferred guests may not exceed 2G-x total<<2G concurrently locked pages

64BIT_0430

IBM eServer™


Summary

CP in z/VM has evolved to accommodate z/ArchitectureHostsGuests

There is much more to doYour requirements help us select and prioritize our efforts

64BIT_0440

z/architecture overview and z/vm 64-bit support -

Documents