Request for Tender

Invitation to tender for Phase 1 of the eX³ project

Technology evaluation system for heterogeneous high performance computing

Simula Research Laboratory AS eX³ ITT 10/2018

https://www.simula.no/

AbbreviationsAI Artificial Intelligence

ARM Advanced RISC Machines

B Byte, synonymous with the ISO unit ‘octet’ and as such is 8 bits long.

B2F Back-To-Front

BBU Battery Backup Unit

BMC Baseboard Management Controller

CM Cluster Management

Core A core is the processing unit within a CPU that receives instructions and performs calculations, or actions, based on those instructions.

CPU Central Processing Unit, a chip this days consisting of multiple processing elements or cores.

CRU Customer Replaceable Unit

CUDA CUDA (Compute Unified Device Architecture), a parallel programming framework by graphics processor manufacturer Nvidia.

DIMM Dual Inline Memory Module

DP Dual-Processor

ECC Error-correcting code memory (ECC-memory)

Firmware A specific class of computer software that provides the low-level control for the device's specific hardware.

FPGA Field Programmable Gate Array

GHS Global Hot Spare

GPGPU General Purpose Graphical Processing Unit

GPU Graphical Processing Unit

HBA Host Bus Adapter

HCA Host Channel Adapter

HIC Host Interface Card

HPC High Performance Computing

IB Infiniband Interconnect Network as described by the InfiniBand® Trade Association

IOPS IO operations per second as measured on file-system level

IPMI Intelligent Platform Management Interface protocol used by BMC

IPU Intelligence Processing Unit

ISA Instruction Set Architecture

3

LUN Logical UNit, a set of disks combined to one logical unit

MD Multiple-Devices aka Linux Software RAID

MI Machine Intelligence

ML Machine Learning

MPI Message Passing Interface

NAS Network Attached Storage

NBD Next Business Day

NPU Neural Processing Unit

NTB Non-Transparent Bridging is part of PCIsig standard version 2 or later

NVDIMM None-Volatile Dual Inline Memory Module

NVMe NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCIS)

OCP Open Compute Platform

OEM Original Equipment Manufacturer

OpenCL OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators

OpenMP OpenMP (Open Multi-Processing) is an application programming interface (API) that supports multi-platform shared memory multiprocessing programming in C, C++, Fortran and several other programming languages.

PCIe Peripheral Component Interconnect Express

PDU Power Distribution Unit

PSU Power Supply Unit

RAID Redundant Array of {Inexpensive|Independent} Disks

RISC Reduced Instruction Set Computer architecture

RPS Redundant Power Supply

SSA-K Norwegian Government Standard Contract for Acquisitions for Software and Hardware

top-bin Refers to product binning in semiconductor device fabrication in which top-bin refers to chips that pass all tests including higher GHz speeds, lower voltage, more functional cores etc. and therefore can be sold at a higher price point.

TPU Tensor Processing Unit

URL Universal Resource Locator

4

5

Terminology

This document aims to use simple terminology1. Some terms are used interchangeably.

Customer Client, Buyer, procurer

Contractor Vendor, Supplier

DIFI Agency for Public Management and eGovernment (Norw.: Direktoratet for forvaltning og IKT)

FOA2 The Norwegian Public Procurement Regulation (Norw.: Forskrift om offentlige anskaffelser)

ITT Invitation to tender

KOFA The Norwegian Complaints Board for Public Procurement (Norw.: Klagenemnd for offentlige anskaffelser)

LOA3 Puplic Procurement Act (Norw.: Lov om Offentlige Anskaffelser)

NDA Non-Disclosure Agreement

offer tender, proposal, bid, submission

RFP Request for Price/Procurement

RFI Request for Information

RFQ Request for quotation/ quote

RFT Request For Tender

RFx RFI, RFQ, RFP or RFT

supplier vendor, seller, contractor, potential supplier, respondent

tender a generic term used to describe making an approach to market (‘going out to tender’)

SSA The Norwegian Government Standard Terms and Conditions for IT Procurement (Norw: Statens standardavtaler for IT-anskaffelser)

1 https://www.anskaffelser.no/sites/anskaffelser/files/competition_and_contracts_text.pdf 2 https://lovdata.no/dokument/SF/forskrift/2016-08-12-974 3 https://lovdata.no/dokument/NL/lov/2016-06-17-73

6

Table of Contents

Appendix 1: Customer Requirement............................................................................131 THE CUSTOMER’S PURPOSE AND REQUIREMENTS.....................................15

1.1 Background Information On The eX³ project....................................................151.1.1 Facilitating national research on next generation HPC..............................151.1.2 Being at the frontier of R&D in the HPC domain.....................................151.1.3 Disclaimer..................................................................................................171.1.4 Invitation to Tender....................................................................................171.1.5 How to read the requirements....................................................................18

1.2 PURPOSE OF THE NEW SYSTEMS..............................................................181.2 HIGH-LEVEL PRODUCT REQUIREMENTS................................................191.3 COOPERATION WITH CONTRACTOR........................................................191.4 Non-Disclosure Agreements - NDA..................................................................191.5 Tentative progression plan.................................................................................191.6 Evaluation and Award criteria............................................................................201.7 Award of contract, justification and opportunity to appeal...............................20

2 TENDER REQUIREMENTS AND DEFINITIONS................................................203 FUNCTIONAL HARDWARE..................................................................................22

3.1 System architecture............................................................................................22An example solution could be:...........................................................................25

3.2 Central Processing Units....................................................................................263.3 Data Representation...........................................................................................263.4 Memory.............................................................................................................263.5 Management nodes - Redundant headnodes......................................................27

Example minimum headnode specification:.......................................................27Example nodes:..............................................................................................27

3.6 Storage Subsystem.............................................................................................283.6.1 Enterprise level hybrid flash storage with IB interfaces............................28Example enterprise level storage specification:..................................................283.6.2 Redundant metadata servers......................................................................29Example MDS specification:..............................................................................29

Example nodes:..............................................................................................293.6.3 Scale-out IB enabled NAS.........................................................................29Example minimum scale-out file system specification:.....................................30

3.7 Login node with Grid GPU...............................................................................30Example specification:........................................................................................30

3.8 Computational nodes.........................................................................................313.9 OCP HGX-2 reference design GPGPU server..................................................32

Example GPGPU node specification:.................................................................333.10 OpenPower GPGPU node...............................................................................33

Example of node specification:...........................................................................343.11 HPC Interconnect architecture.........................................................................34

3.11.1 Infiniband HDR topology.......................................................................343.12 Networking and connectivity...........................................................................35

3.12.1 10Gbps TOR switch................................................................................35

7

3.12.2 TOR switch OS.......................................................................................363.13 Physical Environment......................................................................................36

3.13.1 Racks, PDUs and accessories.................................................................363.13.2 Various length 10GBase-T compliant Cat7 S-FTP PIMF (LSZH) and 10Gbps TwinAX cables......................................................................................37

4 FUNCTIONAL SOFTWARE...................................................................................374.1 Cluster Management software...........................................................................384.2 Operating system...............................................................................................384.3 Global shared File Systems...............................................................................384.4 Batch and scheduler System..............................................................................394.5 Security..............................................................................................................394.6 Language Support..............................................................................................39

4.6.1 Tools and Libraries.....................................................................................394.6.2 Intel tools for academia..............................................................................394.6.3 Intel tools specific to Intel SP Gold 6139P................................................394.6.4 Arm Allinea Studio 18.4 and Performance Libraries.................................394.6.5 Nvidia tools for Deep Learning and AI......................................................404.6.6 OpenPower compiler and SDK..................................................................40

4.7 Third-party Software.........................................................................................405 TRAINING, DOCUMENTATION AND COOPERATION.....................................40

5.1 Training..............................................................................................................405.2 Ease-of-Use........................................................................................................405.3 Documentation...................................................................................................40

6 PROGRESS AND TIME SCHEDULE.....................................................................416.1 Installation of Initial Configuration...................................................................416.2 Installation of Complete Configuration.............................................................41

7 ACCEPTANCE AND APPROVAL TESTS..............................................................418 COMMERCIAL TERMS..........................................................................................41

8.1 Price...................................................................................................................418.2 Payment.............................................................................................................418.3 Terms of Delivery..............................................................................................41

9 ADDITIONAL PURCHASES..................................................................................4210 SYSTEM MAINTENANCE, AVAILABILITY AND RELIABILITY.................42

10.1 Maintenance.....................................................................................................4310.2 Availability and Reliability..............................................................................44

11 SYSTEM PERFORMANCE..................................................................................4511.1 Standard Benchmarks......................................................................................4511.2 Simula benchmark kernels...............................................................................46

11.2.1 Simula benchmark kernels......................................................................4611.2.2 Un-optimized “hands-off” runs...............................................................4611.2.3 Optimized runs........................................................................................46

11.3 Directory structure for USB/CD deliverable...................................................46Appendix 2: Contractor description of the deliverables...............................................52

The Agreement, clause 1.1 Scope of the Agreement...............................................52If the Contractor is of the view that there are obvious errors or ambiguities in the Customer requirements specification, the Contractor shall point this out here.......52

8

The Agreement, clause 2.1.1 Software and equipment...........................................53The Agreement, clause 2.1.3 Applicability of standard licence and agreement termsand conditions..........................................................................................................53The Agreement, clause 2.1.6 Warranty period and guaranteed performance..........53The Agreement, clause 2.7 External legal requirements.........................................53The Agreement, clause 4.3 Free software................................................................53

Appendix 3: Customer technical platform...................................................................5911.4 CURRENT HARDWARE PLATFORMS.......................................................59

11.4.1 Current systems.......................................................................................5911.4.2 Application stack.....................................................................................5911.4.3 Software Development Environment......................................................5911.4.4 CPU-based applications..........................................................................5911.4.5 GPU-based applications..........................................................................59

Most frequently used GPU applications........................................................59 Available GPU applications..........................................................................59

11.5 PHYSICAL INFRASTRUCTURE..................................................................5911.5.1 Street address..........................................................................................59

11.6 DC construction...............................................................................................6011.7 Network connection between TOR stack and simula networking..................6011.8 Rack size..........................................................................................................6011.9 Power...............................................................................................................6011.10 Cooling...........................................................................................................6111.11 Fire protection................................................................................................6111.12 Shipment and transportation..........................................................................6111.13 Special requirements in the computer room..................................................6111.14 Photos from the site.......................................................................................62 The Agreement, clause 1.1 Scope of the Agreement If in Appendix 1 the Customerstipulates that the deliverables shall function together with the Customer's current technical platform, the Customer's technical platform shall be described here.......65

Appendix 4: Delivery date and other deadlines...........................................................70Project and progress plan.........................................................................................70The Agreement, clause 2.1.2 Training.....................................................................71The Agreement, clause 2.2.4 Performance of the Customer acceptance test..........71The Agreement, clause 2.2.6 Commissioning.........................................................71The Agreement, clause 5.3 Use of subcontractors...................................................71The Agreement, clause 11.5.2 Liquidated damages in the case of delay................71

Appendix 5: Approval test............................................................................................76The Agreement, clause 2.2.2 Duty to examine........................................................76

Appendix 6: Administrative provisions........................................................................81Erratas, additions and changes to tender..................................................................81Customer Representatives.......................................................................................82 Contractor Representatives.....................................................................................82 Sub-Contractor Representatives.............................................................................82 Wages and working conditions...............................................................................82 The Agreement, clause 1.5 The representatives of the parties................................83The Agreement, clause 2.4 Wages and working conditions....................................83

9

The Agreement, clause 2.6 Form of communication - in writing............................83Appendix 7: Total price and pricing provisions...........................................................88

The Agreement, clause 2.1.2 Customisations and installation, etc..........................89The Agreement, clause 2.1.4 Documentation and training......................................90The Agreement, clause 2.1.6 Warranty period and guaranteed performance..........90The Agreement, clause 3.1 Consideration...............................................................90The Agreement, clause 3.2 Invoicing dates and payment terms..............................91The Agreement, clause 3.5 Price adjustments.........................................................91The Agreement, clause 4.1 Right of ownership of equipment................................91The Agreement, clause 4.2.1 Limited right of disposal...........................................91The Agreement, clause 4.3 Free software................................................................92

Appendix 8: Changes to the general contractual wording...........................................97Appendix 9: Changes subsequent to the conclusion of the Agreement.....................102Appendix 10: Licence terms and conditions for standard software and free software....................................................................................................................................107

The Agreement, clause 2.1.3 Applicability of standard licence terms and conditionsand agreement terms and conditions......................................................................107The Agreement, clause 4.3 Free software..............................................................107

10

11

I Appendix 1: Customer

requirement specification

Appendix 1 of Purchase Agreement

Customer requirementspecification

1 THE CUSTOMER’S PURPOSE AND REQUIREMENTS

1.1 Background Information On The eX³ project

1.1.1 Facilitating national research on next generation HPC

Computational infrastructures are essential for an increasing number of disciplines, and it is particularly important for research that requires demanding calculations or generates large amounts of data. The project “Experimental Infrastructure for Exploration of Exascale Computing”, or eX³, will develop competence and techniquesfor successful exploitation of the coming generation supercomputers. Such exascale computers will be able to perform at a minimum of a billion billion (10^18) floating point operations per second. This new generation of high performance computing, which will also be needed to handle the exponentially increasing, complex data that isbeing generated in research and in applications, is a major target for international research.

Simula’s eX³ project was one of the 19 projects selected for funding out of 92 applications submitted to the most recent round of the Research Council of Norway’s (RCN) program for research infrastructures. The project will build and run a national HPC infrastructure that will help prepare Norwegian researchers, big data users, data center management, and industry for the coming exascale era of computing. The upcoming exascale computing will rely on an intricate interplay between thousands ofsophisticated processing nodes, each with a large number of cores, deep memory hierarchies and equipped with accelerators, organized in complex communication topologies. The aggregated level of complexity in a system designed for billion-way concurrency represents a major challenge with many features: How to program such computers? How to port existing code, and how to reach a satisfactory level of reliability and efficiency while maintaining an acceptable energy footprint? The eX³ infrastructure will provide a platform for researching answers to such questions, utilizing relevant technology albeit on a smaller scale than in a full-blown exascale system.

1.1.2 Being at the frontier of R&D in the HPC domain

While competing technologies in hardware, middleware, and software are being driven by major research projects in the United States, China, Japan and the EU, it is essential for Norwegian HPC research groups to be at the frontier of research.

15

The eX³ project is currently scheduled to run for five years, with a high potential for future extensions and additions. In addition to the host institution Simula, the project consortium also counts the national HPC management body Sigma2, HPC research groups from NTNU, University of Tromsø and University of Bergen, as well as the HPC technology providers Dolphin, Numascale, Skala Technologies and Fabriscale.

In order to fulfill the goal of the eX³ project and deliver on the terms contracted with RCN, the system that Simula will build and offer to the research community has to be based on bleeding-edge components that will play important roles in the exascale setting. The HPC industry is characterized by quickly changing generations of hardware components that each brings more computing power and new possibilities, sometimes complemented by more disruptive technology steps. The different generations of components tend to hit the market as waves that fade out after 18-24 months when a new generation is ready to roll in. For eX³, it is critically important to buy into the relevant incoming technological waves, rather than surfing on the top of the existing waves. This marks the difference between the experimental nature of eX³ and traditional, production-oriented systems. In the eX³ context, aggressive experimentation will be allowed, for instance by reserving the whole system for detailed performance measurements or testing unusual combinations of heterogeneouscomponents. In total, the eX³ infrastructure will be a quite unusual installation, of which there are very few comparable systems around, even internationally. Such a system is already high in demand, and due to its visibility, eX³ is expected to be a showpiece for both researchers and technology providers. Moreover, the eX³ infrastructure will play an important role in educating master and PhD students, at partner institutions and their collaborating universities, to become the next generation HPC experts.

The following sections of this procurement document describe the technical requirements for the initial (Phase 1) purchase of equipment for the eX³ infrastructure.Further purchases are planned over two coming phases in 2019/2020. Preparing for the Phase 1 purchases, the eX³ partners have conducted extensive surveys of availableand soon-to-appear technologies and hardware components that are appropriate for the eX³ system. In particular, the final system will be very heterogeneous, involving several different architectures based on different CPUs, GPGPUs and emerging processors designed specifically for machine learning algorithms. For Phase 1, the emphasis is on CPU and GPGPU nodes.

On the other side, a successful outcome of the eX³ project will require that the system build on the competence present in Simula and in the other eX³ partner organizations. This requirement is also posed by RCN as the funding agency, in that the project is granted on basis of the competence demonstrated in the project application. Too large deviations from the competence base would threaten the success of the project, and

16

thus violate the underlying considerations. While this condition sets a certain direction, it is also compliant with the international technology trends for top-tier HPC systems in which the available competence is very relevant. Another requirement, which is fundamental to keep the total cost at a reasonable level, is that the equipment purchased in the eX³ project is compatible with the existing computing infrastructure at the host institution (Simula).

Based on the totality of the above considerations, some of the requirements in the current procurement document points, directly or indirectly, to specific architectures. In some cases, due to targeting the start of an incoming technology wave, only one specific product or a specific group of products adequately cover such architectural choices at the present time.

1.1.3 Disclaimer

In the following system specification we describe different Phase 1 components of the eX3 system as generically as possible, while respecting the conditions described above and concentrating on the technologies that is required or desired in the system. To the extent that we provide examples of any component in terms of referring to specific products, this is only to increase the readability of the document and to help the reader visualize how a generic component could potentially be realized. Any such example is not to be interpreted as a wish for the Contractor to include that specific item.

The Contractor has full freedom to suggest solutions composed from any product combination that fulfills the generic specifications and that are, in the Contractor's subjective opinion, the most competitive offer they can make in light of the evaluation criteria described in the accompanying invitation to tender.

1.1.4 Invitation to Tender

Simula Research Laboratory hereby invites interested contractors to an open competition for tendering the first phase of purchases of equipment for the national research infrastructure eX3. This infrastructure will contain several computer nodes interconnected to form a high performance computing (HPC) facility. The accompanying procurement document specifies the technical requirements for this purchase in its Appendix 1. It is noted that eX3 will be a highly experimental facility set up to make state-of-the-art technology components available for academic research purposes.

17

1.1.5 How to read the requirementsIn the following sections, we specify the technical requirements for the eX3 Phase 1 purchase. These requirements will be marked according to the convention defined below. As explained in the invitation to tender, this convention will be actively used inthe assessment of offers provided.

M denotes MANDATORY requirements:

A requirement that must be adhered to or a performance requirement that mustbe met in order for the tendered solution to be accepted by Simula. Simula willnot consider an offer that fails to meet any of the mandatory specification requirements.

D denotes DESIRABLE features:

If a desirable feature cannot be supplied initially, the tender should state whether, and if so when, the feature will be added to the installed system.

R denotes REQUESTS for information:

A response must be given to all such requests. Requests for information are intended to prompt a description of the construction, philosophy or operation of the tendered solution in areas that are valuable to Simula in the tender evaluation process.

The accompanying document “Rules of the Competition” outlines how we will assess the offerings.

1.2 PURPOSE OF THE NEW SYSTEMSM[1] Heterogeneity: In the offered solution we are looking for a high degree of heterogeneity. The objective of the first phase of the eX³-project is to

• evaluate emerging processing and offload engine technologies applicable to domain-specific problems such as machine learning and artificial intelligence.

• provide the latest emerging processing and offload-engine technology to the researchers and scientists at Simula and partners.

• explore scalability of heterogeneous systems with a mixed architecture.

• explore network topologies and advanced upcoming networking features for high-performance computing.

18

We, therefore, would like to see offered a combination of the latest top-bin CPU architectures from Intel Xeon Scalable, AMD Epyc, Arm644 and potentially OpenPower/Power9. The solution should provide the best GPGPU technologies in themarket from Nvidia or AMD.

1.2 HIGH-LEVEL PRODUCT REQUIREMENTSM[2] front-end: The offered solution should include a front-end that will work for future expansions of the eX³ system, as an integral part of a larger heterogeneous system.

1.3 COOPERATION WITH CONTRACTORD[3] Collaboration: It is acknowledged that the cost of operation and maintenance ofthe required system may be significantly reduced if Simula and the successful Contractor establish an active and working relationship. Access to engineering and development resources as well as global product support within the respective company is valued. Remote access to computational resources and early access to new technology will be valued as well. This, in turn, will enable the successful bidderto propose a more ’aggressive’ Tender which could potentially give a qualitative edge for the second phase of the eX³ project.

Outreach. Simula Research Laboratory AS and partners aim to introduce new HPC techniques towards exascale to new user groups and partners. Contractors partnering with Simula will gain the extended benefit of penetrating into the Norwegian HPC community as well as EU projects. The lessons learned in the eX³ project will benefit decision makers in both academia as well as the industry.

1.4 Non-Disclosure Agreements - NDAThe members of the tendering committee are willing to sign NDA agreements if deemed necessary. Appendix 1 2 is for NDAs.

1.5 Tentative progression planThe tentative plan for this procurement can be found in Appendix 4 and section 2.5 in the accompanying document “Rules of the Competition”.

4 Cavium ThunderX2 CN9980, Ampere Quicksilver, Qualcomm Firetail or equivalent.

19

1.6 Evaluation and Award criteriaPlease refer to section 6 of the accompanying document “Rules of the Competition”.

1.7 Award of contract, justification and opportunity to appealPlease refer to sections 5 and 6 of the accompanying document “Rules of the Competition”.

2 TENDER REQUIREMENTS AND DEFINITIONSA heterogeneous supercomputer system is a highly complex product, and we realize the difficulties related to the evaluation of the different proposals and the technical specifications. There is no simple metric that can be used to compare the descriptions of the different proposed systems. In order to obtain some consistency in the descriptions of the offered systems this document has been sectioned into 12 appendices in accordance with the Norwegian Agency for Public Management and eGovernment (DIFI) standard contract for hardware and software procurements SSA-K:

• A00 Draft Contract (SSA-K, 2015), including the following appendices merged into one pdf:

• A01 Appendix 1: Customer requirement specification

• A02 Appendix 2: Contractor description of the deliverables

• A03 Appendix 3: Customer technical platform

• A04 Appendix 4: Delivery date and other deadlines

• A05 Appendix 5: Approval test

• A06 Appendix 6: Administrative provisions

• A07 Appendix 7: Total price and pricing provisions

• A07A Attachment Price and Requirement Matrices

• A08 Appendix 8: Changes to the general contractual wording

• A09 Appendix 9: Changes subsequent to the conclusion of the Agreement

• A10 Appendix 10: License terms and conditions for standard software and freesoftware

20

• A11 Appendix 11: Benchmarks (Not in phase 1)

• A12 Appendix 12: Non-Disclosure Agreement with Contractor

Sections in this appendix, Appendix 1 of SSA-K contains a detailed specification of the system requirements and of the requested information. These sections indicate some expected levels of performance, characteristics, functionalities, and services of what we consider a successful heterogeneous technology evaluation system.

We require the Contractor to use these sections for preparing the Tender, as all the requirements and queries in these sections will be used in the evaluation of the Tenderand much of the data will be used for the final Purchase Agreement.

More precisely, the Contractor must respond to the requirements and queries in Appendix 1 sections 1 through 11 by completing Appendices 2 through 12 of the draftof the Purchase Agreement. Appendix 2 shall give an overview of the Contractor’s proposal. If the Contractor requires any changes or amendments to the General Terms of the Purchase Agreement, these should be given in Appendix 9. The correspondencebetween the sections of Appendix 1 and the appendices 2 through 12 of the Purchase Agreement is 1-to-1 as specified by SSA-K.

The Contractor shall reply to each of the items in sections 1 through 11. Depending onthe nature of the item, the Contractor shall reply with a description, explanation, one or more numbers, Yes (compliant) or No (not compliant), or other (if appropriate) as specified in Appendix 2 table 15 and fill out Appendix A07A workbook requirement sheet.

The Tender must be accompanied by sufficient commentary for us to understand the extent to which the requirements in sections 1 through 11 are met. We expect the Contractor to give a detailed description of how the proposed system achieves the expected functionality and performance, how the characteristics are implemented, how the functionalities are supported, and how the service levels are achieved.

The requirements and queries in sections 1 through 11 apply to both the initial and thecomplete configuration of the proposed system. If no upgrades are planned during the contract period, the Contractor can ignore the answers for either the initial or the complete configuration (whichever is appropriate).

We acknowledge that some information regarding the complete configuration may notbe known at the time of writing of the Tender. In such a case, the Contractor is asked to provide as much information as possible.

Any other information can be included at the discretion of the Tenderer with referenceto Appendix 12.

21

The Contractor must guarantee that the mandatory requirements will be met and must give necessary guarantees for all the configurations and performances of the proposedsystem.

All the proposals that satisfy the technical specifications will be evaluated thoroughly according to the scheme described in the accompanying document “Rules of the Competition”. The evaluation will consider the arguments put forward and weigh the strong and weak points of the offers very carefully.

It is stressed that the overall system must be built within the available budget. If one or more components of similar or better quality/performance can be obtained more favourably from other sources, then this must be appropriately reflected and the cost justified in the Tender.

Simula reserves the right to adjust the overall system size up or down (in meaningful steps) according to whether the system meets or exceeds the available budget in accordance with a procurement with negotiations. Simula has spent considerable time gathering price information in order to set a realistic budget for phase 1 and believe the indicated number of items is possible based on the set budget and knowledge of industry discount rates for academic tenders. In addition, Simula and partners may scale the system based on an evaluation on whether the system meets the requirements for a particularly advantageous acquisition (financially and otherwise).

The format of this procurement document will be reused in the future procurement processes for later phases of the eX³ infrastructure.

3 FUNCTIONAL HARDWAREIn the following 2 sections, we describe different components of the eX³ system in a generic way, concentrating on the technologies that are required or desired in the system. To the extent that we provide examples of any component or systems in termsof referring to specific products, this is only to increase the readability of the document and to help the reader visualize how a generic component could potentially be realized. Any such example is not to be interpreted as a wish for the Contractor to include the specific item. The Contractor has full freedom to suggest solutions composed from any product combination that fulfill the generic specifications and that is, in the Contractor's subjective opinion, the most competitive offer they can make in light of the evaluation criteria described in this document.

3.1 System architectureAs indicated in mandatory requirements in section 1.2 through 1.3, to ensure a high degree of heterogeneity the overall system shall be a standard heterogeneous Beowulf

22

Linux cluster with HDR Infiniband HPC interconnect, 10Gbps provisioning and failover networks, gigabit network for management and a combination of nodes with varying instruction set architectures and offload-engine technologies.

M[4] CM, OS and Admin nodes: The system should include redundant head-nodes running a modern cluster management (CM) suite on-top of an electable Linux distribution. Please refer to section 3.6.

M[5] Supporting nodes: The solution should include a login node with Grid GPU, redundant HA metadata servers with high-end x86-64 general purpose processors. Thelogin node should be equivalent to the admin nodes with the exception of the additional Grid GPU. Login and admin nodes should be interchangeable in case of failures, and the desirability of least number of CRUs applies (as explained in D[7]). For the metadata servers (MDS) for globally shared file-systems, we require x64 or AMD64 ISAs. These systems should include the latest emerging memory hierarchy technologies such as NVMe and NVDIMM-N for fast metadata operations. For a more detailed specification please refer to section 3.6.2.

M[6] Compute nodes CPUs: Compute nodes should have the latest generation top-bin processors for the respective ISAs offered.

D[7] Fewest CRUs: The total solution should have as few versions of customer replaceable components as possible, such as chassis types, HDDs, DIMMs, PSUs, CPUs, HCAs, HBAs.

M[8] Enterprise grade disks: The disk drives in the supporting nodes should be 2.5”enterprise-grade NL-SAS 12Gbps 2TB unless otherwise is stated. The compute nodes should however have enterprise-grade 2.5” 1TB NVMe drives unless the node does not support NVMe, in which case we prefer enterprise-grade 1TB write-optimized SSD’s instead. This is to ensure uniformity across compute nodes in the cluster. Nodeslocal scratch disk-space should be high performing and will be globally exported scratch with BeeGFS on demand BeeOND feature as commonly seen in HPC clusters.

M[9] Tri-mode RAID controllers: Administrative nodes, login nodes, metadata and GPGPU servers should have raid controllers such as the industry standard Broadcom MegaRaid Trimode HBA or equivalent OEM host bus adapter (HBA) which supports SATA, SAS and NVMe.

D[10] Node RAID controller: Compute nodes with dedicated HW raid or HW assisted SW raid such as Intel ERST2 is preferable. Software Raid level 0 (MD) is also acceptable for compute nodes.

M[11] HW backdoor guarantee and vulnerability statement: Contractor must guarantee that, to the best of their knowledge, hardware is without backdoors, and that

23

they are not aware of any security vulnerabilities in the management engines, management controllers, trusted platform modules or any other constituent part of the servers, switches, PDU's etc. If Simula or other consortium members find vulnerabilities, the contractor must replace the affected hardware or firmware at no additional cost.

M[12] HPC interconnect: For the solution, we would like the latest 200Gbps Infiniband HDR technology. Even though Infiniband is a standard, history has shown that interoperability between Infiniband vendors implementations is not trivial. As Simula is heavily invested in Mellanox at this time, and Mellanox currently has the top-performing interconnect, we request a bid with Mellanox hardware.

Based on the above-outlined system architecture, a suggested solution may be:

24

An example solution could be:

Qty Description

2 High-Available Administrative headnodes

1 Login node w/Grid GPGPU

2 HA Metadata servers for BeeGFS

2 40-port Infiniband HDR managed switches w/RPS, B2F Fan

14 40-port Infiniband HDR unmanaged switches w/RPS, B2F Fan

27 200Gbps HDR infiniband HCA, PCIe 4.0 16.0GT/s, 2 port, QSFP28, socket direct

4 Innova-2 Flex Open VPI, dual-port QFP28, EDR / 100GbE, KU15P, No memory, No Crypto, PCI4.0 x8, HHHL, passive heat sink, tall bracket, ROHS R6

250 Various Infiniband HDR cables, 0.5m – 3m (ref table below)

1 10Gbps 10GBase-T TOR switch 48 port + 6 qty 100/25/10Gbps ports

4 DP nodes with Intel Xeon Scalable Platinum top-bin 8180M

4 DP nodes with AMD Epyc top-bin 7601

4 DP nodes with top-bin Arm645 processors

4 DP nodes with Intel Xeon Scalable Gold 6138P6 with integrated Arria 10GX1150 FPGA

1 OCP OpenPower node with top-bin Power9 22core and 4 qty Volta GV100 GPUs

1 OCP HGX-2 system w/top-bin 8180M with 16 qty GV100

1 Global shared FS for AI/ML with 6-node quadnode chassis with ~40TB

1 Enterprise level storage system, Dual Controller each w/128GB cache and Dual EDR Infiniband (quad IB in total), 1400TB raw capacity, minimum 10TB SSDcache, such as NetApp E5760i or equivalent

4 48u wide enterprise racks each with two managed 3-phase PDUs

Table 1: Example solution (Please refer to section 1.1.3. This list is meant only to increase the readability of the procurement document and should not be interpreted as a request for the specific items listed. )

5 Cavium ThunderX2 CN9980, Ampere Quicksilver, Qualcomm Firetail or equivalent.6 https://en.wikichip.org/wiki/intel/xeon_gold/6138p

25

3.2 Central Processing UnitsM[13] CPUs: Processing nodes, as opposed to management nodes, should have the latest generation “top-bin” processors and GPGPUs. Instruction set architectures (ISA) could be any combinations of x86-64, Amd64, Aarch64 or OpenPower ISA v3 fulfilling the minimum specifications provided below.

3.3 Data RepresentationM[14] IEEE Standard for Floating-Point Arithmetic: Tenders should document forthe offered general purpose processors and GPGPUs their conformity to the IEEE 754-2008 standard or equivalent.

D[15] GPCPU AI/ML specifics: Contractors are encouraged to include documentation of various nodes floating point implementation as well as any machinelearning (ML/AI) specific formats such as FP32, BFloat16 all the way down to INT8.

D[16] CPU AI/ML specifics: Contractors offering AI/ML offload-engine technology with reduced precision floating point representation such as BFloat16 down to INT8 aimed towards machine learning/AI increases the value to offer. Several processors companies are on the verge of releasing processors with among other things accelerated BFloat16.7

3.4 MemoryM[17] DIMM densities: As specified above, the least possible number of CRUs are desirable. DIMMs should be interchangeable across all nodes and architectures. DIMMs should be at least 64GB DDR4-2666 registered ECC RDIMM or higher densities unless otherwise is stated.

M[18] Performance optimized memory topology: Regardless of architectures, the servers and nodes should be populated in a balanced performance optimized manner. Such that the number of DIMMs fill the memory controllers on the sockets and such that the Numa nodes within servers will have an equal amount of memory.

D[19] NVMe/NVDIMM-{N|P}: It is desirable that computational servers include some portion of NVDIMM 3D-Xpoint technology. Systems based on Xeon Scalable, AMD Epyc as well as Cavium ThunderX2 claim support for it. As for DIMMs, NVDIMM density should be highest possible within reason and installed in a performance-optimized fashion.

D[20] 128GB DIMM: Higher densities and data-rates are preferable as one should beable to upgrade systems to maximum capacity without replacing hardware. Offer

7 Cooper Lake, Lake Crest, Rome ao.

26

should include optional pricing for systems with highest density DIMMs compatible with the offered hardware.

3.5 Management nodes - Redundant headnodesThe redundant head/admin nodes are for cluster management, imaging/provisioning, monitoring, user admin, backup login, batch and scheduler services, Infiniband segment manager and so forth.

M[21] Headnode config: The offer should include two (2) redundant admin/head nodes with failover capabilities.

Example minimum headnode specification:

Qty Description

1 2u DP server node with 16TB raw 12Gbps SAS storage

2 Xeon Scalable Gold 16core low-bin volume CPU

Alternative: AMD Epyc 20core volume CPU

12(8)

32GB DDR4-2666MHz RDIMM (384GB)Alt: 32GB DDR4-2666MHz RDIMM (256GB)

1 Industry standard Broadcom MegaRaid Trimode 9460-16i HBA w/BBU,OEM or equivalent

8 2TB 12Gbps SAS/NL-SAS disks

2 10GBase-T ports (RJ45)

4 1Gbps ports (RJ45)

1 1Gbps BMC IPMI 2.0, Redfish support

3 PCIe Gen3 x16 slots

2 PCIe Gen3 x8 slots

1 D[22]: OCP slot 2 (PCIe Gen3 x16)

Table 2: Example management node specification

Example nodes:

Dell PowerEdge 740xd/R7525, HPE ProLiant PL380/PL385 Gen10, Supermicro SuperServer 2029UZ-TR4+/AS-2123BT, Cisco UCS C240/C4200 M5, Lenovo ThinkSystem SR650, QCT QuantaPlex D52BW-2U, Sugon I620-G30, Inspur NF5280M5 or equivalent.

27

3.6 Storage Subsystem

3.6.1 Enterprise level hybrid flash storage with IB interfaces

M[22] Enterprise storage: Integral to the phase 1 solution the offer should include apowerful enterprise level hybrid flash storage system for homes, apps (modules environment) and data. The offered Infiniband enabled storage system should have approximately 1PB nominal storage space after LUN binding. It will form the backbone for our storage requirements for the next few years, and be expandable with3 additional 60-bay or larger chassis. Further details are given in the table below.

Example enterprise level storage specification:

Qty Description

1 Industry standard enterprise level IB enabled storage, OEM8 or equivalent

1 Dual Controller w/128GB cache or more

4 Infiniband EDR HIC interface or more

3 60 bay or larger shelves (CE+2DE)

175 8TB Enterprise NL-SAS 12Gbps HDDs w/128MB cache or higher capacity

1 SSD-cache Feature Enabler

5 2TB read-intensive optimized Enterprise SSDs

1 Selectable iSER, SRP, NVMe-oF and NVMe-oRoCE protocol firmware Feature Activation Enabler Code9

1 HPC tiering Feature Activation Code

1 Management software

Upgradeable to at least 360 disks

Table 3: Example Enterprise level IB storage specification

M[23] Extended support: The storage system should include year 5 NBD support.

R[24] HIC with HDR: It is desirable to have 200Gbps HICs. Please indicate HIC and/or raid controller roadmap.

8 NetApp E-series and DDN SFA storage systems are widely OEM’ed by many contractors in HPC9 All firmwares/protocols should be available at no cost for academic use.

28

3.6.2 Redundant metadata servers

M[25] MDSs: The enterprise storage solution should include two (2) high-available metadata servers for fast BeeGFS metadata operations. They should include redundantsystem disks as well as raided NVMe storage for metadata and NVDIMM-N technology for use with PMEM aware/DAX enabled filesystems such as XFS or EXT4.

Example MDS specification:

Qty Description

1 2u DP server node with 8TB NVMe storage for Metadata

2 Xeon Scalable Gold or AMD Epyc mid-bin volume CPU

12 32GB DDR4-2666MHz RDIMM (384GB)

12 32GB NVDIMM-N DDR4-2666 RDIMM (384GB) w/retention BBU

1 Broadcom Trimode 9460-16i Raid HBA w/BBU, OEM or equivalent

3 2TB SAS 12Gbps NL-SAS (Mirrored system disks raid 1+ GHS)

8 NVMe 2TB disks x2 mode (GHS is desirable)

4 1Gbps ports (RJ45)

2 10GBase-T (RJ45) ports

1 1Gbps BMC IPMI 2.0, Redfish support

3 PCIe Gen3 x16 slots

2 PCIe Gen3 x8 slots

1 OCP Slot 2 (PCIe Gen3 x16)

Table 4: Example Redundant MDS specification

Example nodes:

Dell PowerEdge 740xd/R7525, HPE ProLiant PL380/PL385 Gen10, Supermicro SuperServer 2029UZ-TR4+/AS-2123BT, Cisco UCS C240/C4200 M5, Lenovo ThinkSystem SR650, QCT QuantaPlex D52BW-2U, Sugon I620-G30, Inspur NF5280M5 or equivalent.

3.6.3 Scale-out IB enabled NAS

M[26] Scale-out IB enabled NAS: The solution shall include 6 node quadnode-chassis redundant scale-out global shared filesystems such as BeeGFS or Weka.IO10.

10 https://insidehpc.com/2018/09/ai-workloads-technical-computing/

29

Example minimum scale-out file system specification:

Qty Description

2 Quadnode chassis enclosures each with 24 U.2 slots

14 Xeon Scalable Gold 6126

72 8GB DDR4-2666MHz RDIMM

6 Raid support 0,1,5,10, Tri-mode RAID, OEM or equivalent

6 128GB SATA M.2 system disks (2 slots available)

36 ~2TB U.2 NVMe disks for a total of 72TB raw for AI/MI/ML workloads

1 Connect-X5 or higher

1 10GBase-T (RJ45) ports

1 1Gbps BMC IPMI 2.0, Redfish support

3 PCIe Gen3 x16 slots

1 PCIe Gen3 x24

Table 5: Example Scale-Out AI/ML NAS specification

3.7 Login node with Grid GPUA GPU enabled login node will also function as a quorum tiebreaker for global sharedfilesystems. The login node should be compatible and interchangeable with admin nodes.

M[27] login node: A login node should have same specification as headnodes, but should also include a Grid GPGPU.

Example specification:

Qty Description

1 2u DP server node with 16TB raw 12Gbps SAS storage

2 Xeon Scalable Gold 16core or AMD Epyc 20core mid-bin volume CPU

12(8)

32GB DDR4-2666MHz RDIMM (384GB)Alt: 32GB DDR4-2666MHz RDIMM (256GB)

1 Broadcom Trimode MegaRaid 9460-16i HBA w/BBU, OEM or equivalent

30

8 2TB 12Gbps SAS disks

2 10GBase-T ports (RJ45)

4 1Gbps ports (RJ45)

1 1Gbps BMC IPMI 2.0, Redfish support

3 PCIe Gen3 x16 slots

2 PCIe Gen3 x8 slots

1 OCP slot 2 (PCIe Gen3 x16)

1 Nvidia Grid GPU Quadro M60

Table 6: GPU enabled login node example specification

Example nodes:Dell PowerEdge 740xd/R7525, HPE ProLiant PL380/PL385 Gen10, Supermicro SuperServer 2029UZ-TR4+/AS-2123BT, Cisco UCS C240/C4200 M5, Lenovo ThinkSystem SR650, QCT QuantaPlex D52BW-2U, Sugon I620-G30, Inspur NF5280M5 or equivalent.

3.8 Computational nodes

M[28] Node heterogeneity: The offer should include at least 3 different top-bin processor instruction set architectures (ISA) such as x64, x64 w/FPGA, AMD64, AArch64 or OpenPower. For instance an offer could include compute servers with top-bin AMD Epyc 7601, Cavium ThunderX2 CN9980 and Power9.

M[29] Nodes CPU density: The offer should include a minimum of 2 dual-processor (DP) compute nodes per architecture/ISA.

M[30] Optional node pricing: The offer should also specify pricing for four (4) DP nodes for each offered CPU architecture.

D[31] Nodes form factor: The quad-node form factor is desirable. If the offer includes dual-port Mellanox ConnectX-6 mezzanine or OCP slot 2 variants there should still be 2 free PCIe Gen3 x16 slots available.

D[32] RAM option: Nodes should preferably be configured with maximum allowable memory per architecture, in a performance optimized fashion.

D[33] NVDIMM: Offers with a portion of NVDIMMs on the compute nodes are desirable.

31

M[34] NVMe/SSD disks: The compute nodes should include 4 qty 1TB NVMe disks or alternatively 4 qty 1TB write-optimized SSD disks if node type does not support NVMe. These disks will be used with BeeGFS on-demand as /global/scratch.

D[35] PCIe generation: It is desirable that offer includes latest generation motherboards with PCIe Gen4 x16 and OpenCAPI11 support. We appreciate the fact that we are on the verge of a generation change. We expect to see Gen4 motherboards this side of the new year, but can not at this time set this requirement as a mandatory.

M[36] PCIe Gen3 x16 and/or OCP slot 2 slots: Each node should have at least 4 PCIe Gen3 x16 slots available. For instance, three PCIe and one OCP slot 2 is acceptable. A proprietary mezzanine Gen3 x16 slot for IB and 10GBase-T is also acceptable. As an example, compute nodes should be able to accommodate a dual-portMellanox HDR Socket-Direct HCA taking up 2 slots, a PCIe based HCA such as Dolphin Interconnect PXH830 and in addition a 300W offload-engine such as Nvidia Tesla and others12 machine learning13 hardware.

M[37] General IO: System should include at least 1 qty 10Gbps interface for provisioning, management and failover for global shared filesystems.

M[38] 10GBase-T: If the systems only provides SFP+ then the offer must include TwinAX cabling.

M[39] BMC: BMC should support IPMI v2.0 and Redfish 1.0 or later. OCP OpenBMC certification is desirable.

D[40] NTB support in BIOS: Simula and partners are exploring the use of PCI Express Non-Transparent Bridging (NTB14) for clustering. The last two generations ofXeon processors supports NTB mode on PCIe ports. It is desirable that the motherboard bios include an option to turn on NTB and consequently that the port is routed on the PCB to a PCIe/OCP slot. There are motherboards available with this feature. There are also motherboards available with iPASS edge connectors.

3.9 OCP HGX-2 reference design GPGPU server M[41] HGX-2: The offer shall include one OCP HGX-2 reference design GPGPU system with maximum specifications.

D[42] IB endpoint: It is desirable that the system has 4 dual-port HDR HCAs rather than the 8 dual-port EDR interfaces it usually is fitted with.

11 https://opencapi.org/ 12 https://en.wikipedia.org/wiki/AI_accelerator#Research_and_unreleased_products 13 https://basicmi.github.io/Deep-Learning-Processor-List/ 14 https://lxr.missinglinkelectronics.com/linux/Documentation/ntb.txt

32

Example GPGPU node specification:

Qty Description

1 DP GPGPU server 300GB/s NVlink

2 Intel® Xeon® Scalable® Platinum 8176M Dual Socket P (LGA 3647) 3 UPI up to 10.4GT/s

16 Nvidia Tesla V100 SXM2 32GB

12 NVSwitches NVlink2

12 128GB DDR4-2666MHz RDIMM (1.5TB)

1 Trimode 9460-16i HBA w/BBU, OEM or equivalent

2 960GB NVMe SSDs (OS system disks)

8 3.84TB NVMe x2 drives (/data, /scratch and /global/scratch)

8 Mellanox Connect-X5 EDR Dual Port

2 1/10/25Gbps 10GBase-T ports (RJ45)

1 1Gbps BMC IPMI 2.0, Redfish capability

6 PCIe Gen3 x16 slots

Redundant PSUs for max config (10kW)

1 Ubuntu 18.04 or RHEL/CentOS 7.5 or later

1 Software stack for ML/AI

Table 7: Example specification of OCP HGX-2 configuration

Example nodes:Nvidia DGX-2, Supermicro SYS9039GP-TVRT or equivalent.

3.10 OpenPower GPGPU nodeD[43] OpenPower ISA node: As part of this procurement we would like offered one OCP OpenPower Barreleye G2 based GPGPU server.

D[44] HDR HCA: HDR is preferable.

33

Example of node specification:

Qty Description

1 OCP OpenPower based GPU enabled system

2 POWER9 CPUs with next generation NVLink 22-core

4 Nvidia Volta V100 SXM2 32GB

16 64GB DDR4-2666MHz RDIMM (1TB)

1 Broadcom Trimode 9460 HBA w/BBU, OEM or equivalent

2 2TB 12Gbps SSD (System disks raid 1)

1 PCIe3 x8 Non-Volatile Memory 3.2 TB SSD NVMe adapter (/scratch and /global/scratch)

2 1/10Gbps 10GBase-T ports (RJ45)

1 1Gbps BMC/OpenBMC IPMI 2.0, Redfish capability

4 PCIe Gen4 x16 slots OpenCAPI

1 PCIe4 x16 1-Port HDR 200 GB IB ConnectX-6 CAPI Capable Adapter

2 Redundant PSUs for max config (2*2200W)

1 CentOS/Red Hat Enterprise Linux 7.5 for Power LE

Table 8: Example OCP OpenPower GPGPU node specification

Example node specification:IBM Power System AC922 Model 8335-GTH, Inspur OpenPower9 TS860 GPU system, or equivalent.

3.11 HPC Interconnect architectureSimula and partners conduct research on emerging exascale network technologies andfor instance, develop adaptive routing algorithms. Therefore the number of cables andswitches requested is greater than the number of nodes. This is deliberate.

3.11.1 Infiniband HDR topology

M[45] 200Gbps non-blocking fat-tree: The HPC interconnect in the solution shouldbe 200Gbps Infiniband HDR capable of 200 nodes in a non-blocking fat-tree.15 Due tothe existing inventory of hardware and interoperability, the offer should include the following components:

15 A sufficient number to allow for testing of various topologies such as Butterfly, Dragonfly ao.

34

Qty Part Number Description

2 MQM8700-HS2F Mellanox® Quantum(TM) HDR InfiniBand Switch, 40 QSFP56 ports, 2Power Supplies (AC), x86 dual core, standard depth, P2C airflow, Rail Kit, RoHS6

14 MQM8790-HS2F Mellanox® Quantum(TM) HDR InfiniBand Switch, 40 QSFP56 ports, 2Power Supplies (AC), unmanaged, standard depth, P2C airflow, Rail Kit,RoHS6

27 MCX654106A-HCAT

ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct 2x PCIe3.0 x16, tall bracket, ROHS R6

4 MNV303611A-EDLT

Innova-2 Flex Open VPI, dual-port QFP28, EDR / 100GbE, KU15P, No memory, No Crypto, PCI4.0 x8, HHHL, passive heat sink, tall bracket, ROHS R6

40 MCP1650-H00AE30

Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 0.5m, black pultab, 30AWG or compatible

80 MCP1650-H001E30 Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 1m, black pulltab, 30AWG or compatible

40 MCP1650-H01AE30

Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 1.5m, black pulltab, 30AWG or compatible

40 MCP1650-H002E26 Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 2m, black pulltab, 26AWG or compatible

30 MCP1650-H02AE26

Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 2.5m, black pulltab, 26AWG or compatible

30 MCP1650-H003E26 Mellanox® Passive Copper cable, IB HDR, up to 200Gb/s, QSFP56, LSZH, 3m, black pulltab, 26AWG or compatible

3 MTR-ALLLIBINC-PREMIUM-12M

The PLATINUM Online all-inclusive Courses Library access. Yearly subscription, full access to subscription time new content. 12-month subscription, including premium content

3 SUP-QM8700-1S Mellanox Technical Support and Warranty - Silver, 1 Year, for QM8700 Series Switch

Table 9: Infiniband HDR hardware

3.12 Networking and connectivityWith the solution we require general networking for provisioning/imagining, administration and login. We will reuse existing inventory of high-end gigabit switches16 with 10Gbps uplinks as management networks.

3.12.1 10Gbps TOR switch

M[46] OpenNetLinux17/ONIE TOR switch: The offer should include a 48-port OpenNetLinux ONIE switch with L2/L3 support. The offered switch should work

16 Force10 S50-01-GE-48T-AC17 https://opennetlinux.org/

35

with future expansion of the solution.

An example TOR switch could have:

• 16 ports 10Gbps SFP+ module

• 2 qty 16 port 10GBase-T RJ45 modules

• 4-6 qty QSFP28 split cables should be available (10/40/100Gbps selectable)

• 1Gbps Console port and Serial/USB console ports

• SNMP support

• redundant PSU (RPU)

Example switches:ONIE/OpenLinuxNetwork switches such as Dell Networking S5000, Arista, MellanoxSN{2|3}000, Cisco Nexus or equivalent OpenLinuxNetwork switches.

M[47] Bridging to datacenters: It is mandatory that the switch could be used to bridge vlans in two remote datacenters within a 50km radius.

R[48] IB gateway/bridging: It is conceivable that the phase 1 front-end system be bridged to further extension of the system. Please propose solutions to bridging IB networks within city near datacenters .

3.12.2 TOR switch OS

M[49] TOR switch OS: The switches should use a Linux based Network Operating System18 (LNOS) such as Cumulus or equivalent which would allow Simula and partners to develop and install switch kernel modules.

3.13 Physical Environment

3.13.1 Racks, PDUs and accessories

M[50] Racks and accessories: The offer should include four wide enterprise racks and rack accessories. Racks shall not be larger than 208cm high (46u) due to physical restrictions of elevator doors. Rack PDUs must be switched/manageable. PDUs should have IEC 60309 32A 3P+N+E input connectors. There should be sufficient

18 https://events.linuxfoundation.org/wp-content/uploads/2017/11/Open-Hardware-and-Open- Networking-Software-How-We-Got-Here-and-Where-We-are-Going-Steven-Noble-Big-Switch-Networks-NetDEF.pdf

36

C13/C14 or C13/C19 power cables for all components offered in the cluster. Please refer to Appendix 3 section 11.8 and 11.9.

Example configuration:

Qty Description

1 42-46u 750mm Wide x 1070mm Deep Enclosure with Sides Black

3 42-46u 750mm Wide x 1070mm Deep Enclosure Without Sides Black

8 Rack PDU 2G, Switched, ZeroU, 22.0kW(32A) 17.3kW(24A), 230V,(30) C13 & (12) C19

sufficient Power Cord Kit, Locking, C13 to C14, 0.6m

sufficient Power Cord Kit, Locking, C13 to C14, 1.2m

sufficient Power Cord Kit, Locking, C13 to C14, 1.8m

Table 10: Example racks, PDUs and accessories configuration

3.13.2 Various length 10GBase-T compliant Cat7 S-FTP PIMF (LSZH) and 10Gbps TwinAX cables

M[51] Cables: The offer should include the number of Cat7 and/or TwinAX cables required for host connections to TOR switches, to management switches and to console ports.

4 FUNCTIONAL SOFTWAREA capable cluster management suite and a rich software stack is the key for the success of this project. It should support the offered instruction set architectures (ISAs) as much as possible and complete the heterogeneity of the system.

37

4.1 Cluster Management software M[52] CM: The offer should include a Cluster Manager for Deep Learning & HPC. System should be delivered with license for minimum 30 nodes. The offer should include pricing for additional licenses such that we can bring standalone machines into the shared resource. The Cluster Manager suite must support x86_64, Amd64 andAArch64 (Arm8) ISAs, as well as support systems with GPGPUs from Nvidia and AMD.

M[53] ICC: Systems should be Intel Cluster Compliant19.

M[54] CM support: In our experience, the Cluster Management software often outlives the cluster hardware. Therefore we request that the CM software should comewith 5 years NBD support.

D[55] CM of Power9: Support for OpenPower/Power9 architectures is preferable, but not required. Please document.

M[56] Modules Environment: The CM suite should include support for modules environment20.

M[57] Easy upgrade: The CM suite should provide online repository for easy updates/patches and support online upgrades.

D[58] Toolkits and API: The CM suite should provide repositories for the most common industry standard APIs, toolkits and compilers such that we may only need to buy a license to get started using them.

4.2 Operating systemM[59] OS: The CM distribution should be based on the latest revision of CentOS 7, currently this is CentOS 7.5.

4.3 Global shared File SystemsM[60] BeeGFS w/on-demand: The offer should include BeeGFS and BeeOND licenseing with NBD support.

M[61] BeeGFS support: As for CM software, parallel filesystems outlive parts of thecluster and we therefore request 5 years software support.

D[62] Weka.IO: Is the new kid on the block for HPC global shared filesystems, specifically in regards to ML/AI workloads. It is desirable that contractor offer this as

19 https://software.intel.com/en-us/articles/intel-cluster-checker-release-notes-and-new-features 20 http://modules.sourceforge.net/

38

an option and that the two filesystems co-exist and can be tested against the same hardware/LUNs.

4.4 Batch and scheduler SystemM[63] Slurm: SLURM is open-source and is bundled with most cluster managers, such as Bright Cluster Manager. The cluster manager should include a fairly new revision of Slurm as well as easily support other batch scheduling systems such as Torque/Maui, OpenPBS, PBSpro asf. The CM suite should allow us to run multiple schedulers simultaneously, on parts of the cluster.

4.5 SecurityR[64] Security overlays: No special security compliance over what the standard in RHEL/CentOS 7 will be requested. But we encourage vendors to document optional TCSEC security enhancements offerings over standard RHEL/CentOS 7 ACLs, SElinux extension etc. Please include documentation of any TCSEC or Orange-Book overlays/product offerings.

4.6 Language SupportA rich middleware stack is key to the heterogeneity of the systems. We therefore request contractors to offer compilers, vendor specific MPIs and other tools for the offered ISAs.

4.6.1 Tools and Libraries

D[65] ML/AI Toolkits: Any vendor specific machine learning/AI SDKs/toolkits should be bundled with the respective ISAs offered.

4.6.2 Intel tools for academia

M[66] Intel Parallel Studio: Five (5) floating licenses with latest Intel Parallel Studio XE Cluster Edition 2019 Linux for academic customers.

4.6.3 Intel tools specific to Intel SP Gold 6139P

M[67] Other Intel SDK offerings: Intel SDK offerings for ML/AI and FPGA OpenCL development kits for Arria should be bundled with x86-64 FPGA offerings.

4.6.4 Arm Allinea Studio 18.4 and Performance Libraries

M[68] AArch64 SDK: Any AArch64 offering should include Arm Allinea SDK and Performance Libraries.

39

4.6.5 Nvidia tools for Deep Learning and AI

M[69] Nvidia SDKs: Should be bundled with HGX-2 system.

4.6.6 OpenPower compiler and SDK

M[70] Power SDKs: Should be bundled with any offering of an OCP OpenPower system.

4.7 Third-party SoftwareD[71] Other compilers or SDKs: The offer should include ISA specific compilers and software development toolkits, such as the PGI Professional Edition compiler andtools for Linux.

5 TRAINING, DOCUMENTATION AND COOPERATION

5.1 TrainingD[72] Training: Vendors may offer training, but it is not required. Please include offerings. Knowledge transfer will be a natural part of the installation process.

R[73] Courses: Vendors should provide a list of available courses on hardware and software.

5.2 Ease-of-UseAssuming that the vendor offers comply with the tender requirements, the system willbe easy to manage and administer.

5.3 DocumentationD[74] Documentation: Access to manuals and documentation for all software and hardware should be available online.

40

6 PROGRESS AND TIME SCHEDULE

6.1 Installation of Initial ConfigurationM[75] Install: Vendor should provide one highly skilled engineer to work alongside Simula’s engineer for one week. Please refer to Appendix 4.

6.2 Installation of Complete ConfigurationM[76] Completion: Please refer to Appendix 4.

7 ACCEPTANCE AND APPROVAL TESTS.Please refer to Appendix 5.

8 COMMERCIAL TERMSM[77] Commercial terms: Please refer to Appendix 6 and 7.

8.1 PriceM[78] Pricing: Please refer to Appendix 7. Offer should include a spreadsheet with a summery sheet with the pricing for the various components from section 3 through 6 are listed in separate sheets. The vendor is free to use their quotation systems format on these sheets. For any questions regarding format of spreadsheet please contact torel@simula.no.

8.2 PaymentM[79] Date of payment: For both the initial and complete configuration, the payment is net 30 days after acceptance of the system or as mutually agreed in the final contract.

Interest. Rate of interest is 12 % p.a. at a maximum.

8.3 Terms of DeliveryM[80] Incoterms: Delivery is DDP (Incoterms 2010), Simula Research Laboratory AS, Martin Linges Vei 25, 1325 Fornebu, Norway.

Airport destination: Gardemoen Oslo Airport (OSL) is closest. Further shipping instructions will be provided if necessary. The risk for accidental causes shall pass

41

from the Vendor to the Customer on the day of acceptance. The Vendor shall provide insurance coverage up to this day. The cost incurred may be specified in the Tender.

9 ADDITIONAL PURCHASESM[81] Solution upgrades pricing: In addition to the upgrade from initial to completeconfiguration, Simula expects that there will be several smaller upgrades of the computer system. These may result from

• new requirements from the existing partners in the consortium.

• new partners joining the consortium.

In particular, we would like to invite research groups at the universities to invest in thecomputer system. This will allow them to get access to a substantially larger system than they can afford on their own, and at the same time increase the overall end-user value of the computer system.

The requests for additional upgrades should be seen as a proof of success for both the Customer and the Vendor, and these requests should be handled in a positive way. TheVendor is requested to describe a price mechanism ensuring that all upgrades of the computer system can remain competitive in comparison with alternative procurement.

M[82] Additional processor capacity: The Vendor should quote the price of additional nodes, processors, offload engines, including a proportional increase in memory, at the time of installation of the initial and complete configuration, respectively.

M[83] Additional memory capacity: The Vendor should quote the price of additional memory, preferably in units of 64Gbyte of the density specified in section 3at the time of installation of the initial and complete configuration, respectively.

M[84] Additional disk capacity: The Vendor should quote the price of additional disk capacity, in units of 8Tbyte, at the time of installation of the initial and complete configuration, respectively.

10 SYSTEM MAINTENANCE, AVAILABILITY AND RELIABILITYPhase 1 hardware and software should be delivered with 3 year next business day (NBD) service and support, unless otherwise is stated.

42

10.1 MaintenanceThe new system will be a production platform for many users, and therefore the new system requires a high level of service, support, and maintenance. The Tender shall contain detailed information describing guarantees on how the support organisation provides fast and high-quality after sales service and technical support.

M[85] Maintenance contract: The Vendor must be prepared to enter into a maintenance contract based on the Norwegian Standard SSA-V21. Systems should be delivered with NBD 3 year support unless otherwise stated above.

M[86] Maintenance period: The proposed Tender must include basic hardware and software maintenance until the end of 2022, unless otherwise is requested. The level of maintenance must be sufficient to meet the requirements for availability given below. Furthermore, the cost of software licences and software updates must be included.

M[87] Organisation of maintenance: The Vendor must provide information on how maintenance will be organised. The suggested organisation should take into account the requirements for availability given below. The Vendor is encouraged to suggest an organisation involving the use of Customer personnel in first line maintenance. The Customer will have on-call support outside regular working hours via the Simula and partners consortium.

M[88] Problem coordination responsibilities: In order to improve the speed of resolution of hardware and software problems, the Vendor must take prime contractual responsibility to assist with problem determination and thereby provide one initial point of contact for all types of fault. We request ownership to the problems, such that we avoid being pushed back and forth between subcontractors, if such exists.

M[89] Technical support services: The Vendor should provide technical support and assistance to resolve any queries Simula may have relating to any aspect of the systemincluding: operation system generation, communications network planning, system capacity planning, system performance analysis and tuning, system software internal analysis, operating system conversions and compilers.

R[90] Planned maintenance: The Vendor should provide information on the frequency and duration of required planned maintenance stops.

R[91] Remote diagnosis: The computer systems should support remote diagnosis/ RAS feature, preferably automatic report back functionality to vendor support, such as NetApp AutoSupport and equivalent.

21 https://www.anskaffelser.no/verktoy/vedlikeholdsavtalen-ssa-v

43

R[92] Supply of spare parts: In order to save man power (on all sides) and down-time of the machine, Simula is interested in first-line services with internal personnel, based on a small supply of first line spare parts. The Vendor organisation shall outline whether and how such backup support can be organised, and how quickly spare parts is available on location. We note that such first-line service with internal personnel will require appropriate hardware maintenance and system administration training (see section 5.1). We also note that the hardware maintenance of the computer system will remain the full responsibility of the Vendor (as specified in the maintenance contract).

10.2 Availability and ReliabilityThe below section is not applicable to phase 1, but will be included in later phases. Normal warranty and support applies. Section below is included for completeness andwill be applicable for a larger system.

The Vendor must describe the level of fault tolerance of the system. The Vendor shall show that the system has various kinds of built-in fault tolerance mechanisms. For example, the discussion shall answer the following questions: Can the system operatordisable/replace parts of the machine without taking the (whole) system down (i.e., killing all running jobs)? Will the entire system go down if there is component failure? Can the system still be operated while part(s) of the machine are disabled because of failure? Can the system easily be configured to run (at reduced capacity) while waiting for spare parts? What system interruption is necessary in order to replace certain components and how does this affect scheduled and running jobs?

D[93] Availability: The Vendor should indicate projected system availability.

D[94] Machine Effectiveness Level: The computer system shall have a documented Machine Effectiveness Level (MEL) in excess of 95% per quarter of a year (moving average). Failure to meet this requirement will be regarded as a Defect as defined in the Purchase Agreement. MEL is a percentage figure determined by dividing the time the computer system is available for running jobs by the total time. The computer system is not available for running jobs if it is available for log-on but batch jobs cannot be run. Furthermore, the computer system is defined to be unavailable if less than 75% of any of the following resources in the computer system is available: CPUs, memory,disk, network, memory bandwidth, and I/O bandwidth.

R[95] MTBF: The eX³ project is for experimental HPC infrastructure, and therefore Simula does not request a desired Mean Time Between Failure (MTBF). The Mean Time Between Failure should however be as high as possible for the individual components.

44

D[96] Resilience: The system should be designed and configured so that potential single points of failure are eliminated as much as possible.

R[97] Hot-swap: The Vendor is requested to provide information on which components (nodes, disks, PCIe cards etc.) of the computer system can be replaced while the system remains in operation and which components of the computer system cannot be replaced while the system is in operation. The Vendor is also requested to provide information on road maps for increased hot-swap capabilities for the computer system.

R[98] Reliability: The Vendor is requested to provide information on important measures taken to ensure high reliability of the computer system including the storagesubsystem.

11 SYSTEM PERFORMANCENot mandatory to phase 1, but sections 11.1 and 11.3 are also desirable for phase 1. We still include section 11.2 for completeness to allow vendors to plan for the next tender. We will not be requesting benchmark reports for this procurement as it is for the current top-bin processors and various offload engines.

For the second phase of the eX³ project, we will request performance reports for processors, accelerators and machine learning/AI. In addition, we will provide Simulabenchmark kernels. Results should then be documented in Appendix 11 and all run logs, outputs and modifications to Makefiles and code should be documented in the accompanied tender CD/USB stick.

11.1 Standard BenchmarksD[99] Standard benchmarks: Vendors may optionally include the below as part of phase 1.

• Spec CPU 2017

• McCalpin STREAM og STREAM MPI

• HPL Benchmark

• HPCG High-Performance Conjugate Gradients benchmark

• MLperf

45

11.2 Simula benchmark kernelsD[100] Simula kernels:For all Simula benchmark kernels, there is a README.md file with further instructions. The tgz archive and readme file containing the codes and datasets can for later phases of project be found on the URL https://www.simula.no/ITT092019/

11.2.1 Simula benchmark kernels

• ARM v8 SVE verification

• AVX512 verification

• IEEE754 verification tests such as IEEE754CC++

• Rat heart electrodynamic model

• TBD

11.2.2 Un-optimized “hands-off” runs

Vendor may not modify code with the exception of changing Makefiles, unless modifications to codes are necessary to execute. In case of modifications, vendor mustsend such patch(es) to Simula.Vendor may do compiler flag optimization and use any commercially or academically available libraries for any architecture. All logs of compilation and runs should be documented in tender CD/USB stick.

Vendor may also tune bios settings and provide documentation of settings such that results may be reproduced.

11.2.3 Optimized runs

In these runs, the vendor may optimize and change code. Vendors may use any compiler, accelerators and libraries available. Benchmark report will require code to demonstrate the validity of results after modification (compared to the files provided by Simula), i.e. results must be reproducible and will be part of acceptance testing.

11.3 Directory structure for USB/CD deliverableD[101] Deliverables: In order to ease evaluation of response we ask vendors to make use of a directory structure loosely resembling the below.

Heterogeneous high performance computing systems for eX³ project

46

→ Phase 1 Technology evaluation system for eX³-project ITT document→

Correspondance→ Appendix 1→

ITT092019 docs→ Appendix 2→

Docs→ OCP HGX-2→

Datasheets→ Whitepapers→ Techpubs→

Storage→ Datasheets→ Whitepapers→ Techpubs→

...

Appendix 11 Benchmarks→ Standard Benchmarks→

Docs→ Runs→

Simula benchmark kernels→ Un-Optimized→

Runs→ Optimized→

Runs→ Appendix 12 Non-Disclosure Agreement (NDA) →

→ Phase 2 Heterogeneous HPC system for eX³-project

→ Phase 3 Heterogeneous HPC system for eX³-project

47

II Appendix 2: Contractor description of

the deliverables

https://www.simula.no/

Appendix 2 of Purchase Agreement

Contractors description of thedeliverables.

50

51

Appendix 2: Contractor description of the deliverables

Based on the sections in Appendix 1 (Customer requirements specification) the Contractor shall describe its deliverables/solution herein. This includes how the offered solution will be part of future phase 2 and phase 3 solutions.

The Contractor should be aware of the fact that deviations, reservations or other changes to the Agreement in Appendix 2 to the Customer's requirements in Appendix 1 in connection with the submission of a tender may result in rejection of the tender by the Customer.

The Contractor must respond to each of the requirement in Appendix 7 workbook which includes pricing for the solution as well as the Contractors compliance to the M, D and R requirement as show in the below example.

Requirement # Description Compliance[Y/N]

Vendor Comments

M[1] Heterogeneity Y 3 General purpose CPUs, 2 GPGPUs, GraphCore IPUs, BrainChip SNP’s

.

M[51] CM software Y

.

M[91] Standard benchmarks Y* *Standard benchmark report included, but lacks MLperf at this time.

.

Table 11: Compliance listing example

The Agreement, clause 1.1 Scope of the Agreement

If the Contractor is of the view that there are obvious errors or ambiguities in the Customer requirements specification, the Contractor shall point this out here.

52

The Agreement, clause 2.1.1 Software and equipment If offered software and equipment do not have such functions, properties and quality as described by standard product descriptions/specifications, user manuals, etc. that the Contractor includes upon sales of these products, this shall be stated here.

If it is necessary to upgrade the technical platform of the Customer, as described in Appendix 3, in order for the deliverables due from the Contractor to work as agreed, this shall be specified here.

The Agreement, clause 2.1.3 Applicability of standard licence and agreement terms and conditionsTo the extent that standard software that is included in the deliverables must be delivered with standard licence terms and conditions, this shall be stated here. Copies of the licence terms and conditions shall be appended as Appendix 10.

To the extent that the provisions of licence terms and conditions governing right of disposal differ from this Agreement's provisions governing right of disposal, this shallbe described here.

The Agreement, clause 2.1.6 Warranty period and guaranteed performanceIf the Contractor stipulates requirements in respect of the maintenance that must be performed in order for equipment warranties to be valid, this shall be specified here.

The Agreement, clause 2.7 External legal requirementsThe Contractor shall describe how the Contractor addresses external legal requirements through its deliverables here.

The Agreement, clause 4.3 Free softwareIf free software is to be used in connection with the deliverables, the Contractor shall prepare an overview of the relevant free software. The overview shall be inserted here. Copies of applicable licence terms and conditions for the relevant free software shall be appended in Appendix 10.

To the extent that the Contractor is aware that free software that the Customer has requested be used as part of the deliverables, is unsuited to satisfying the Customer’s

53

requirements or, infringes, or is alleged by anyone to infringe, third party copyrights, the Contractor shall point this out here.

54

III Appendix 3: Customer technical

platform

Appendix 3 of Purchase Agreement

Customer technical platform.

58

Appendix 3: Customer technical platform

11.4 CURRENT HARDWARE PLATFORMS

11.4.1 Current systems

Currently most of Simula’s compute systems are standalone. A capable Cluster Management system will allow us to bring the existing systems into the cluster as a global shared resource. Please refer to Appendix 1 section 4.1.

11.4.2 Application stack

Simula and the consortium maintains a large academic and commercial application stack. It is therefore important that modules environment is a integral part of the cluster management system.

11.4.3 Software Development Environment

Simula and the consortium maintains a large set of academic and commercial SDK, compilers and libraries. Integral to the solution is the cluster management software. With it we hope to streamline and make available software packages as a shared resource for all users.

11.4.4 CPU-based applications

A large collection of academic research codes. TBD.

11.4.5 GPU-based applications

Most frequently used GPU applications

TBD.

Available GPU applications

TBD.

11.5 PHYSICAL INFRASTRUCTURE

11.5.1 Street address

Simula Research Laboratory AS,Martin Linges Vei 25,

59

1325 Fornebu, Norway.

The nearby hotel Scandic Fornebu is available for accommodations for service engineer during the installation week.

11.6 DC constructionThe computer room has a limited raised data-floor. Clearance underneath is only approximately 15cm. Connection to electric power will be below the raised floor. Network basket will be available above racks for between rack connectivity. There is no water cooling available for phase 1 deployment. Maximum load on the raised flooris roughly 2000 kg/m².

11.7 Network connection between TOR stack and simula networkingThe TOR switch in rack 1 or IO-rack will be uplinked to Simula router using a 10Gbps LC-LC SR multimode fiber.

11.8 Rack sizeMaximum rack height is 208cm or approximately 46u. Data center ceiling is much higher and if racks does not come populated, and if they may be tilted we suggest using 48u racks. They need to be tilted slightly to get them into the elevator and through datacenter doors. Please refer to section 11.14.

11.9 PowerPower distribution under the raised floor is delivered through IEC60309 3-phase receptacles. Each rack will be powered through twopower cables, delivering 3-phase 32A/400V each. Nocentral UPS system is available at this phase, butdistribution is a standard filtered distributed TN-Ssystem.

60

Illustration 1: IEC 60309 (6H), BS 4343, DIN 49462/63, CEE 17, EN 60309

11.10 CoolingData centers at Simula are ambiently cooled through two (2) redundant Emicon coolers. Large heat extraction pipes draw heat out of the room. As part of A07 spreadsheet section “A07-11.9-10 - Solution Physical requirements”, the offer should include BTU/hr and kW/hr rating such that we can plan for it in advance. It is vital forthe projects success that as soon as the contract is signed Vendor invites to an initial Site Preparation Meeting with Customer to clarify all solution requirements.

11.11 Fire protectionAutomatic fire extinguisher system in operation already. No Inergen fire suppression system is installed at the moment.

11.12 Shipment and transportationPlease refer to Appendix 1 section 8.3.

11.13 Special requirements in the computer roomAny requirements for the installation should be addressed in Site Preparation Meetingimmediately after contract has been signed. The cost for required modifications are covered by the Customer.

61

11.14 Photos from the site

62

Illustration 3: Trunck unload area.

Illustration 4: Entrance door. 208cm clearance.

Illustration 2: From front door to elevator is flat.

63

Illustration 5: Elevator has 208cm height clearance.

64

Illustration 6: Simula entrance door. 208cmheight clearance and 80cm width.

Illustration 7: Entrance to dataroom 1. 208cm height clearance, more than 80cm width.

The door is raised approximately 10cm from floor level. Lift will be provided to lift it to data-room floor level.

The Agreement, clause 1.1 Scope of the AgreementIf in Appendix 1 the Customer stipulates that the deliverables shall function together with the Customer's current technical platform, the Customer's technical platform shall be described here.

If it is relevant for the deliverables, the Customer should also describe the architectureand the system landscape in which the software being procured will be installed.

The versions of the software included in the technical platform should also be stated here. The Customer should also relevant to describe the software's upgrade schedule, any planned major upgrades and routines for periodic maintenance.

65

IV Appendix 4: Delivery date and other deadlines

Appendix 4 of Purchase Agreement

Delivery day and otherdeadlines.

69

Appendix 4: Delivery date and other deadlinesDeadline Dato

Tender procurement announcement October 19th 2018

Tender deadline (fixed) November 19th 2018, 12:00 CEST

Tenders opening (fixed) November 19th 2018, 13:00 CEST

Award of contract November 26th 2018

Contract signing December 6th 2018

Customer takes delivery 4-8 weeks after signatureTable 12: Tentative progression plan

Project and progress plan

Milestone Days Date

1. Agreement is signed. - December 6th 2018

2. Customer has received written notice that the solution is ready for the Factory Acceptance Test

3. Time limit for installation preparation, and the Contractor’s final inspection, ref SSA-K, 2015 2.1.5

4. Arrival on site and start of installation and mounting

5. Customer has received written notice that the solution is ready for the Customer acceptance test

6. Customer staff to be certified in using the offered Solution, shall be fully trained by this date

7. The Customer has conducted the acceptance test andsent the Contractor notice that the test was approved

8. The approval period has expired and the Customer has sent notice that the deliverables have been approved (delivery date)

9. Expiry of maintenance & support contract

10.

Expiry of Maintenance & Support Contract extension option

Table 13: Project progression plan

70

The Agreement, clause 2.1.2 TrainingThe training described as a part of the deliverables, the date(s) set out in the Table 1. No additional training except for installation collaboration will be required for this tender.

The Agreement, clause 2.2.4 Performance of the Customer acceptance testThe acceptance test (Appendix 5) shall be commenced and completed in accordance with the deadlines set out in Table above.

The Agreement, clause 2.2.6 CommissioningThe schedule for preparing for commissioning is set out here.

The Agreement, clause 5.3 Use of subcontractorsSubcontractors that are approved shall be specified here.

The Agreement, clause 11.5.2 Liquidated damages in the case of delayLiquidated damages in the case of delay will be not be carried out in this phase of the project.

71

V Appendix 5: Approval

test

Appendix 5 of Purchase Agreement

Approval test.

75

Appendix 5: Approval test

A functional verification that systems are in working condition will be done, and standard service and product warranties apply.

This phase of the project does not include any formal approval test, verification of benchmark results specified in Appendix 1 section 11. The later phases of this procurement will include approval test and verification of benchmark runs.

The Agreement, clause 2.2.2 Duty to examineIf Appendix 1 stipulates that a specific approval test shall be performed, the type and scope of this test shall be described in detail here.

If error definitions other than those stipulated in the Agreement's clause 2.2.2 are agreed, the Customer shall specify this here.

If the Customer wants to specify other or more detailed criteria for approving the deliverables which differ from what is stipulated in the Agreement's clause 2.2.2, the Customer shall specify this here.

If deadlines for approving the deliverables other than those stipulated in the Agreement's clause 2.2.2 shall apply, this shall be stated here.

76

VI Appendix 6: Administrative

provisions

Appendix 6 of Purchase Agreement

Administrative provisions.

80

Appendix 6: Administrative provisions

All communication regarding this procurement should go through the Mercell system as explained in accompanying document “Rules of the Competition”.

For a introduction into Mercell communication module, please refer to the guide.

If bidder find the tender unclear or lacking information, and the bidder is unable to use Mercell InfoCenter system, one may send mail or e-mail to:

Simula Research Laboratory ASc/o Tore H. LarsenBoks 134, 1325 Lysaker

torel@simula.no

Written questions sent by mail should be marked “Questions regarding Tender eX³ ITT 10/2018”. Questions must be sent at least 6 days prior to the tender deadline date. Questions received after this date may not be answered.

Answers to questions received will be sent to all bidders by using Mercell InfoCenter messaging system referred to herein. This will ensure that all bidders automatically receive the same answer simultaneously anonymously by e-mail. No questions will beanswered on an individual basis.

Erratas, additions and changes to tenderBefore the end of the deadline, the Contractors have the right to make corrections, additions and changes to the tender that are not essential. Such changes will be published to all bidders simultaneously through Mercell InfoCenter messaging system, as well as connected systems Doffin and TED. Simula takes no responsibility of failure of the systems. Bidders are encouraged to actively monitor the various systems messaging system for this tender.

If erratas, additions and changes to tender is coming to late for bidders to be able to answer, Simula will consider extending the tender deadline. In such a case all bidders are informed through the messaging system and will have equal opportunity to postpone responds.

In case the bidders find mistakes or information lacking pertaining to this tender, bidders are encouraged to inform Simula in writing as soon as possible.

81

Customer Representatives

Role Name E-mail Phone

Administrativeresponsible

Are Magnus Bruaset

arem@simula.no

CEO Dr. Aslak Tveito aslak@simula.no

Project Manager Tore H. Larsen torel@simula.no +47 918 33 670

Below the Contractor must include whom is responsible for the tender as well engineers responsible for execution/installation. Please include company transcripts/CV for each engineer.

Contractor Representatives

Name Position E-mail Phone

Sub-Contractor Representatives

Name Position E-mail Phone

Wages and working conditionsDocumentation showing the Contractor's compliance with the Contractor's obligationsas stipulated in clause 5.5 of the Agreement (Wages and working conditions) shall be inserted here. The documentation can consist of either an appended self-declaration ora third-party declaration showing conformity between the relevant collective wage agreement and the actual wages and working conditions for the performance of the Contractor's and any subcontractors' obligations.

82

The Agreement, clause 1.5 The representatives of the partiesThe authorised representatives of the parties, as well as procedures and notice periods for any replacement thereof, shall be specified in more detail here.

The Agreement, clause 2.4 Wages and working conditions Documentation showing the Contractor's compliance with the Contractor's obligationsas stipulated in clause 2.4 of the Agreement (Wages and working conditions) shall be inserted here. The documentation can consist of either an appended self-declaration ora third-party declaration showing conformity between the relevant collective wage agreement and the actual wages and working conditions for the performance of the Contractor's and any subcontractors' obligations.

Further clarification concerning the implementation of the Agreement's clause 2.4 may be agreed here.

The Agreement, clause 2.6 Form of communication - in writingIf the Customer wants notices, requirements or other enquiries associated with this Agreement to be provided in a manner other than in writing and sent to the postal address or electronic address provided on the first page of the Agreement, this shall bestated here.

83

84

VII Appendix 7: Total price and

pricing provisions

Appendix 7 of Purchase Agreement

Total price and pricingprovisions.

Appendix 7: Total price and pricing provisions

General about provisions

All prices shall be quoted exclusive of Value Added Tax, but include customs duties and any other indirect taxes. All prices shall also include insurance until risk is transferred to the Customer, shipping, installation and mounting (including racks) and return of packaging. Risk is transferred to the Customer on the installation date, as perSSA-K, 2013,15.1.

Offer should include a spreadsheet with a summery sheet with pricing for the various components from Appendix 1 section 3 through 5 is listed in separate sheets as per example workbook “A07A Appendix 7 Price and requirement matrices”. For any questions regarding spreadsheet torel@simula.no. Contractor's are free to adopt it to their liking within reason as long as the general format remains the same. Technical maintenance and software support for three years is to be included in the price, exceptfor cluster management and shared filesystem software which should include 5 year support. Key components such as headnodes and enterprise level storage should include 5 years support. Prices in workbook shall be given in NOK.

With regards to the prices for expansions and the extension of the technical maintenance and software support agreement for the fifth year, the Contractor is free to choose currency denomination, but the domination chosen shall be clearly stated.

The currency exchange rate between the offer currency and NOK, on the day of the tender deadline, will be used for the evaluation of the Tender. The costs for expandingthe Solution shall be included for the respective nodes. In the event that the Customer requires additional network equipment, the Contractor shall state the discount on catalogue price, as a separate option in the workbook under section “3.12.1 - 10Gbps TOR Ethernet Switch”.

Terms of Payment

The payment plan is that Customer pays the total amount 30 days after final approval by the Customer.

88

Invoicing

Invoicing is done according to the progress plan described in A04 Appendix 4 Project and Progress Plan Table 1 milestones.

Invoice address:

Simula Research Laboratory ASPostboks 1341325 LysakerNorway

Electronic invoices: invoice@simula.no

Invoice shall be marked with the following informationReference number: eX³ ITT 10/2018

Name: CEO Dr. Aslak TveitoOther:

All prices and the detailed terms governing the consideration to be paid by the Customer for the deliverables provided by the Contractor shall be set out here in Appendix 7. The Customer must think through the price format (hourly rate, unit price, fixed price, etc.). The Contractor shall base its tenderon this and create templates for this in Appendix 7. Any special payments arrangements such as discounts, prepayments, part payments, and different payment dates must also be specified.

The Agreement, clause 2.1.2 Customisations and installation, etc. The Contractor's hourly rates for performing customer customisations, installation, etc. shall be stated here.

If it has been agreed that the Contractor will deliver services in the form of customer customisations, installation or other tasks after the software and equipment have been handed over or made available to the Customer, and an error occurs in software that iscovered by clause 2.1.3, paragraph six, of the Agreement, a maximum financial limit for the Contractor's obligation to work out temporary solutions that work around such errors can be agreed. If such a limit is agreed, the upper limits shall be stated here.

Example price table for the Contractor's standard hourly rates for consultancy services:

89

Description Hourly rate, excl. VATJunior consultant (e.g. 0-3 years' experienceSenior consultant (e.g. 3-10 years' experience)

Table 14: Hourly rates

The Agreement, clause 2.1.4 Documentation and trainingIf training forms part of the deliverables pursuant to Appendix 1, the consideration fortraining shall be specified here.

The Agreement, clause 2.1.6 Warranty period and guaranteed performanceIf the warranty period will deviate from clause 2.1.6 of the Agreement, this shall be stated here.

The Agreement, clause 3.1 ConsiderationAll prices and the detailed terms governing the consideration to be paid by the Customer for the deliverables provided by the Contractor shall be set out here.

If prices shall be stated exclusive of Value Added Tax, but including customs duties and other indirect taxes, the Customer shall specify the alternative pricing system here.

If the Customer will allow prices for components that are delivered from abroad to be stated in a foreign currency, this shall be stated here.

If software and equipment shall not be delivered DPP (Incoterms) to the address stated on the first page of the Agreement, the alternative risk model shall be specified here.

Example of price table for goods: Goods Reference to description in

Appendix Price excl. VAT Discount

Table 15: Example price table for goods

90

Example of price table for services: (e.g. courses/training, unit price for installation, etc.): Services Reference to description in

Appendix Price excl. VAT Discount

Table 16: Example table for services

The Agreement, clause 3.2 Invoicing dates and payment termsIf consideration for software or equipment shall not be invoiced on the date delivery isdeemed to have taken place pursuant to clause 2.1.5, the alternative invoicing date shall be stated here.

Other payment terms, and any terms and conditions relating to the use of EHF, shall be set out here.

The Agreement, clause 3.5 Price adjustmentsAny price adjustment provisions other than those stipulated in the Agreement's clause 3.5 that have been agreed, shall be set out here.

The Agreement, clause 4.1 Right of ownership of equipmentIf a vendor's fixed charge shall be agreed for equipment that is delivered pursuant to this Agreement, this shall be stated here.

The Agreement, clause 4.2.1 Limited right of disposalIf the right of disposal terms and conditions shall deviate from clause 4.2.1 of the Agreement, this shall be stated here. Any assumptions and restrictions in relation to the number of users, place/equipment for exercising the right of disposal, etc. must also be described here.

91

The Agreement, clause 4.3 Free softwareIf the Contractor has special hourly rates for assisting the Customer with the activities designated "supplementary and chargeable service" in clause 4.3, paragraph six, of theAgreement, these shall be stated here.

92

VIII Appendix 8: Changes to the

general contractual wording

Appendix 8 of Purchase Agreement

Changes to the generalcontractual wording.

96

Appendix 8: Changes to the general contractualwording

Changes to the general contractual wording shall be set out here, unless the general contractual wording refers such changes to a different Appendix.

Changes can be made to all the clauses in the Agreement, even where there is no clear reference to the fact that changes can be agreed. Changes to the contractual wording shall be specified here so that thewording of the general contractual wording remains unchanged. It must be stated clearly and unequivocally which clause or clauses in the Agreement have been changed and the result of the changes.

The Contractor should, however, be aware of the fact that deviations, reservations or changes to the Agreement in connection with the submission of a tender may result in rejection of the tender by the Customer.

Example of change table:

Clause of the Agreement Shall be replaced byChapter x.x.x, paragraph y New formulation/text

Table 17: Change table for contract

97

IX Appendix 9: Changes subsequent to the conclusion of

the Agreement.

Appendix 9 of Purchase Agreement

Changes subsequent to theconclusion of the Agreement.

101

Appendix 9: Changes subsequent to the conclusion of the Agreement

The Contractor shall maintain a directory of the changes on an ongoing basis, which directory shall form Appendix 9, and shall without undue delay provide the Customer with an updated copy thereof. The Customer must maintain its own overview of the change requests it has sent, the change estimates it has received, and the changes order it has issued.

Example of change directory:

Change no. Description Effective date Archive reference

Table 18: Change table example

102

X Appendix 10: License terms and

conditions for standard and free

software

Appendix 10 of Purchase Agreement

License terms and conditionsfor standard software

and free software.

106

Appendix 10: Licence terms and conditions for standard software and free software

Copies of applicable licence terms and conditions for standard software and free software shall be appended in Appendix 10.

The Agreement, clause 2.1.3 Applicability of standard licence terms and conditions and agreement terms and conditionsCopies of the licence term and conditions for standard software included in the deliverables must be appended here.

The Agreement, clause 4.3 Free softwareIf the deliverables include free software, copies of the licence terms and conditions that apply for the relevant free software must be inserted here.

107

XI Appendix 11: Standard

Benchmarks reports and

SRL benchmark results

Appendix 11 of Purchase Agreement

Standard benchmark reportsand SRL benchmark results.

111

XII Appendix 12: Non-Disclosure

Agreement

Appendix 12 of Purchase Agreement

Non-Disclosure Agreementbetween Contractor and

Customer

115