intel® manycore platform software stack (intel®...

Intel® Manycore Platform Software Stack

(Intel® MPSS)

User's Guide for Windows*

April 2017

Copyright © 2013-2017 Intel Corporation

All Rights Reserved

US

Revision: 4.4.0

World Wide Web: http://www.intel.com

Intel® Manycore Platform Software Stack (Intel® MPSS) User's Guide for Windows* April 2017 2

Disclaimer and Legal Information

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter

drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from

published specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-

4725 or by visiting: http://www.intel.com/design/literature.htm

Intel, the Intel Logo, the Intel Inside logo, Xeon, Intel Xeon Phi, Intel Vtune are trademarks or registered trademarks of intel corporation or its subsidiaries in the U. S. and/or other countries.

*other names and brands may be claimed as the property of others.

Copyright © 2017, Intel Corporation. All rights reserved.

Intel® Manycore Platform Software Stack (Intel® MPSS) April 2017 User's Guide for Windows*

3

Revision History

Revision Number

Description Revision Date

4.4.0 Document update for the release of the Intel® MPSS 4.4.0 March 2017

4.3.3 Document update for the release of the Intel® MPSS 4.3.3. February 2017

4.3.2 Document update for the release of the Intel® MPSS 4.3.2.

Updated the guide’s structure to be more parallel with the user’s guide for Linux*. Updated information about systools.

January 2017

4.3.1 Document update for the release of the Intel® MPSS 4.3.1. November 2016


Added instructions on how to establish bridged network

connections with coprocessors, updated SSH access instructions, and supplemented Appendix B with description of the Intel®

MPSS System Tools.

September 2016

4.2.2 Document update for the release of the Intel® MPSS 4.2.2. Added information on new configuration parameters.

September 2016


Improved document structure, new Appendix A with coprocessor

configuration parameters, updated configuration instructions reflecting changes in the structure of configuration files.

August 2016

4.2.0 Document update for the release of the Intel® MPSS 4.2.0 June 2016

4.1.1 Release of the Intel® MPSS 4.1.1 User’s Guide for Windows* April 2016


Table of Contents 1 About this manual .......................................................................................... 8

1.1 Overview of this document ............................................................................... 8 1.2 Notational conventions .................................................................................... 8

1.2.1 Command syntax ................................................................................. 9 1.3 Terminology ................................................................................................... 9

2 Intel® MPSS at a glance ................................................................................ 12

2.1 Coprocessor hardware and system architecture ................................................ 12 2.2 Programming models..................................................................................... 14

2.2.1 Offload programming model ................................................................ 14 2.2.2 Symmetric programming model ........................................................... 14 2.2.3 Native programming model ................................................................. 15

2.3 Intel® MPSS software architecture and components .......................................... 15 2.3.1 The coprocessor operating system ....................................................... 15 2.3.2 Intel® MPSS middleware libraries ......................................................... 15 2.3.3 Intel® MPSS drivers and services ......................................................... 15 2.3.4 Tools and utilities ............................................................................... 16

2.4 Related documentation .................................................................................. 17

3 Installation process overview ....................................................................... 19

3.1 Hardware and software prerequisites ............................................................... 19 3.1.1 BIOS configuration ............................................................................. 19 3.1.2 Supported host operating systems ....................................................... 20 3.1.3 Software prerequisites ........................................................................ 20

3.2 Installing Intel® MPSS ................................................................................... 20 3.2.1 Intel® MPSS installation ...................................................................... 20 3.2.2 Upgrade instructions .......................................................................... 21 3.2.3 Installing Windows* cross-sdk ............................................................. 21 3.2.4 Uninstalling Intel® MPSS ..................................................................... 21

3.3 Updating coprocessor’s firmware ..................................................................... 21 3.4 Initial coprocessor boot.................................................................................. 22

3.4.1 Starting the coprocessor ..................................................................... 22 3.4.2 Initial configuration ............................................................................ 23 3.4.3 Specifying administrator’s SSH key ...................................................... 23

3.5 Validating Intel® MPSS installation................................................................... 24

4 Coprocessor OS configuration ....................................................................... 25

4.1 Configuration files ......................................................................................... 25 4.1.1 Updating configuration files ................................................................. 25

4.2 Boot configuration ......................................................................................... 25 4.2.1 Specifying the Linux* kernel ............................................................... 26 4.2.2 Coprocessor-side kernel command line parameters ................................ 26 4.2.3 Coprocessor’s root file system ............................................................. 26 4.2.4 Automatic boot .................................................................................. 26

4.3 Basic administration tasks .............................................................................. 27 4.3.1 Adding groups and users to the coprocessor’s file system ....................... 27 4.3.2 Removing users and groups from the coprocessor’s file system ............... 27 4.3.3 Specifying the host SSH keys .............................................................. 28

4.4 Troubleshooting ............................................................................................ 28


5

5 Network Configuration ................................................................................. 30

5.1 MAC address assignment ............................................................................... 30 5.2 Host firewall configuration .............................................................................. 30 5.3 Supported network configurations ................................................................... 30

5.3.1 Static pair ......................................................................................... 31 5.3.2 Bridging ............................................................................................ 32

5.4 Interacting with the coprocessor ..................................................................... 37 5.4.1 External tools .................................................................................... 37 5.4.2 SSH access to the coprocessor ............................................................ 37 5.4.3 Transferring files to and from the coprocessor ....................................... 40

6 Managing the coprocessor’s file system ........................................................ 43

6.1 Adding persistent files to the coprocessor’s file system ...................................... 43 6.1.1 Using the smart package manager ....................................................... 44 6.1.2 Editing files interactively ..................................................................... 44 6.1.3 Transferring files over SSH.................................................................. 44 6.1.4 Installing RPM packages (root only) ..................................................... 44

6.2 Virtual block devices ...................................................................................... 44 6.3 Host file system share ................................................................................... 45

6.3.1 Using the server manager ................................................................... 46 6.3.2 Using PowerShell* cmdlets to create an NFS share (optional) .................. 54

6.4 CIFS share on Windows* ............................................................................... 55 6.4.1 Configuring a CIFS share .................................................................... 55 6.4.2 Mounting a temporary CIFS share ........................................................ 59 6.4.3 Troubleshooting ................................................................................. 59

7 Intel® MPSS component configuration and tuning ........................................ 60

7.1 Coprocessor operating system configuration ..................................................... 60 7.1.1 Disabling and enabling the memory control group (cgroup) ..................... 60

7.2 Enabling Windows* mic gdb debugging for offload processes .............................. 60 7.3 Enabling Windows* mic debugging for myo applications .................................... 61 7.4 COI configuration .......................................................................................... 61

7.4.1 Limiting maximum number of COI processes ......................................... 61 7.4.2 COI offload user options ..................................................................... 61 7.4.3 Micuser ownership ............................................................................. 61

8 Intel® MPSS tools.......................................................................................... 63

8.1 micctrl ......................................................................................................... 63 8.1.1 Checking the coprocessor’s state ......................................................... 63 8.1.2 Booting the coprocessor ..................................................................... 64 8.1.3 Starting the coprocessor ..................................................................... 64 8.1.4 Shutting down the coprocessor ............................................................ 64 8.1.5 Stopping the coprocessor .................................................................... 64 8.1.6 Resetting the coprocessor ................................................................... 65 8.1.7 Rebooting the coprocessor .................................................................. 65 8.1.8 Waiting for the coprocessor state change .............................................. 65 8.1.9 Restoring the default PFS image .......................................................... 65 8.1.10 Restoring the coprocessor default configuration ..................................... 66 8.1.11 Downloading the coprocessor kernel log ............................................... 66 8.1.12 Displaying the MPSS log ..................................................................... 66

8.2 micinfo ........................................................................................................ 66 8.2.1 Options ............................................................................................ 66


8.2.2 Usage ............................................................................................... 67 8.3 miccheck ..................................................................................................... 68

8.3.1 Options ............................................................................................ 69 8.3.2 Examples .......................................................................................... 69

8.4 micsmc-cli.................................................................................................... 70 8.4.1 Options ............................................................................................ 70 8.4.2 Usage ............................................................................................... 70 8.4.3 Examples .......................................................................................... 72

8.5 micfw .......................................................................................................... 73 8.5.1 Options ............................................................................................ 73 8.5.2 Usage ............................................................................................... 73 8.5.3 Examples .......................................................................................... 74

8.6 micflash ....................................................................................................... 75 8.6.1 Options ............................................................................................ 75 8.6.2 Usage ............................................................................................... 75 8.6.3 Examples .......................................................................................... 76

8.7 micbios ........................................................................................................ 77 8.7.1 Options ............................................................................................ 77 8.7.2 Usage ............................................................................................... 77

8.8 The libsystools library .................................................................................... 79 8.8.1 Function groups ................................................................................. 79

A Intel® MPSS configuration parameters ......................................................... 80

A.1 Meta configuration ........................................................................................ 80 A.1.1 Including other configuration files ........................................................ 80

A.2 Boot control ................................................................................................. 80 A.2.1 EfiImage ........................................................................................... 80 A.2.2 KernelImage ..................................................................................... 81 A.2.3 KernelSymbols .................................................................................. 81 A.2.4 InitRamFsImage ................................................................................ 81 A.2.5 AutoBoot .......................................................................................... 81

A.3 File system configuration parameters .............................................................. 82 A.3.1 RootFsImage ..................................................................................... 82 A.3.2 RepoFsImage .................................................................................... 82 A.3.3 BlockDevice ...................................................................................... 83

A.4 Kernel configuration ...................................................................................... 83 A.4.1 KernelExtraCommandLine ................................................................... 83

A.5 Network configuration ................................................................................... 83 A.5.1 Host network configuration ................................................................. 84 A.5.2 Coprocessor network configuration ....................................................... 84 A.5.3 MAC address assignment .................................................................... 85

B Intel® MPSS boot process ............................................................................. 86

B.1 Kernel command line ..................................................................................... 86 B.2 Instruct the driver to boot the coprocessor ....................................................... 86 B.3 Coprocessor Linux* kernel execution ............................................................... 87 B.4 Coprocessor’s root file system ........................................................................ 87


7

List of Figures Figure 1 Example workstation configuration ............................................................................ 12 Figure 2 Intel® Xeon Phi™ coprocessor x200 architecture ......................................................... 13 Figure 3 Spectrum of programming models ............................................................................. 14 Figure 4 Static pair configuration ........................................................................................... 31 Figure 5 Internal bridge configuration ..................................................................................... 33 Figure 6 External bridge configuration .................................................................................... 36 Figure 7 Persistent file system concept ................................................................................... 43 Figure 8 Boot process for Intel® MPSS .................................................................................... 86

List of Tables Table 1 Supported host operating systems .............................................................................. 20

§

About this manual


1 About this manual

This guide is intended to provide you with a firm understanding of the Intel® Manycore Platform Software Stack (Intel® MPSS), what it is, how to configure it, and how to use its components.

This document pertains to systems containing at least one Intel® Xeon Phi™

coprocessor x200. Please note that utilizing systems with both Intel® Xeon Phi™ coprocessor x200 and previous generation coprocessors is not supported.

1.1 Overview of this document

Section 2 provides a high level overview of the Intel® Xeon Phi™ coprocessor x200

design and Intel® MPSS architecture.

Section 3 is a step-by-step guide to installing Intel® MPSS. It also shows basic

configuration steps.

Section 4 shows how to configure the coprocessor OS

Section 5 describes supported network configurations, explains when each might be

used, and shows how to configure them. Additionally it shows how to interact with the coprocessor using a selection of external tools.

Section 6 presents methods for managing the coprocessor’s file systems, describes how to add supplementary software to the coprocessor OS, and shows how to

configure and mount NFS and CIFS shares on the coprocessor.

Section 7 presents configuration options for Intel® MPSS components, including the coprocessor’s Linux* kernel, and the COI offload interface

Section 8 lists tools distributed with the software stack.

Appendix A describes the Intel® MPSS-specific configuration parameters.

Appendix B is an in-depth description of the coprocessors’ boot process.

1.2 Notational conventions

This document uses the following notational conventions.

micctrl --start Commands and their arguments in prose sections are italicized.

C:\Program Files\Intel\MPSS\ Files and directories in prose sections are italicized.

About this manual


9

micN

Indicates any coprocessor name – mic0, mic1, mic2 etc. where N=0, 1, 2 … 255 (e. g. the micN.conf file resembles any of the files mic0.conf, mic1.conf etc.).

%INTEL_MPSS_HOME% Indicates the software stack installation folder.

By default it is C:\Program Files\Intel\MPSS

COURIER text Indicates code and commands entered by the user. A backslash \ indicates the command continues into the next line.

Italic COURIER text OS terminal output.

“User>” Command entered on the host with user privileges.

“Admin>” Command entered on the host with administrator privileges.

“[micN]$” Command entered on a coprocessor with user or root privileges.

“[micN]#” Command entered on a coprocessor with root

privileges.

1.2.1 Command syntax

The following conventions are used in this document:

<…> indicates a variable value to be supplied.

[…] indicates an optional component.

(x|y|…|z) indicates a choice of values.

Example syntax of the micctrl --start command:

Admin> micctrl –-start [mic0|…|micN]

The optional [mic0|…|micN] parameter indicates a space-separated list of

coprocessors, for example mic0 mic2 mic3 etc.

1.3 Terminology

ABI Application Binary Interface

CCL Coprocessor Communication Link

About this manual


COI Coprocessor Offload Infrastructure

Coprocessor Intel® Xeon Phi™ coprocessor x200

DHCP Dynamic Host Configuration Protocol

GDB GNU Debugger

HCA Host Channel Adapter

IPoIB Internet Protocol over InfiniBand*

K1OM Architecture of the Intel® Xeon Phi™ Coprocessor x100 Product Family

LDAP Lightweight Directory Access Protocol

Lustre A parallel, distributed file system

MAC Media Access Control

MIC Many Integrated Core

MPI Message Passing Interface

MPSS Intel® Manycore Platform Software stack

MYO Mine, Yours, Ours shared memory infrastructure

NIS Network Information System

OFED Open Fabric Enterprise Distribution

PCIe3 PCI Express Gen 3.0

QPI Intel® QuickPath Interconnect

RHEL* Red Hat* Enterprise Linux*

RPM RPM Package Manager

About this manual


11

SCIF Symmetric Communication Interface

SLES* SUSE* Linux* Enterprise Server

SMP Symmetric Multi-Processor

SSD Solid State Drive

SSH Secure Shell

Sysfs A virtual file system

VEth Virtual Ethernet

X86_64 Architecture of the Intel® Xeon Phi™ Coprocessor x200 product family

§

Intel® MPSS at a glance


2 Intel® MPSS at a glance

This section provides an overview of the Intel® Manycore Platform Software Stack. It contains a high level description of Intel® Xeon Phi™ coprocessor x200 hardware and system architecture, discusses programming models Intel® MPSS is designed to support, shows how various components support those programming models, and concludes with a listing of other available documentation.

2.1 Coprocessor hardware and system architecture

The coprocessor is a PCIe add-in card designed for installation into an Intel® Xeon™

processor-based platform. A typical platform configuration is shown in Figure 1.

Figure 1 Example workstation configuration

The coprocessor is composed of up to 72 cores, each supporting 4 hardware threads and standard AVX 512 instructions. The coprocessor is equipped with up to 16GB of

on-package, high-bandwidth MCDRAM memory (there is no traditional DDR memory). The coprocessor has a PCIe interface working as a Gen3 PCIe root port. A Non-Transparent Bridge (NTB) chip is used to connect the device as a PCIe end port to the host.



13

The coprocessor has no permanent file system. Instead, the file system is

implemented on the virtual block device located in the host’s file system.

Each coprocessor runs a standard Linux* kernel with some minor accommodations for the MIC hardware architecture. Because it runs its own OS, the coprocessor is not cache coherent with the host or other PCIe devices.

Figure 2 Intel® Xeon Phi™ coprocessor x200 architecture



2.2 Programming models

The Offload, Symmetric and Native programming models offer a diverse range of usage models. An overview of these options is depicted in Figure 3.

Figure 3 Spectrum of programming models

2.2.1 Offload programming model

In the Offload model, one or more processes of an application are launched on one or

more host processors. These processes, represented in the figure by main(), can offload computation, represented by work(), to attached coprocessors, taking advantage of the many-core architecture with its wide vector units and high memory bandwidth. If the application is composed of more than one process, the processes often communicate using some form of message passing, such as MPI (Message Passing Interface) thus MPI_*() is shown on the host. This offload process is

programmed via the use of offload pragmas supported by the Intel® C/C++ and FORTRAN compilers. When an application is created with one of these compilers, offloaded execution will fall back to the host in the event that a coprocessor is not available. This is why an instance of work() is shown on the host as well.

2.2.2 Symmetric programming model

The Symmetric programming model is convenient for existing HPC applications that

are composed of multiple processes, each of which could run on the host or the

coprocessor, and use some standard communication mechanism such as MPI. In this model, computation is not offloaded but rather remains within each of the processes comprising the application. In such cases, where the application is MPI based, OFED enables high bandwidth/low latency communication using installed Intel® True Scale or Mellanox* InfiniBand* Host Channel Adapters.



15

2.2.3 Native programming model

The Native programming model is a variant of the symmetric model where one or more processes of an application are launched exclusively on one or more coprocessors. From the Intel® MPSS architecture perspective, these programming models typically depend on SCIF and the Virtual Ethernet (VEth) driver to launch processes on the coprocessor(s).

2.3 Intel® MPSS software architecture and components

2.3.1 The coprocessor operating system

Underlying all computation on the coprocessor is a standard Linux* kernel with changes in the Power Management (PM) driver and Performance Monitoring Unit (PMU) driver to support the coprocessor. The Linux* kernel and initial file system image are installed into the host file system as part of the software stack. After the

installation each coprocessor will need to be configured in accordance with the expected workload/application. This configuration is covered in Section 4.

The Linux* environment on the coprocessor utilizes BusyBox to provide a number of Linux* utilities. However, these utilities may have limited functionality when compared to similar tools provided with the typical Linux* distribution. For more information

regarding BusyBox, refer to http://www.busybox.net/.

2.3.2 Intel® MPSS middleware libraries

The compiler runtimes depend on the COI (Coprocessor Offload Infrastructure) library to offload executables and data for execution on the coprocessor, and use MYO (Mine, Yours, Ours shared memory infrastructure) to provide a virtual shared memory model

simplifying data sharing between processes on the host and coprocessor. Some functions in the Intel® Math Kernel Library (Intel® MKL) automatically offload work to the coprocessor using the COI library.

COI, MYO, and other software stack components rely on the SCIF (Symmetric Communication Interface) user mode API for PCIe communication services between

the host processor, coprocessors, and optionally installed InfiniBand* host channel adapters. SCIF delivers high bandwidth data transfers and sub-sec write latency to

memory shared across PCIe, while abstracting the PCIe communication details.

The COI, MYO, and SCIF libraries are also available to other applications. Section 2.4 lists additional documentation on these libraries.

2.3.3 Intel® MPSS drivers and services

The following drivers run on the host system enabling communication with the

coprocessor:

Mic.sys MicVeth.sys

http://www.busybox.net/



Drivers are responsible for initializing, booting, and managing the coprocessor. They also implement the host-side parts of the SCIF protocol, and Virtual Ethernet (Veth) - two communication interfaces between the host and coprocessor. The virtio block device (virtblk) uses the virtio data transfer mechanism to implement a block device on the coprocessor. The device saves data in a specified storage location on the host and can therefore persist across coprocessor reboots.

An MpssControlService runs on the host, controlling access to the coprocessor’s file

system.

The Intel® Omni-Path Architecture enables direct data transfers between the

coprocessor’s memory and installed Intel® Omni-Path Host Fabric Adapters. The ibscif (InfiniBand* over SCIF) driver emulates an InfiniBand* HCA to the higher levels of the OFED stack. This driver uses SCIF to provide high bandwidth, low latency communication between multiple coprocessors in a host platform, for example

between MPI ranks on separate coprocessors.

2.3.4 Tools and utilities

Intel® MPSS includes several system management tools and utilities (for detailed

instructions on utilizing these tools refer to Section 8):

micctrl is a utility for controlling (boot, shutdown, reset) each (or all) of the installed coprocessors. micctrl also offers options to simplify the configuration process.

Configuration tasks include controlling user access and adding coprocessors to a TCP/IP network. For detailed information execute:

User> micctrl -h

micinfo displays information about installed coprocessors, the host’s operating system

and the Intel® MPSS host driver. For detailed information execute:

User> micinfo -h

micfw is used for updating the coprocessor’s firmware and displaying the version of firmware loaded on the coprocessor or residing in a firmware image file. For detailed

information about micfw execute:

User> micfw -h

miccheck runs a suite of sanity checks on host systems containing coprocessors. The

win32com Python extension must be installed before running miccheck. For detailed information execute:

User> miccheck --help

micbios allows to change certain coprocessor’s BIOS settings, as well as to use the Firmware Update Protection feature.

User> micbios --help

libsystools is a C-library interface to the coprocessor for applications running on the

host. These are wrappers built on top of the MIC host driver interfaces, available in the form of WMI entries and scif calls. Various functions are available to query the

http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-architecture-fabric-overview.html



17

state of each coprocessor, and list or modify some of its parameters. Library functions are categorized in the following groups:

• Identify Available Coprocessors.

• Error Reporting.

• Device Access.

• Query Coprocessor State.

• General Device Information.

• PCI Configuration Information.

• Memory Information.

• Processor Information.

• Coprocessor Core Information.

• Version Information.

• LED Mode Information.

• Turbo State Information.

• SMC Configuration Information.

• Throttle State Information.

• Coprocessor OS Power Management Configuration.

• Thermal Information.

• Power Utilization Information.

• Memory Utilization Information.

• Power Limits.

For detailed information about libsystools, refer to the help page: C:\Program Files\Intel\MPSS\docs\libsystools\index.html.

2.4 Related documentation

Listed below are documents installed alongside the software stack in their default

locations.

COI documentation and tutorials:

The following COI documentation is installed during base Intel® MPSS installation:

C:\Program Files\Intel\MPSS\docs\coi\release_notes.txt

C:\Program Files\Intel\MPSS\docs\coi\MIC_COI_API_Reference_Manual_1_0.pdf



C:\Program Files\Intel\MPSS\sdk\include\intel-coi – header files containing full API description

C:\Program Files\Intel\MPSS\sdk\tutorials\coi\README_Windows_1.txt – instructions on how to get coprocessor side binaries and general building and use of COI tutorials.

MYO documentation and tutorials:

MYO tutorials and other documentation are installed in the C:\Program Files\Intel\MPSS\sdk\tutorials\myo directory. The README.txt file in this directory contains instructions on running the MYO tutorials.

SCIF documentation and tutorials:

The C:\Program Files\Intel\MPSS\sdk\tutorials\scif\README.txt contains instructions on running the SCIF tutorials.

§

Installation process overview


19

3 Installation process overview

This section describes how to install the coprocessor hardware and software and initialize the default configuration.

Read through this section before proceeding with installation, to ensure the required

components and facilities are available. To avoid setup issues follow these instructions in their presented order.

3.1 Hardware and software prerequisites

Host system hardware

Each coprocessor requires a 64-bit x16 PCIe slot with 75W power output capability. Refer to your motherboard’s manual to identify compatible slots.

Note: Installing more than 8 coprocessors in a single host platform is not supported.

Note: Host platforms with more than 256GB are not supported yet. This limitation will be removed in the upcoming release.

Coprocessor physical installation

If PCIe devices are installed into IO hubs of different CPU sockets, they will communicate across the Quick Path Interconnect (QPI). The bandwidth of this communication is typically lower than communication bandwidth between two devices on the same IO hub.

3.1.1 BIOS configuration

1. Update your host platform’s BIOS to the latest version and reset the settings to

their default values.

2. Enable Large Base Address Registers (BAR) support in the host platform BIOS.

BIOS and OS support for large (32+GB) Memory Mapped I/O Base Address Registers (MMIO BARs) above the 4GB address limit must be enabled.

3. Modify the following BIOS settings to ensure maximum and stable performance:

a. If available on your platform, set the Extended ATR setting to 0x1.

b. Enable Intel® Turbo Boost Technology.

4. Disable Virtualization Technology for Directed I/O (VT-d).



3.1.2 Supported host operating systems

Intel® MPSS 4.4.0 was validated against specific versions of Microsoft* Windows* as the host operating system. Table 1 lists supported host operating systems.

Table 1 Supported host operating systems

Supported Host OS Version

Microsoft* Windows* 8.1

Microsoft* Windows* 10

Microsoft* Windows* Server 2012 R2

Microsoft* Windows* Server 2016

3.1.3 Software prerequisites

The software stack requires the following software to be installed on the host system:

Microsoft* .NET Framework* 4.5 or higher Python* 2.7.5 x86-64 or higher (Python 3.x is not supported) Pywin32 build 220 or higher (https://sourceforge.net/projects/pywin32)

3.2 Installing Intel® MPSS

This section outlines how to install, upgrade, and uninstall the software stack.

3.2.1 Intel® MPSS installation

The software stack is distributed in a single zip archive.

1. Download and extract the mpss-<version>-windows.zip archive.

2. Double-click the mpss-<version>.exe file.

3. Follow the on screen instructions to complete the installation

Note: The default installation path is C:\Program Files\Intel\MPSS\. It can be changed during installation.

Note: If a Windows* security prompt appears, allow the installation of the driver and software to proceed.

3.2.1.1 Unattended installation

1. Open the Command Prompt and navigate to the directory where the mpss-<version>-windows.zip file was extracted. For example:

https://sourceforge.net/projects/pywin32



21

User> cd C:\Users\<username>\Downloads\mpss-<version>

2. Enter the command below.

User> "mpss-<version>.exe" /s /v /qn /V"/quiet /norestart"

Unattended installation may take several minutes to complete.

3.2.2 Upgrade instructions

Upgrading the Intel® MPSS can be achieved by following instructions in Section 3.2.1. Users may choose to manually uninstall the previous version or let the installer automatically search and remove previous release prior to installing the latest one.

Note: Upgrading the software stack may restore the default configuration. Back up your configuration files and system images before performing the upgrade.

3.2.3 Installing Windows* cross-sdk

The SDK for the coprocessor’s native compiler is included in the installation zip file

package. It also includes a squashfs repository image with supplementary software for the coprocessor. The SDK is required in order to compile and run applications for the coprocessor. Follow the steps below to install the SDK:

1. Extract the mpss-<version>-windows.zip file.

2. Install the software stack as in Section 3.2.1

3. Double click the mpss-essentials-<version>.exe file.

4. Follow the on screen instructions to complete the installation.

Installing the SDK is mandatory when using offload programming directives or cross compilers.

It is unnecessary to uninstall previous software versions. The installer will

automatically remove previous versions prior to installing the current one.

Note: The Windows SDK does not contain header files necessary for cross-compiling Linux kernel netfliter modules.

3.2.4 Uninstalling Intel® MPSS

To uninstall the software stack open the Control Panel, choose Programs and features and remove the “Intel(R) Xeon Phi(TM) coprocessor” application.

3.3 Updating coprocessor’s firmware

It is required to update coprocessor’s firmware after each installation or update of the

software stack. Current firmware is distributed with the software stack installation. The readme-windows.pdf file (distributed within package mpss-<version>-windows.zip) lists the versions of the included firmware.



Note: Running Intel® MPSS with incorrect firmware version is not supported and may lead to erratic behavior.

Follow the instructions below to update the coprocessor. Note, however, that these instructions will not work if the flash files (i.e. files with the .hddimg suffix) were moved or deleted from their installation directory.

1. Check the status of each coprocessor.

Admin> micctrl -s

If the status of every coprocessor is ready to boot, proceed to step 2; otherwise, reset the coprocessors.

Admin> micctrl -rw

2. Verify the version of firmware installed on the coprocessor:

Admin> micfw device-version

If the versions are the same as versions described in Section 2.2.3 of the readme-

windows.pdf file, the next steps might be skipped.

Note: When firmware installed is below to Intel® MPSS 4.3.2 version, step 3 has to be executed twice to complete the SMC firmware update.

3. Update the firmware of each coprocessor:

Admin> micfw update all

Or update only a specified coprocessor:

Admin> micfw update micN

Once the update process completes the state of coprocessors will be changed to

ready to boot.

Note: Do not execute any other applications or modify any coprocessor’s state while micfw is executing.

Note: micfw might fail to update coprocessor’s firmware if it hadn’t been consequently

updated with each subsequent version of the software stack.

4. Perform the cold host system reboot to apply all changes.

3.4 Initial coprocessor boot

3.4.1 Starting the coprocessor

After successful completion of the previous steps, the coprocessor(s) can be booted. Enter the following command:

Admin> micctrl --start



23

The call to micctrl exits when it determines the coprocessor either booted successfully

or failed to boot.

You can check the coprocessor’s status to verify that booting was successful. Run the following command:

Admin> micctrl –s

The command’s output should indicate that coprocessor is online.

Running the above commands also initiates the default per-coprocessor configuration.

Note: The default configuration boots the coprocessor when the host driver loads, causing it to boot during the host OS startup.

3.4.2 Initial configuration

Two files contain coprocessor configuration parameters:

C:\Program Files\Intel\MPSS\default.conf contains configuration parameters

parsed to every coprocessor in the system.

C:\Program Files\Intel\MPSS\micN.conf contains configuration parameters for a specific coprocessor.

The configuration files are created automatically when the micctrl command is run for the first time after the initial installation of the software stack. The micctrl command also creates configuration files if they were deleted from the host file system or after restoring the default configuration.

You can modify the default configuration if it does not meet the requirements of your

system (refer to Section 4 for more information).

Reboot the coprocessor to apply the new configuration:

Admin> micctrl --stop

Admin> micctrl --start

3.4.3 Specifying administrator’s SSH key

Access to the coprocessor is provided through the secure shell utilities. The initial login can only be authenticated with an SSH key pair. Section 5.4.2 contains instructions on generating SSH keys and accessing the coprocessor through SSH using PuTTy tools.

Specify the location of the administrator’s public SSH key in the default.conf

configuration file. The default value is:

RootPublicKey %INTEL_MPSS_HOME%\filesystem\ssh\id_rsa

%INTEL_MPSS_HOME% is an environment variable created by the software stack indicating its installation directory (the default installation path is C:\Program

Files\Intel\MPSS).



The key will be copied to every coprocessor’s file system during each boot. The

administrator can then use their private key to log in to the coprocessor using the username root.

3.5 Validating Intel® MPSS installation

Intel® MPSS provides utilities to perform basic tests to validate the installation.

Section 4.4 provides troubleshooting advice if problems are encountered during software stack installation and usage.

The micinfo tool provides information about the host and the coprocessor hardware

and software. It can be used, for example, to verify that your coprocessor is running

the correct firmware version. Note that certain information is only available when executing micinfo with root privileges:

Admin> micinfo

More information can be obtained by running the micinfo --help command.

The miccheck tool performs sanity checks on systems containing coprocessors.

Admin> miccheck

Obtain more information on the utility by running the miccheck --help command.

§

Coprocessor OS configuration


25

4 Coprocessor OS configuration

The coprocessor operating system requires a kernel and a file system image. These components are installed into the host’s file system as part of the software stack. The kernel command line is constructed based on a set of configuration files on the host and is passed to the coprocessor’s kernel during boot.

Note: Configuration parameters and procedures for the previous generation Intel® Xeon Phi™ coprocessor x100 are not supported in this version of the software stack.

4.1 Configuration files

Configuration is controlled by parameters in files located in %INTEL_MPSS_HOME%, which by default is C:\Program Files\Intel\MPSS\. The default.conf file contains configuration parameters shared by all coprocessors in the system. Additionally, each

coprocessor has an associated micN.conf configuration file, where N is the integer ID of that coprocessor (for example: mic0.conf, mic1.conf, etc.).

Note: If the same parameter is present in default.conf and any micN.conf file, the latter takes precedence in configuring the coprocessor.

Each configuration file contains a list of parameters and their arguments. Each parameter must be on a single line. Comments begin with the ‘#’ character and terminate at the end of the line. There are several configuration parameter categories:

1. Parameters that configure the coprocessor Linux* kernel.

2. Parameters that configure the root file system.

3. Parameters that configure the boot command line.

4. Parameters that configure the host and coprocessor virtual Ethernet connection.

4.1.1 Updating configuration files

Executing the micctrl --restoreDefaultConfig command after upgrading the software

stack updates all deprecated or changed configuration parameters.

Note: Running the above command may overwrite your configuration files and file system images.

4.2 Boot configuration

To boot a coprocessor, micctrl must:

Determine the kernel to boot.

Identify and/or build the file system image.

Build the kernel command line.



Parameters in the default.conf and micN.conf configuration files are evaluated for this

purpose.

The following sections describe parameters that are evaluated for this purpose.

4.2.1 Specifying the Linux* kernel

The KernelImage and KernelSymbols parameters in micN.conf files determine the

coprocessor OS boot image and its associated system address map file. Their default values are:

KernelImage %INTEL_MPSS_HOME%\filesystem\bzImage.bin

KernelSymbols %INTEL_MPSS_HOME%\filesystem\System.map

4.2.2 Coprocessor-side kernel command line parameters

Additional kernel command line parameters can be provided in the KernelExtraCommandLine configuration parameter in the default.conf configuration file. Refer to Appendix A.4.1 for more information.

The complete list of kernel parameters is available at

https://www.kernel.org/doc/Documentation/kernel-parameters.txt.

4.2.3 Coprocessor’s root file system

The RootFsImage parameter in micN.conf files specifies the coprocessor’s root file

system image. During initialization the software stack creates the %INTEL_MPSS_HOME%\filesystem\micN.ext4 file system image files for every

coprocessor. Each of those files has a default size of 1GB.

4.2.3.1 Coprocessor’s file system on diskless hosts

Coprocessors installed in hosts with network storage or with limited storage capabilities can be configured to use a single root file system image. Multiple coprocessors can use a single file system image in the read-only mode and use it to create a file system in their RAM. Note that RAM file system will not persist across coprocessor reboots.

To establish this configuration the RootFsImage parameter in the coprocessors’ micN.conf files should indicate the same file system image. Additionally, the mic_root_in_ram=1 option should be set in the KernelExtraCommandLine parameter in the default.conf (to apply to all coprocessors) or in the micN.conf file (to apply to a

specified coprocessor).

4.2.4 Automatic boot

The AutoBoot parameter controls whether coprocessor is booted automatically during the host system startup. By default, this parameter is present in all micN.conf files, and set to Enabled. Set the parameter value to Disabled in any micN.conf file to

change the default behavior and disable the automatic boot feature for that

https://www.kernel.org/doc/Documentation/kernel-parameters.txt



27

coprocessor. Remove the parameter from micN.conf files and add it to the default.conf file to enable or disable this feature for all coprocessors in the system. If the parameter is not present in any configuration file the behavior is as for AutoBoot Disabled.

4.2.4.1 Coprocessor shutdown

The following commands can be used to reset, reboot or shut down the coprocessor:

micctrl -r – resets the coprocessor to the ready to boot state;

micctrl -R – reboots the coprocessor to the online (booted) state;

micctrl -S - shuts down the coprocessor;

4.3 Basic administration tasks

This section discusses how to perform some common administration tasks, such as

adding and removing users and groups. Note that the initial configuration of the coprocessor OS only creates the root user account, and does not parse any user credentials to the coprocessor file system. This data can be provided manually in the same manner as in a standard Linux* distribution.

4.3.1 Adding groups and users to the coprocessor’s file system

Once logged in to the coprocessor, the administrator can use standard Linux*

commands to add groups and users to its file system.

[micN]# groupadd <group>

[micN]# adduser <user>

The <group> and <user> elements specify the group and username to be added.

Invoke the commands with the --help option for more information.

4.3.2 Removing users and groups from the coprocessor’s file

system

Removing a group or user from the coprocessor’s file system is accomplished through the standard Linux* commands: groupdel and userdel.

[micN]# groupdel <group>

[micN]# userdel <user>

The <group> and <user> elements specify the username and group to be deleted.

Note that the home directory of the user will still exists in /home. Using the –r option with the userdel command removes the user’s home directory. It can also be removed manually with the command below.

[micN]# rm –rf /home/<user>



4.3.3 Specifying the host SSH keys

Users can run the command below to transfer their public SSH key to the coprocessor. This operation allows logging in to the device using a private key. Refer to Section 5.4.2 for details on generating user SSH keys and using the PuTTY tool to establish ssh connection to coprocessors.

Note: This operation requires the user to have an account in the coprocessor’s file system.

[micN]$ echo PUBLICKEY >> ~/.ssh/authorized_keys

PUBLICKEY indicates the OpenSSH format public key. This command adds the

specified public key to the authorized_keys file in the coprocessor file system. This command must be executed from a terminal prompt logged in as the desired user.

4.4 Troubleshooting

This section describes using the Windows* Event Viewer, which may help

troubleshooting the following events:

The coprocessor times out during the boot process;

The coprocessor does not start;

The micctrl utility logs information related to events in the Windows* Event Viewer. To open the event viewer, click the Start button and type Event Viewer. In the left

column expand Applications and Services and double-click MPSS Log.

The following window will be displayed:

This event log should be consulted in the event that any observable failures occur while starting or stopping coprocessors. Failures include timeouts and errors running micctrl --start, micctrl--stop, micctrl -r, and micctrl -b.



29

The event log can be saved to a file from the command line with the command below.

Admin> wevtutil qe /f:text "MPSS log" > output.txt

It is also possible to print the software stack related information from Event Viewer on

the console with the micctrl tool.

Admin> micctrl -e

§

Network Configuration


5 Network Configuration

The coprocessor does not have hardware Ethernet capabilities. Instead, virtual Ethernet drivers emulate Ethernet devices to enable a standard TCP/UDP IP stack on the host and coprocessor.

Parameters in each coprocessor’s micN.conf configuration file are consulted to create a

virtual Ethernet interface for each coprocessor. Those parameters and their syntax are described in detail in Appendix A.5.

During the software stack installation a static pair network configuration is

established; see Section 5.3.1 for a full description.

5.1 MAC address assignment

The coprocessor has no pre-assigned MAC address because it lacks hardware network

interface. Therefore, a MAC address must be generated and assigned to each virtual network device.

During the coprocessor’s boot, micctrl generates MAC addresses for the network endpoints. Those addresses are based on serial numbers of the devices. This behavior can be changed by specifying the HostMacAddress and CardMacAddress configuration

parameters in the micN.conf files. Their syntax and supported values are described in Appendix A.5.3.

5.2 Host firewall configuration

If the coprocessor’s host system firewall is enabled, it may require configuration to

allow NFS mounting of host exports. NFS requires access to the following ports:

tcp/udp port 111 - RPC 4.0 portmapper

tcp/udp port 2049 - nfs server

5.3 Supported network configurations

The supported network configuration schemes allow coprocessors to operate in a wide

range of network environments.

The static pair topology creates a private subnetwork between the host and each coprocessor (Section 5.3.1).

The bridged topologies:

o Internal bridge allows coprocessors to communicate with the host and with each other (Section 5.3.2.1).

o External bridge additionally allows coprocessors to communicate with

external compute nodes (Section 5.3.2.2).



31

5.3.1 Static pair

Static Pair Configuration creates separate private network between the host and each coprocessor with IP addresses assigned to every network endpoint.

Static pair is the default network configuration. It is sufficient for Intel® C++ and

FORTRAN compiler pragma-based offload computation on a standalone workstation.

The static pair configuration creates a separate private network between the host and each coprocessor, and assigns an IP address to each of the network endpoints. Figure 4 depicts this configuration. A private network is configured between the host

and each coprocessor. Notice that mic0 and mic1 are on separate subnets.

Figure 4 Static pair configuration

5.3.1.1 Static pair configuration

To establish or modify this configuration edit the micN.conf file. The values of the HostNetworkConfig and CardNetworkConfig parameters should resemble the examples below:

HostNetworkConfig inet static address=172.31.1.254

netmask=255.255.255.0

CardNetworkConfig inet static address=172.31.1.1

netmask=255.255.255.0 gateway=172.31.1.254

The IP addresses of the host and each coprocessor must be on the same subnet.

A descriptor of each coprocessor endpoint should be added to the host’s

%WINDIR%\System32\drivers\etc\hosts file to associate IP addresses with each coprocessor’s hostname. For example:

172.31.1.1 mic0

172.31.2.1 mic1



Similarly a descriptor of the corresponding host endpoint should be added to each

coprocessor’s /etc/hosts file to associate the host’s endpoint IP address with the host’s hostnames. For example, mic0’s /etc/hosts might contain:

127.0.0.1 localhost.localdomain localhost

::1 localhost.localdomain localhost

172.31.1.254 host

172.31.1.1 mic0

Note that in this example /etc/hosts includes descriptors of both the host endpoint and the local endpoint.

Endpoints are brought up during the coprocessor’s boot. At this point the ipconfig command output executed on the host should be similar to:

Ethernet adapter mic0:

Connection-specific DNS Suffix . :

Link-local IPv6 Address . . . . . : fe80::b57a:a0e0:c5eb:5b24%29

IPv4 Address. . . . . . . . . . . : 172.31.1.254

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . :

5.3.2 Bridging

A network bridge is a way to connect two Ethernet segments or collision domains in a

protocol independent way. It is a Link Layer device which forwards traffic between networks based on MAC addresses and is therefore also referred to as a Layer 2 device.

The software stack directly supports two types of bridged networks described below.

5.3.2.1 Internal bridging

An internal bridge allows the connection of one or more coprocessors within a single

host system as a subnetwork. In this configuration, each coprocessor can communicate with the host and other coprocessors in the platform. Figure 5 shows an example internal bridge configuration.



33

Figure 5 Internal bridge configuration

Such network configuration could, for example, be used to support communication between the ranks of an MPI application that is distributed across coprocessors and host.

The internal bridge requires setting a new host bridge endpoint and connecting

coprocessor network adapters to it.

5.3.2.1.1 Internal bridge configuration

Open Network and Sharing Center and select Change adapter settings. A

window will show displaying available network adapters that should contain virtual network adapters for each coprocessor in the system (named mic0, mic1, etc.). Select adapters you wish to connect to the bridge and choose Bridge Connections from the context menu. A network bridge will be created automatically with default configuration and chosen adapters will be connected to it.

Right click the Network Bridge, which should be now visible in the window, and select Properties. The new window displays a list of network adapters connected to the bridge, allows to remove or add additional adapters and contains configuration options for the bridge.



Select Internet Protocol Version 4 (TCP/IPv4) and click Properties. Mark the

Use the following IP address option and provide IP address and subnet mask for the network bridge (e.g. 172.31.1.254, 255.255.255.0). Confirm your changes by clicking the OK button.

Set correct IP address for each coprocessor connected to the bridge by modifying the

CardNetworkConfig parameter in their respective configuration files, netmask value should be the same for all coprocessors. The examples below show configuration for coprocessors mic0 and mic1. Coprocessors will use their new configuration after a reboot.



35

mic0.conf:


netmask=255.255.255.0 gateway=172.31.1.254

mic1.conf:


netmask=255.255.255.0 gateway=172.31.1.254

Connecting coprocessors to the bridge causes the HostNetworkConfig parameter to be

ignored.

Executing the ipconfig command in the command prompt will display details on the

network configuration on the host. Once logged in on the coprocessor, use the ifconfig

command to view information about network configuration.

To change the MAC address of the network bridge open its properties, click Configure, choose the Advanced tab, select the Network Address property and provide the MAC address. By default, the network bridge obtains its MAC address from

one of the connected adapters.

5.3.2.2 External bridging

The external bridge configuration bridges one or more coprocessors to an external

network. This is a typical configuration required when coprocessors are deployed in a cluster to support remote communication between them and/or CPUs across different compute nodes.

Figure 6 depicts a cluster in which the coprocessors on each host node are bridged to

an external network. The IP addresses in such a configuration can be assigned statically or by a DHCP server on the network, but must generally be on the same subnet. In most cluster environments it is usually recommended to employ static IP

schemes, which allows to correlate nodes with their known IP addresses.

InfiniBand* based networking is not shown in this figure. InfiniBand* based networking will usually provide significantly higher bandwidth than the IP networking supported by the Intel® MPSS Virtual Ethernet driver. Many clusters use Ethernet*

networking for low bandwidth communication such as command and control, and use InfiniBand* networking for high bandwidth communication as application data transfer.



Figure 6 External bridge configuration

5.3.2.2.1 External bridge configuration

Note: Make sure that you have provided a large enough IP address pool to accommodate nodes of the externally bridged networks.

Add coprocessors’ virtual network adapters to a network bridge and connect the bridge to the external network.

a) Static IP address assignment

Follow the instructions in Section 5.3.2.1.1. Remember to reboot the coprocessors if during configuration you modified their configuration files.

b) IP address assignment by DHCP server



37

Open the properties window of the network bridge and in the Internet Protocol

Version 4 (TCP/IPv4) properties select Obtain an IP address automatically and Obtain DNS server address automatically.

To allow coprocessors to obtain their IP addresses from the DHCP server modify the CardNetworkConfig parameter in their configuration files. The parameter

should have the following value:

CardNetworkConfig inet dhcp

Reboot the coprocessors to apply the new configuration.

5.4 Interacting with the coprocessor

The following sections describe how to generate SSH keys, access the coprocessor and transfer files to its file system. The examples are based on the PuTTYgen key generation tool, PuTTY SSH client and winSCP SCP client.

5.4.1 External tools

Download the latest version of PuTTY and PuTTYgen from

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. WinSCP can be downloaded from http://winscp.net/. Install the tools in the same location as the external tools: C:\Program Files\Intel\MPSS\bin.

5.4.2 SSH access to the coprocessor

Steps in this section show how the system administrator can establish a key-based

SSH access to the coprocessor.

To allow non-root users to log in to the coprocessor create additional user accounts in

its file system and specify each user’s public key in their ~/.ssh/authorized_keys file.

Section 4.2.4.1 explains how to perform these tasks.

Note: By default, only key-based authentication can be used to log in to the coprocessor.

Any attempt to establish SSH connection using password will be rejected by the ssh daemon on the coprocessor. This behavior can be changed by modifying the

/etc/ssh/sshd_config file.

1. Open PuTTYgen (puttygen.exe). In the program window click Generate. After

completing the on-screen instructions the following screen is displayed.

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

http://winscp.net/



2. Click Save private key and save the private key in a secure location not readable

by other users. This key will be used to connect to the coprocessor. Optionally,

save the key with a passphrase, or click Yes to ignore the warning.

3. Select Save public key to save the public key in a file.

4. Set the value of the RootPublicKey parameter in the default.conf configuration file to the location of your public key. The key will be propagated to each coprocessor

during boot.

5. Open PuTTY.exe. In the Category box, expand Connection -> SSH and select

Auth. In the Private key file for authentication field, browse to the private key

saved in step 2.



39

6. In the Category box, select Session. In the Host Name (or IP address) field,

enter the coprocessor’s IP address (mic0 shown). You may save the configuration

under custom name by providing the name in Saved Sessions box and clicking

Save. This allows for easy reconnection to the coprocessor without specifying the

private key each time. Click Open, this will establish connection with the

coprocessor mic0.

7. When the session opens to the coprocessor enter the username root. You should

successfully log in without the need to provide password.



After completing these steps, a program such as WinSCP can be used to transfer files

to and from the coprocessor.

5.4.3 Transferring files to and from the coprocessor

This section describes using the WinSCP tool (version 5.7.6 shown) to transfer files to

and from the coprocessor. Instructions here assume that key-based authentication is

established.

1. Launch WinSCP and select File Protocol: SCP. To transfer files to coprocessor

mic0 type 172.31.1.1 in the Host Name field and your username as specified in

the coprocessor’s file system in the User Name field (the administrator should

enter root). Do not enter a password. Click Advanced… to display the Advanced

Site Settings dialog.

2. In the Advanced Site Settings dialog, select SSH -> Authentication. In the

Private key file field, enter the full path to the private key file generated above

(or click […] to browse to the file).



41

3. Select Environment -> SCP/Shell. In the Shell field, select /bin/bash. Click

OK to accept new configuration and return to the previous window.



4. Optionally, click Save to save the new site to the tree in the left pane. Click Login

to use the tool.

§

Managing the coprocessor’s file system


43

6 Managing the coprocessor’s file

system

The persistent files system (PFS) is the root file system of the Intel® Xeon Phi™

coprocessor x200 is maintained on the host with a virtual block device on the coprocessor. This device redirects read/write requests to and from the host over PCIe. This solution enables the user to manage the coprocessor’s file system much like the one in a standard Linux* OS. The mechanism is depicted in Figure 7

Figure 7 Persistent file system concept

6.1 Adding persistent files to the coprocessor’s file system

Files added to the coprocessor’s file system during normal use are stored persistently

by default. A user has several options to make changes to the persistent file system.

Supplementary software for the coprocessor is installed on the host with the

Windows* cross-sdk (mpss-essentials-<version>.exe).



6.1.1 Using the smart package manager

The smart package manager allows users to install additional software on the coprocessor. The squashfs repository image on the host (e.g. the repository installed with the Windows* cross-sdk) is passed to the RepoFsImage configuration parameter in each micN.conf file. The repository image is automatically detected and configured by the smart package manager. Available packages can be installed by running the

smart install <package> command, for example:

[micN]# smart install perf

You can find more information on the smart package manager at

https://labix.org/smart.

Refer to Appendix A.3.2 for more information on the RepoFsImage parameter.

6.1.2 Editing files interactively

Editing files interactively assumes that the user logs in to the coprocessor using SSH, and introduces desired changes directly in the coprocessor’s OS in the same manner as in a standard Linux* distribution.

6.1.3 Transferring files over SSH

Copying a file over SSH assumes that the user is able to log in to the coprocessor

using SSH.

Copy a file from the host to the coprocessor with WinSCP, pscp, or another similar

utility. Refer to Section 5.4.3 for instructions on transferring files using WinSCP.

6.1.4 Installing RPM packages (root only)

Installing an RPM package assumes that the user has access to the root account in the coprocessor’s file system and is able to log in using SSH.

To install a package, copy the RPM file(s) to the coprocessor’s file system (over

SSH/SCP) and issue the following command:

[micN]# rpm –ihv <package_name>

6.2 Virtual block devices

Users can specify additional block devices with supplementary data or software for the coprocessor. The BlockDevice configuration parameter allows to specify up to 6

additional block devices for each coprocessor.

https://labix.org/smart



45

Each virtual block device must be backed by a supported1 image file on the host and

cannot be shared by multiple coprocessors. Each specified block device will be identified by the coprocessor as a /dev/vd[a-z] device.

Example below shows how to specify a read/write block device named storage1 backed by a file at <image file location>.

BlockDevice name=storage1 mode=rw path=<image file location>

Once the block device is specified use the commands below create a mount point and

attach the device to the coprocessor’s file system.

[micN]$ mkdir –p <mount point>

[micN]$ mount /dev/vd[a-z] <mount point>

The software stack creates symbolic links to each device at /dev/mpss_mapper<block

name>, allowing users to refer to block devices by their designated names. The commands below show how to mount a block device named storage1.

[micN]$ mkdir –p <mount point>

[micN]$ mount /dev/mpss_mapper/storage1 <mount point>

Add the virtual block device to the /etc/fstab file in the coprocessor’s file system to mount it automatically upon every boot.

Refer to Appendix A.3.3 for more information on the BlockDevice parameter.

6.3 Host file system share

This section shows how to set up, configure and mount an NFS Share on Windows*

Server 2012 R2 and Windows* Server 2016. Section 6.3.1.1 through 6.3.1.3 show how to use the Server Manager user interface to set up an NFS share. For an alternative method using PowerShell* cmdlets, skip to Section 6.3.2.

Note: Feature described in this section is not available on Windows* 8.1 and Windows* 10

1 Supported image files include, among others: cpio, ext2, ext3, ext4, hddimg, iso, squashfs, tar, vdi and vmdk



6.3.1 Using the server manager

6.3.1.1 Adding the NFS server

1. Start Server Manager. On the dashboard, in the Manage menu select Add

Roles and Features.

2. In the Add Roles and Features Wizard, click Installation type in the left

column. Select Role-based or feature-based installation.



47

3. Click Server Selection and Select a server from the server pool. Select the

server.

4. Click Server Roles. Under Roles, expand File and Storage Services, then

expand File and iSCSI Services. Select Server for NFS.



5. A confirmation dialog box will appear. Click the Add Features button.

6. Returning to the Add Roles and Features Wizard, select Confirmation in the

left column. Click the Install button.



49

7. In the Results pane, confirm that the installation was successful.

6.3.1.2 Provisioning a folder for the NFS share

1. In Server Manager, click File and Storage Services in the left column of the

Dashboard.

2. Select the desired server from the list. Click Shares in the left column.



3. Click To create a file share, start the New Share Wizard.

4. In the New Share Wizard, click Select Profile and select NFS Share - Quick.

5. Click Share Location and select the desired server from the list. Specify a path

for the share.



51

6. Click Share Name and specify a name for the share (click OK if prompted to

create the share location directory).

7. Click Authentication. Select No server authentication, Enable unmapped

user access, select Allow unmapped user access by UID/GID.



6.3.1.3 Specifying the NFS share permissions

1. Click Share Permissions and click Add. In the Add Permissions dialog box,

change the Share Permissions field for the All Machines group to No Access.

This prevents unspecified hosts from accessing the NFS share. Click Add.

2. Return to the New Share Wizard and click Add. Now, select Host in the Add

Permissions dialog box. Specify a host and set its Share Permissions field to

Read/Write. Select Allow root access. Click Add.



53

3. Return to the New Share Wizard. Review the Share Permissions settings.

Verify the Host has Read/Write permissions and Root Access is Allowed.

4. In the left column, click Permissions then click Confirmation. Confirm your

settings and click Create.



5. Confirm your results and close New Share Wizard and Server Manager.

6. Proceed to Section 6.3.2.1.

6.3.2 Using PowerShell* cmdlets to create an NFS share

(optional)

The tasks in previous sections can be accomplished using the built-in PowerShell* NFS

cmdlets in Windows* Server 2012 R2 and Windows* Server 2016, as shown below:

Note: The backslash \ character at the end of a line indicates continuation of a command to the next line. It is not part of the command itself.

1. Add the NFS server:

User> Add-WindowsFeature FS-NFS-Service

2. Provision the directory to be shared:

User> New-Item C:\MY_NFS_TEST -type directory

User> New-NfsShare -Name nfs-tests \

-Path C:\MY_NFS_TEST -EnableUnmappedAccess $True \

-Permission no-access -Authentication sys

3. Grant read/write and root access permissions for Host 172.31.1.1:

User> Grant-NfsSharePermission –Name nfs-tests \

-Permission readwrite -ClientName 172.31.1.1 \

-ClientType Host -AllowRootAccess $True

4. Confirm share permission settings:

User> Get-NfsSharePermission –Name nfs-tests

5. Proceed to Section 6.3.2.1.



55

6.3.2.1 Mounting the NFS share

Note: Refer to Section 5.4.1 for information on downloading and installing PuTTY tools.

1. Change to the directory where PuTTY tools are installed.

User> cd <PATH_TO_PUTTY_TOOLS>

2. Log in to the coprocessor as root; example below shows logging in to coprocessor

mic0.

User> putty.exe -ssh -i <PATH_TO_PRIVATE_KEY> [email protected]

<PATH_TO_PRIVATE_KEY> indicates the location and name of the private key file

associated with this coprocessor.

Note: If prompted for a password, see Section 5.4.2 for instructions on setting up key-based SSH authentication

3. While logged in to the coprocessor, add a directory with the same name as the

NFS share:

[micN]# mkdir /tmp/nfs-tests

4. Mount the NFS share in the coprocessor’s file system. The nolock option is

required in this step:

[micN]# mount -t nfs -o nolock \

172.31.1.254:/nfs-tests /tmp/nfs-tests

5. Navigate to the directory that you created on the coprocessor. Verify the contents

of the NFS share are visible.

[micN]# cd /tmp/nfs-tests

6.4 CIFS share on Windows*

This section provides instructions on how to set up and provision a CIFS (Common Internet File System) share between the Windows* host and coprocessor.

6.4.1 Configuring a CIFS share

To mount a CIFS share, you must grant share permissions to Everyone and turn off password protected sharing.

1. Open the Network and Sharing Center, click Change advanced sharing settings, expand All Networks and enable the Turn off password protected sharing option.

2. Choose a folder you wish to share and grant appropriate permissions for that

folder to Everyone.



a. Open properties of the folder, select the Security tab and click Edit… to

change the folder permissions.

b. Click Add…, type Everyone in the Enter the object names to select field and click Ok.



57

c. Grant appropriate permissions to the Everyone group, which should now be

visible in the Groups or user names filed.

3. Choose the Sharing tab in the properties window of the folder you wish to share and click Share. Add Everyone to the share group, assign appropriate permissions and select Share

4. Click Advanced Sharing in the Sharing tab and select Share this folder.



5. Navigate to Permissions and assign appropriate permissions to Everyone.

Note: You have to enable sharing permissions in both Share and Advanced Sharing. The current mount utility on the coprocessor does not support nested host share paths.

6.4.1.1 Mounting a CIFS share

The root user can manually mount and umount password protected shares on the

coprocessor.

To mount a CIFS share, issue the following command on the coprocessor.

[micN]# echo “<SHAREPATH> <MOUNTPOINT> cifs \

guest,file_mode=0777,dir_mode=0777 0 0” >> /etc/fstab

Use the command below to mount a CIFS share for a specified user or group, for

example Everyone.

[micN]# echo “<SHAREPATH> <MOUNTPOINT> cifs username=everyone,\

password=,file_mode=0777,dir_mode=0777 0 0” >> /etc/fstab

<SHAREPATH> is the network addressable CIFS share. As an example, assuming that

mic0’s host network adapter has the static IP address 172.31.1.254, the share path

would be //172.31.1.254/<path to share>. The <MOUNTPOINT> is the directory in

the coprocessor’s file system to mount the CIFS share to. For example, the mount

point could be /mount/host_share. The directory used for the mount point must exist

before mounting the CIFS share. This command adds an entry to the coprocessor’s

fstab.



59

6.4.2 Mounting a temporary CIFS share

Users can issue the following command to temporarily mount a share. This operation can be used for verifying the share is functional and users have appropriate permissions to open it.

[micN]$ mount –t cifs <SHAREPATH> <MOUNTPOINT> -o guest

6.4.3 Troubleshooting

When accessing the share files on the coprocessor, you could encounter cannot

allocate memory error. This error can be avoided by maximizing the throughput for file sharing and/or network applications in the registry.

Set the value of the following registry keay to 2 or 3 (for large file transfers).

HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters\size

Optionally, set the value of the key below to 1.

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory

Management\LargeSystemCache

After introducing the changes, reboot your host or restart the Server service in

services.msc.

References:

http://support.microsoft.com/kb/232271

http://msdn.microsoft.com/en-us/library/ee377058(v=bts.10).aspx

§

http://support.microsoft.com/kb/232271

http://msdn.microsoft.com/en-us/library/ee377058(v=bts.10).aspx

Intel® MPSS component configuration and tuning


7 Intel® MPSS component

configuration and tuning

This appendix contains information on configuring particular elements of the

coprocessor operating system, debugging information, and COI (Coprocessor Offload Infrastructure) configuration.

7.1 Coprocessor operating system configuration

7.1.1 Disabling and enabling the memory control group

(cgroup)

In this release the memory control group is disabled by default, but it can be enabled

in the default.conf file. Enabling the memory cgroup decreases the amount of memory

available to applications on the coprocessor by about 120MB.

Note: The following settings can only be modified by a user with administrator privileges.

To enable the memory cgroup, remove the cgroup_disable=memory line from the <CommandLine> element in C:\Program Files\Intel\MPSS\default.conf file, and restart

the coprocessor.

<CommandLine> root=/dev/vda windows cgroup_disable=memory idle=halt

earlyprintk=dbgp,keep </CommandLine>

7.2 Enabling Windows* mic gdb debugging for offload processes

To support the use of the Windows* MIC GDB debugger through Microsoft* Visual

Studio*, environment variables must be set to allow the debugger access the files required to attach to the coprocessor-side offload processes. To do this, execute the following steps:

1. Close Microsoft* Visual Studio*.

2. Go to Control Panel->System->Advanced system settings.

3. In the Advanced tab, click Environment Variables…

4. Click New. Set Variable name to AMPLXE_COI_DEBUG_SUPPORT and Variable

value to TRUE.

5. Click New. Set Variable name to MYO_WATCHDOG_MONITOR and Variable value to -1.

6. Reopen Microsoft* Visual Studio*, allowing it to pick the new environment.



61

Upon execution of steps 1-6, the necessary debug files will be created for the

Windows* MIC GDB debugger to find and attach to the coprocessor-side offload processes for debugging issues in offload programs.

7.3 Enabling Windows* mic debugging for myo applications

The MYO WATCHDOG MONITOR is a feature of the MYO runtime library, which monitors whether the host-side and coprocessor-side are still operating and have not

suffered a fatal error. If a fatal error occurs on one peer (host or coprocessor), the other one will abort with an error message. The MYO WATCHDOG MONITOR performs this task by verifying messages are sent and received at a minimum rate (default 1 hertz).

Note: When a MYO application is stopped at a breakpoint in a debugger, the MYO

WATCHDOG MONITOR assumes the host or coprocessor side of the MYO application has suffered a fatal error and aborts. This will prevent debugging of the MYO application.

To debug MYO applications in this release, the MYO WATCHDOG MONITOR must be

turned off prior to starting the MYO application on the host. To do so, set the environment variable: MYO_WATCHDOG_MONITOR to -1.

Note: There is no support for attaching to a MYO application where the MYO WATCHDOG MONITOR is operating. This will be resolved in a future release of the MYO runtime.

7.4 COI configuration

7.4.1 Limiting maximum number of COI processes

The coi_daemon on the coprocessor spawns separate processes for each host process that performs offloading to the coprocessor. The maximum number of processes is

limited by --max-connections parameter.

7.4.2 COI offload user options

The coi_daemon on the coprocessor spawns processes on behalf of client processes on the host processor. The coi_daemon has several options for assigning ownership of these COI processes.

7.4.3 Micuser ownership

When operating in micuser mode, each COI process spawned by the coi_daemon is

owned by user micuser.

By default, micuser mode is configured in the coprocessor’s /etc/init.d/coi file:

_Dynamic User Ownership



When operating in _Dynamic mode, each COI process spawned by the coi_daemon is owned by a new, unique user created by the coi_daemon. Files and directories created by such process cannot be accessed by other COI processes, effectively isolating all COI Processes from each other for better security.

§

Intel® MPSS tools


63

8 Intel® MPSS tools

The software stack contains a number of tools and utilities for system administrators. These tools aid in configuration tasks, provide information about the host system and coprocessor, verify the installation, can be used to modify certain configuration settings and allow executing remote applications on coprocessors.

8.1 micctrl

The micctrl utility is a multi-purpose toolbox for the system administrator. It provides these categories of functionality.

Coprocessor state control – boot, shutdown and reset control.

Configuration file initialization and propagation of values.

First argument of each micctrl command must specify the action to perform, and can be followed by a number of additional sub-options. Each micctrl command (except micctrl -l) may be followed by a space-separated list of coprocessors, which is shown in the syntax statements as [list of coprocessors] and specifies the devices to control.

For example, the list may be mic1 mic3, if these are the coprocessors to control. If no coprocessors are specified, the command will be executed for all coprocessors in the system.

Note: Administrator privileges are required to run micctrl commands (except micctrl --status, micctrl --wait, micctrl --help and micctrl --version).

8.1.1 Checking the coprocessor’s state

Command syntax:

User> micctrl (-s|--status) [-v|--verbose] \

[list of coprocessors]

Displays the current status of all or selected coprocessors in the system.

The --verbose option causes the command to display current postcode.

Each coprocessor can be in one of the following states:

shutdown – the coprocessor is shut down; power management level S5

ready to boot – the coprocessor is ready to receive a boot command; power

management level S0

online – the coprocessor OS is booted and ready to work; power management

level S0

shutting down – transition to the shutdown state.

waking up – transition to the ready to boot state.

Intel® MPSS tools


booting – transition to the online state.

resetting – transition to the ready to boot state.

error – the coprocessor encountered an error and needs to be restarted by issuing

the micctrl -r command.

8.1.2 Booting the coprocessor

Command syntax:

Admin> micctrl (-b|--boot) [-w|--wait] [list of coprocessors]

The driver boots all or selected coprocessors. This command requires the

coprocessors’ state to be either ready to boot or shutdown. The --wait option waits for the state of coprocessors to change to online.

8.1.3 Starting the coprocessor

Command syntax:

Admin> micctrl --start [list of coprocessors]

The driver boots all or selected coprocessors and waits until their state is changed to

online. This command requires the coprocessors’ state to be either ready to boot or shutdown and results in their state changed to online. It is equivalent to running the micctrl -b -w command.

8.1.4 Shutting down the coprocessor

Command syntax:

Admin> micctrl (-S|--shutdown) [-w|--wait] [-f|--force] \


The driver gracefully shuts down all or selected coprocessors and changes their state to shutdown. This command requires coprocessors to be either in the online or in the ready to boot state. The --wait option causes the command to wait until the state of coprocessors is changed to shutdown. The --force option can be used to force

shutdown regardless of the current state of coprocessors.

Note: Using the --force option may corrupt the coprocessor’s file system.

8.1.5 Stopping the coprocessor

Command syntax:

Admin> micctrl --stop [list of coprocessors]

The driver gracefully shuts down all or selected coprocessors and waits until their state changes to shutdown. This command requires coprocessors to be either in the

Intel® MPSS tools


65

online or in the ready to boot state. This command is equivalent to running micctrl -S -w.

8.1.6 Resetting the coprocessor

Command syntax:

Admin> micctrl (-r|--reset) [-w|--wait] [-f|--force] \


The driver resets all or selected coprocessors that are in a stable state (online, ready

to boot, shutdown or error) and changes their state to ready to boot. Unlike the --shutdown command, this command does not wait for the coprocessor OS to shut down gracefully. The --wait options causes the command to wait until coprocessors

reach the ready to boot state. The --force option resets coprocessors regardless of

their current state.

Note: Using the --force option may corrupt the coprocessor’s file system.

8.1.7 Rebooting the coprocessor

Command syntax:

Admin> micctrl (-R|--reboot) [-w|--wait] \


The driver gracefully shuts down all or selected coprocessors and boots them. This command is equivalent to running micctrl -Sw followed by micctrl -b. The --wait option causes the command to wait until the state of coprocessors changes to online.

8.1.8 Waiting for the coprocessor state change

Command syntax:

User> micctrl (-w|--wait) [list of coprocessors]

The --wait option waits for coprocessors to be in a stable state (online, shutdown,

ready to boot, or error) or returns an error. It is intended for use in shell scripts and

for users to verify whether the startup, shutdown, or reset procedure was completed

successfully. It has a built-in timeout value of 120 seconds.

8.1.9 Restoring the default PFS image

Command syntax:

Admin> micctrl --restoreDefaultImage [list of coprocessors]

This command deletes the file system image files of all or selected coprocessors and

creates new ones with default contents.

Intel® MPSS tools


Note: Before creating new PFS image files, this command deletes all PFS image files and any content that was added to them after the installation of the software stack.

8.1.10 Restoring the coprocessor default configuration

Command syntax:

Admin> micctrl --restoreDefaultConfig [list of coprocessors]

This command deletes configuration files of all or selected coprocessors and creates

new ones with default values.

8.1.11 Downloading the coprocessor kernel log

Command syntax:

Admin> micctrl (-l|--log) [list of coprocessors]

The driver downloads all or selected coprocessors’ kernel log information if it is available.

8.1.12 Displaying the MPSS log

Command syntax:

Admin> micctrl (-e|--eventLog)

The driver downloads and displays Intel® MPSS-related information from the Event

Viewer.

You can also access information in the Event Viewer manually by following instructions

in Section 4.4.

8.2 micinfo

The micinfo tool displays information about each coprocessor installed in the system.

The tools displays firmware-, sensors- and driver-related information. Some of the data can be retrieved even if the coprocessor is not in the online state.

Note: Administrator privileges are required to view all information provided by micinfo.

8.2.1 Options

--help

Displays the help documentation for the tool.

Intel® MPSS tools


67

--version

Displays the version of the tool.

--verbose

Displays extra information about internal operations or messages.

-d, --device=<device-list>

Specifies coprocessors for which data will be retrieved. This option accepts

coprocessors’ names in form of a comma-separated list, range or a combination of both. For example: mic0-mic3, mic1,mic3-mic5, mic2. If no coprocessors are specified, the tool lists information for all coprocessors in the system.

-g, --group=<group-list>

Allows to narrow down the output of the tool to a specified subject called group and determined by the <group-list> argument. This option accepts a single name of a group or a comma-separated list of groups’ names. Valid groups are listed below.

system: shows information about the host.

versions: shows each coprocessor’s firmware version and serial number.

board: shows information about the coprocessor board.

core: shows number of cores, their voltage and frequency.

thermal: shows information about the cooling fan and thermal related data.

memory: shows information about each coprocessor’s memory.

8.2.2 Usage

Admin> micinfo [OPTIONS] [(-g |--group=)<groups>]

[(-d |--device=)<device-list>]

1. Executing micinfo with no options will display all information for all of the groups and for all coprocessors in the system.

User> micinfo

micinfo Utility Log

Created On Wed Jul 6 05:51:33 2016

System Info:

<output truncated>

Version:

<output truncated>

Board:

<output truncated>

PCIe Max read req size : Insufficient Privileges

<output truncated>

Core:

Intel® MPSS tools


<output truncated>

Thermal:

<output truncated>

Die Temp : Not Available

<output truncated>

Memory:

<output truncated>

Insufficient Privileges is displayed when micinfo is not run with administrator

privileges.

Not Available is displayed if a value could not be retrieved. For example, a hardware sensor does not report its value or the coprocessor is not in the online state.

2. In the example below micinfo displays information about version and core groups. The output was specified for coprocessor mic0.

User> micinfo --group="version,core" -–device=mic0

micinfo Utility Log

Created On Wed Jul 6 05:50:35 2016

Device No: 0, Device Name: mic0 [x200]

Version:

SMC Firmware Version : 121.31.10446

Coprocessor OS Version : 4.9.13-mpss_4.4.0.4483 GNU/Linux

Device Serial Number : 0123456789AB

BIOS Version : GVPRCRB8.86B.0014.D18.1703092150

BIOS Build date : 03/09/2017

ME Version : 3.2.2.8

Core:

Total No of Active Cores : 60

Threads per Core : 4

Voltage : 900.00 mV

Frequency : 1.20 GHz

8.3 miccheck

The miccheck tool executes a suite of diagnostics tests that verify the configuration and current status of coprocessors and the software stack.

The default tests include:

Checking number of coprocessors the OS detects in the system.

Checking whether drivers are loaded.

Checking number of coprocessors the driver detects in the system.

Checking the POST code returned by each coprocessor.

Checking whether the systools daemon is running on the coprocessor.

Optional tests include:

Intel® MPSS tools


69

Verifying whether each coprocessor’s firmware version is the same as provided

in the software stack release package.

Verifying whether the coprocessor can be pinged.

8.3.1 Options

--help


--version


--verbose

Displays extra information about internal operations or messages.


Specifies coprocessors for which the tests will be executed. This option accepts


-f, --firmware

Executes the firmware diagnostic test, which verifies whether each coprocessor’s firmware version is the same as provided in the software stack release package.

-p, --ping

Executes the ping diagnostic test which verifies whether each coprocessor can be pinged.

8.3.2 Examples

1. Running the default test cases.

Admin> miccheck

Executing default tests for host

Test 0: Check number of devices the OS sees in the system

... pass

Test 1: Check required drivers are loaded ... pass

Test 2: Check number of devices driver sees in the system

... pass

Executing default tests for device: mic0

Test 3 (mic0): Check device state and POST code ... pass

Test 4 (mic0): Check systoolsd is running in device ... pass

Status: OK

2. Running the default and ping tests on coprocessor mic0.

Intel® MPSS tools


Admin> miccheck --ping --device=mic0

Executing default tests for host

Test 0: Check number of devices the OS sees in the system

... pass

Test 1: Check required drivers are loaded ... pass

Test 2: Check number of devices driver sees in the system

... pass

Executing default tests for device: mic0

Test 3 (mic0): Check device state and POST code ... pass

Test 4 (mic0): Check systoolsd is running in device ... pass

Executing optional tests for device: mic0

Test 5 (mic0): Check device can be pinged over its network

interface ... pass

micchmicbi

Status: OK

8.4 micsmc-cli

The micsmc-cli tool can be used to monitor the coprocessor’s status and settings, and

also to introduce changes to its configuration.

8.4.1 Options

--help


--version


--verbose

Shows extra information about internal operations or messages.




8.4.2 Usage

Admin> micsmc-cli (disable|enable) [options] <settings-list>

<device-list>

disable <settings-list> <device-list>

Intel® MPSS tools


71

Disables one or more specified configuration settings for the selected coprocessors. If

no coprocessors are specified the command will apply to all coprocessors in the system.

enable <settings-list> <device-list>

Enables one or more configuration settings for the selected coprocessors. If no

coprocessors are specified the command will apply to all coprocessors in the system.

<device-list>

This element is a space-separated list of coprocessors’ names, ranges of names or a single name. At least one coprocessor must be specified.

<settings-list>

This element accepts a single setting or a comma-separated list of settings. The

available settings are 'ecc', 'led', 'turbo' and 'all'; the special word 'all' can be used to replace all other settings. At least one setting must be specified.

User> micsmc-cli show-settings [options] <settings-list>


show-settings <settings-list> [(-d |--device=)<device-list>]

Displays the current values of a specified setting for the selected coprocessors. If no

coprocessors are specified the command will apply to all coprocessors in the system.

User> micsmc-cli show-data [show-data options] [options]


show-data [show-data options] [(-d |--device=)<device-list>]

Displays general information based on option switches for each selected coprocessor.

If no coprocessors are specified the command will apply to all coprocessors in the system. This command accepts the options listed below, which will filter the output. If no options are specified the command will display all the information.

o -a, --all

Replaces --info, --cores, --mem, --freq, and --temp.

o -c, --cores

Displays dynamic (instant) core information.

o -f, --freq

Display dynamic (instant) frequency and power usage information.

o -m, --mem

Displays dynamic (instant) memory usage information.

Note: The displayed total memory also includes the memory cached and buffered by the

kernel, thus the sum of the free and used memory will not add up to displayed total. See the documentation of /proc/meminfo for more information.

Intel® MPSS tools


o -t, --temp

Displays dynamic (instant) temperature information.

o -i, --info

Displays static coprocessor information.

o --offline

Displays each offline coprocessor in a comma-separated list.

o --online

Displays each online coprocessor in a comma-separated list.

8.4.3 Examples

1. Enabling the led setting for coprocessor mic0.

Admin> micsmc-cli enable led mic0

Enable Setting Results:

mic0:

led : Successful

2. Disabling the led setting for coprocessor mic0.

Admin> micsmc-cli disable led mic0

Disable Setting Results:

mic0:

led : Successful

3. Displaying information on the led setting for all coprocessors in the system.

User> micsmc-cli show-settings led

Display Current Settings:

mic0:

led : disabled

mic1:

led : disabled

4. Displaying the frequency and power usage information for all coprocessors.

User> micsmc-cli show-data --freq

mic0:

Observed Frequency and Power Usage:

Core Frequency : 1.3000 GHz

PCIe : 24.0 Watts

2x3 : 36.0 Watts

2x4 : 19.0 Watts

Average 0 : 85.0 Watts

Current : 79.0 Watts

Maximum : 157.0 Watts

VCCP : 5.0 Watts

VCCU : 4.0 Watts

VCCCLR : 14.0 Watts

Intel® MPSS tools


73

VCCMLB : 17.0 Watts

VCCMP : 3.0 Watts

NTB1 : 3.0 Watts

mic1:

Observed Frequency and Power Usage:

<output truncated>

8.5 micfw

The micfw tool is used to update the coprocessor’s firmware. It can also be used to

retrieve the firmware version from a coprocessor or an image file.

8.5.1 Options

--help


--version


--verbose





--interleave

Shows the progress of the update in an output suitable for logs.

8.5.2 Usage

Admin> micfw (update [--file <image-file>] [--verbose]|

device-version|file-version <image-file>)


update [--file <image-file>] [--verbose] [(-d |--device=)<device-list>]

Attempts to update the firmware of each specified coprocessor. If the path to an

image file is not provided, the first compatible image in the default flash directory (OS dependent) will be used.

device-version [(-d |--device=)<device-list>]

Retrieves the firmware version of each specified coprocessor. If the <device-list> element is omitted then information regarding all coprocessors is displayed.

Intel® MPSS tools


file-version <image-file>

Retrieves the firmware version of the specified file.

8.5.3 Examples

1. Retrieving firmware version of all coprocessors in the system.

User> micfw device-version

mic0:



SMC Version : 121.31.10446

NTB EEPROM Version : 0204000D

2. Retrieving firmware version from a specified file.

User> micfw file-version <image-file>

GVPRCRB8.*.hddimg:

BIOS : GVPRCRB8.86B.0014.D18.1703092150

ME : 3.2.2.8

SMC Fab A : 121.31.10447

SMC Fab B : 121.31.10446

3. Updating firmware of two coprocessors using a specified firmware capsule.

Admin> micfw update mic0,mic1 --file GVPRCRB8.9899.hddimg

The update process may take several minutes, please be

patient:

Flashing process with file:

BIOS : GVPRCRB8.86B.0014.D18.1703092150

ME : 3.2.2.8

SMC Fab A : 121.31.10447

SMC Fab B : 121.31.10446

mic0: All firmware updates for this coprocessor completed

successfully.


successfully.

Summary:

mic0 : Successful

mic1 : Successful

Waiting for card(s) state(s) to be 'Ready'...

*** NOTE: You must reboot the host for the KNL changes to take

effect!

Note: The cold host system reboot is required to apply all changes. Do not execute any

other applications or modify any coprocessor’s state while micfw is executing.

Intel® MPSS tools


75

8.6 micflash

The micflash tool allows to update the coprocessor’s firmware through the micfw tool.

8.6.1 Options

-help


-v


-device



- log

This option is deprecated.

-noreboot


-nodowngrade


-smcbootloader


8.6.2 Usage

Admin> micflash [-h] [-getversion [File]][-device <dev>][-v]|

[-update [File]][-device <dev>][-v]

-getversion [-d|-device <device-name>]

Retrieves the firmware version of each specified coprocessor. If the <device-name> element is omitted then information regarding all coprocessors is displayed.

-update [file][-d|device <device-name>]

Attempts to update the firmware of each specified coprocessor. If the path to an image file is not provided, the first compatible image in default flash directory (OS dependent) will be used.

Note: Administrator privileges are required to flash the coprocessor’s firmware.

Intel® MPSS tools


8.6.3 Examples

Retrieving firmware version of all coprocessors in the system.

Admin> micflash -getversion

mic0:



SMC Version : 121.31.10446

NTB EEPROM Version : 0204000D

Retrieving firmware version in a specified file.

Admin> micflash -getversion GVPRCRB8.hddimg

GVPRCRB8.*.hddimg:

BIOS : GVPRCRB8.86B.0014.D18.1703092150

ME : 3.2.2.8

SMC Fab A : 121.31.10447

SMC Fab B : 121.31.10446

Updating firmware of two coprocessors using a specified firmware capsule.

Admin> micflash –update -device mic0,mic1 GVPRCRB8.9899.hddimg

The update process may take several minutes, please be

patient:

Flashing process with file:

BIOS : GVPRCRB8.86B.0014.D18.1703092150

ME : 3.2.2.8

SMC Fab A : 121.31.10447

SMC Fab B : 121.31.10446


successfully.


successfully.

Summary:

mic0 : Successful

mic1 : Successful

Waiting for card(s) state(s) to be 'Ready'...

*** NOTE: You must reboot the host for the KNL changes to take

effect!

Note: The host system reboot is required to apply all changes. Do not execute any other applications or modify any coprocessor’s state while micflash is executing.

Intel® MPSS tools


77

8.7 micbios

The micbios tool allows to change certain coprocessor’s BIOS settings, as well as to

use the Firmware Update Protection feature without the explicit knowledge of the syscfg tool. The coprocessor needs to be rebooted to apply new configuration.

Micbios accepts the following commands:

getinfo – read the BIOS configuration value.

set-pass – set up a BIOS password.

set-cluster-mode – modify the memory cluster mode in the coprocessor.

set-fwlock – enable or disable the firmware update lock feature.

set-ecc – enable or disable the error-correcting code.

set-apei-supp – enable or disable the advanced configuration and power interface. This option also enables to see the Errinject and FFM logging BIOS options.

set-apie-einj – enable or disable error injection.

set-apei-einjtable – enable or disable error injection tables.

set-apei-ffm – enable or disable FFM.

Note: Administrator privileges are required to view all information provided by micbios.

8.7.1 Options

--help

Shows the help documentation for the tool.

--version

Shows the version of the tool.

8.7.2 Usage

Admin> micbios [-h] [(getinfo|set-pass| \

set-cluster-mode|set-ecc|set-apei-ffm|set-apei-einjtable| \

set-apei-einj|set-fwlock|set-apei-supp)]

In the example below micbios sets the ecc mode to ‘disable’ for coprocessor mic0.

Admin> micbios set-ecc disable mic0

Successfully Completed

Adding --help to each command provides extra information about valid modes.

Admin> micbios set-ecc --help

Intel® MPSS tools


usage: micbios.py set-ecc [-h] [--password PASSWORD] mode

devices

positional arguments:

mode Mode Options: enable, disable, auto

devices Coprocessor's ID

optional arguments:

-h, --help show this help message and exit

--password PASSWORD The BIOS password must be provided to

change settings state. If not specified '' is the default

In the example below micbios sets the cluster mode to ‘all2all’ for coprocessor mic0.

Admin> micbios set-cluster-mode all2all mic0


Adding --help provides more information about the command.

Admin> micbios set-cluster-mode --help

usage: micbios.py set-cluster-mode [-h] [--password PASSWORD]

mode devices


mode Mode Options: all2all, snc2, snc4,

hemisphere, quadrant, auto


optional arguments:


--password PASSWORD The BIOS password must be provided to

change settings state. If not specified '' is the default

In the example below micbios sets the password ‘test’ to coprocessor mic0. The default password is empty.

Admin> micbios set-pass “” test mic0


Add --help to get more information about the command.

Admin> micbios set-pass --help

usage: micbios.py set-pass [-h] old_pass new_pass devices


old_pass The BIOS old password, if not specified '' is

the default.

new_pass The BIOS setting new password


optional arguments:


Intel® MPSS tools


79

8.8 The libsystools library

The libsystools library provides applications running on the host system with a C-

library interface to the coprocessor. Elements of the library are wrappers, built on top of the MIC host driver interfaces, that are available in the form of sysfs entries (in Linux*), WMI entries (in Windows*), and scif calls. Various functions are available to query the state of the coprocessor, and list or modify some of its parameters.

8.8.1 Function groups

Library functions are categorized into groups listed below.

1. Identify Available Coprocessors.

2. Error Reporting.

3. Device Access.

4. Query Coprocessor State.

5. General Device Information.

6. PCI Configuration Information.

7. Memory Information.

8. Processor Information.

9. Coprocessor Core Information.

10. Version Information.

11. LED Mode Information.

12. Turbo State Information.

13. SMC Configuration Information.

14. Throttle State Information.

15. Coprocessor OS Power Management Configuration.

16. Thermal Information.

17. Power Utilization Information.

18. Memory Utilization Information.

19. Power Limits.

20. Core Utilization Information.

21. Free Memory Utilities.

All libsystools functions return MICSDKERR_SUCCESS after successful completion, upon failure they return one of the error codes defined in the micsdkerrno.h file.

For detailed information about each of the functions refer to the help documentation, which is available in HTML format.

§

Intel® MPSS configuration parameters


A Intel® MPSS configuration

parameters

This section describes the parameters in the default.conf and micN.conf configuration

files. The parameter syntax in the following sections sometimes exceeds one line, however, each parameter in the actual configuration files must not contain a newline until the end of the entry.

As mentioned in Section 4, any parameter can be present in the default.conf file or in

any micN.conf file. A parameter in default.conf will apply to all coprocessors in the system unless the same parameter is present in micN.conf (coprocessor-specific

configuration files take precedence over default.conf).

Reboot the coprocessor to apply any configuration changes.

A.1 Meta configuration

A.1.1 Including other configuration files

Parameter Syntax (micN.conf):

Include <config_file_name>

Each configuration file can include other configuration files. The Include parameter

indicates the configuration file(s) to be included. The configuration file(s) to be

included must reside in the %INTEL_MPSS_HOME% directory or full path should be provided. The configuration parser processes each parameter sequentially. When the Include parameter is encountered, each included configuration file is immediately processed. If a parameter is set multiple times, the last instance of the parameter setting will be applied.

By default, the default.conf file is included at the start of each micN.conf file. This allows the coprocessor-specific files to override any parameter set in default.conf.

A.2 Boot control

Parameters listed in this section control the boot process of the coprocessor. They are located in each coprocessor’s micN.conf file.

A.2.1 EfiImage

Parameter Syntax:

EfiImage <target>



81

Initial value:

EfiImage %INTEL_MPSS_HOME%\filesystem\efiImage.hddimg

The EfiImage parameter specifies the location of the coprocessor OS EFI image.

A.2.2 KernelImage

Parameter Syntax:

KernelImage <linux_kernel_image>

Initial value:

KernelImage %INTEL_MPSS_HOME%\filesystem\bzImage.bin

The KernelImage parameter specifies the location of the coprocessor OS boot image.

A.2.3 KernelSymbols

Parameter Syntax:

KernelSymbols <system_address_map_file>

Initial value:

KernelSymbols %INTEL_MPSS_HOME%\filesystem\System.map

The KernelSymbols parameter specifies the location of the coprocessor OS address

map file.

A.2.4 InitRamFsImage

Parameter Syntax:

InitRamFsImage <target>

Initial value:

InitRamFsImage %INTEL_MPSS_HOME%\filesystem\core-

initramfs.cpio.gz

The InitRamFsImage parameter specifies the file system hierarchy in the coprocessor’s initial file system. The <target> argument is interpreted as the file name of a CPIO file

system archive.

A.2.5 AutoBoot

Parameter Syntax:

AutoBoot (Enabled|Disabled)



Initial value:

AutoBoot Enabled

The AutoBoot parameters controls whether the coprocessor is booted automatically

during the host OS startup. In the default configuration the parameter is present in all micN.conf files and set to Enabled. If this parameter is not present in any configuration file the behavior is the same as for AutoBoot Disabled.

A.3 File system configuration parameters

Parameters listed in this section configure the coprocessor’s file system. They are located in the micN.conf file.

A.3.1 RootFsImage

Parameter Syntax:

RootFsImage <file system file location>

Initial value:

RootFsImage %INTEL_MPSS_HOME%\filesystem\mic0.ext4

The RootFsImage parameter describes a special virtual block device named root which is backed by a file containing the root file system for the coprocessor OS.

The coprocessor will fail to boot if this parameter is not specified.

A.3.2 RepoFsImage

Parameter Syntax:

RepoFsImage %INTEL_MPSS_HOME%\filesystem\repo-card.squashfs

The RepoFsImage parameter describes a special virtual block device named repo,

which is backed by a squashfs repository image for the coprocessor OS, for example the repository image which is installed on the host alongside the Windows* cross-sdk (mpss-essentials-<version>.exe).

This parameter is used to create a block device, which is automatically mounted in the

/usr/share/mpss/packages/ directory on the coprocessor. This path is added as a local repository for the coprocessor OS and is available through the smart package manager.

Modifying this parameter allows to indicate different repository image for the

coprocessor. If it is set to a file that does not exist, it is ignored and a warning message is displayed.



83

A.3.3 BlockDevice

Parameter Syntax:

BlockDevice name=<block name> mode=[rw|ro] path=<image file

location>

This parameter has to be manually specified in one or more micN.conf configuration

files.

The BlockDevice parameter indicates virtual block devices that will be used on the

coprocessor OS as standard block devices (for example as hard disk drives). The mode option allows to specify access mode to each device (rw for read/write access, ro for read-only access). The path option indicates an image file on the host which backs the block device.

The BlockDevice parameter allows to specify up to 6 additional block devices. The

virtual device will be available on the coprocessor OS as a /dev/vd[a-z] device node.

Additionally, /dev/mpss_mapper/<block name> symbolic links will be created in the coprocessor OS allowing users to easily access the block devices by their designated names. For example, a block device named storage1 can be accessed by

/dev/mpss_mapper/storage1 symbolic link.

Note: Multiple coprocessors cannot share a single block device.

Note: Names root and repo are restricted for virtual block devices used by the RootFsImage and RepoFsImage parameters.

A.4 Kernel configuration

Parameters listed in this section influence or control the execution of the coprocessor’s

Linux* kernel through values passed to the kernel in the startup command line. They are located in the default.conf file.

A.4.1 KernelExtraCommandLine

Parameter Syntax:

KernelExtraCommandLine "<kernel_parameters>"

The KernelExtraCommandLine parameter specifies kernel command line parameters to

be passed to the coprocessor’s kernel on boot. Drivers for the coprocessor use a number of kernel command line parameters generated by the host driver. Default

parameters may be subject to change in future releases.

A.5 Network configuration

Parameters in this section control the network configuration of the coprocessor. The

HostNetworkConfig and CardNetworkConfig parameters are located in the micN.conf configuration files. The HostMacAddress and CardMacAddress parameters are not



created automatically, the administrator may place them manually in micN.conf files to change the default method of assigning MAC addresses.

A.5.1 Host network configuration

Parameter Syntax:

HostNetworkConfig (inet static address=<host IP> \

netmask=<host netmask> | none )

Initial value:

HostNetworkConfig inet static address=172.31.N+1.254 \

netmask=255.255.255.0

Description:

The default host network configuration is static. The IP address and netmask are assigned to the micN interface when it is created.

When the host network configuration is set to “none”, micctrl perform no operations.

The IP and netmask addresses are not assigned to the micN interface. To enable connection between the host and the coprocessor, modify network configuration manually (for example by editing the Internet Protocol Version 4 (TCP/Ipv4) properties in adapter settings in Windows*).

A.5.2 Coprocessor network configuration

Parameter Syntax:

CardNetworkConfig (inet|inet6 static address=<coprocessor IP> \

netmask=<coprocessor netmask> gateway=<coprocessor gateway> | \

inet|inet6 dhcp | none )

Initial value:

CardNetworkConfig inet static address=172.31.N+1.1 \

netmask=255.255.255.0 gateway=172.31.N+1.1

Description:

The inet6 value is used in IPv6 networks.

The default configuration is static. The coprocessor’s IP and netmask addresses are assigned during each boot.

If the configuration is changed to inet dhcp, the DHCP server will assign the IP,

netmask and gateway addresses.



85

A.5.3 MAC address assignment

Parameter syntax:

HostMacAddress (Serial|Random|<host MAC>)

CardMacAddress (Serial|Random|<coprocessor MAC>)

Description:

MAC addresses must be generated for the virtual network interfaces of the host and coprocessor. By default, MAC addresses are generated based on the serial numbers of

the devices. If a device does not have a usable serial number, the MAC address is generated randomly.

The least significant bit is set in MAC addresses generated for host endpoints, and not

set in MAC addresses generated for the coprocessor endpoints. In addition, the top three octets of the generated MAC addresses have the IEEE assigned value E4:FA:FD, which enables identification of the coprocessor’s interface.

The system administrator may override the default Serial behavior by explicitly setting

the HostMacAddress and CardMacAddress parameters in the micN.conf files. If the Random string is provided, a random address is generated overriding the serial number-based one. It is also possible to explicitly specify MAC addresses for the host and coprocessor network endpoints, for example:

CardMacAddress 00:1E:67:56:00:01

§

Intel® MPSS boot process


B Intel® MPSS boot process

Booting the Linux* kernel on the coprocessor requires a number of steps. Figure 8 shows the sequence of steps performed during the Intel® MPSS boot process.

micctrl --start

micctrl creates kernel command line

micctrl injects Linux* image into the coprocessor s

memory

micctrl injects Linux* initrd into the coprocessor s

memory

MIC driver triggers boot process on the

coprocessor

Linux* kernel boots and executes init

script in the attached initrd

Init script detects the virtio block root

filesystem

Init script loads Linux* drivers and

creates block nodes

Init script mounts /dev/vda onto

/new_root

Init script switches root to /new_root

Inform host that the coprocessor is ready

for use

Figure 8 Boot process for Intel® MPSS

B.1 Kernel command line

On most Linux* based systems loading and executing the kernel image is controlled

by the grub boot loader. In the grub configuration file, each possible kernel definition contains a number of parameters to be passed to Linux* through its kernel command line. In the Intel® MPSS boot system, this is done by micctrl parsing its configuration files. The kernel command line is created based on values in those files and placed in the coprocessor’s memory for retrieval.

B.2 Instruct the driver to boot the coprocessor

The micctrl utility requests the coprocessor to start executing the Linux* image.

When the driver receives the boot request, it first checks whether the coprocessor is in

the ready state. If it is not, an error will be returned and booting will not be attempted. Otherwise the state of the coprocessor is set to booting.

Intel® MPSS boot process


87

micctrl then saves the image file name for later retrieval. It also sets the mode to

indicate it is booting a Linux* image

micctrl copies the requested kernel command line and the requested Linux* image

files into the coprocessor’s memory.

The last step is to write to the coprocessor register instructing it to execute the

injected image.

Once the kernel is booted, the micctrl –s command will show that the coprocessor is

Online.

B.3 Coprocessor Linux* kernel execution

The Linux* kernel executes by initializing hardware, starting kernel services, and

setting all the CPUs to the online state. Once the kernel is ready, it initializes its initial ram disk image and starts executes the init script in the image.

The initial ram disk contains loadable modules required for the actual root file system.

Many of the arguments passed in the kernel command line are addresses required for the modules to access host memory. The init script parses the kernel command line

for needed information and loads the driver modules.

For the final step, the init script verifies the kernel command line’s root= parameter in

the kernel command line is /dev/vda, and take the appropriate actions.

B.4 Coprocessor’s root file system

A small initial file system is written to the coprocessor’s memory. This file system

contains only the drivers and utilities necessary to mount the virtio file system.

After the file is written and the coprocessor is booted, the init script loads necessary

drivers, creates block nodes, and mounts the virtio root device.

The virtio block file system is activated as the root device by calling the Linux*

switch_root utility. This special utility instructs the Linux* kernel to remount the root device on the mount point, release all file system memory references to the old initial ram disk and start executing the new /sbin/init.

/sbin/init performs the standard Linux* user level initialization. All the information required must already exist in the virtio block backing file.

The virtio root file system is implemented in the MpssControlService system service, which provides a virtual hard disk drive to the coprocessor. The virtual HDD is backed

by a file on the host system. From the users’ perspective, the device looks and acts like a standard HDD. Any modifications persist across reboots.

Adding or removing users, installing software, mounting a CIFS or NFS share, and

other administrative duties are accomplished in the same manner as on a standard Linux* OS. User data also persists across reboots.

References:

http://www.ibm.com/developerworks/library/l-virtio/

§

http://www.ibm.com/developerworks/library/l-virtio/

intel® manycore platform software stack (intel®...

Documents