pki appliance installation guide

PKI Appliance

Installation Guide

Public Key Infrastructure by PrimeKey

Ver: 2.7.2

2018-01-19

Copyright ©2018 PrimeKey SolutionsPublished by PrimeKey Solutions ABLundagatan 16171 63 SolnaSweden

To report errors, please send a note to [email protected]

Notice of RightsAll rights reserved. No part of this book may be reproduced or transmitted in any form by any means,electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of thepublisher. For more information on getting permission for reprints and excerpts, contact [email protected]

Notice of LiabilityThe information in this book is distributed on an “As Is” basis without warranty. While every precaution hasbeen taken in the preparation of the book, neither the authors nor PrimeKey shall have any liability to anyperson or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly bythe instructions contained in the book or by computer software and hardware products described in it.

TrademarksMany of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and PrimeKey was aware of a trademark claim,the designations appear as requested by the owner of the trademark. All other product names and servicesidentified throughout this book are used in editorial fashion only and for the benefit of such companies withno intention of infringement of the trademark. No such use, or the use of any trade name, is intended toconvey endorsement or other affiliation with this book.

Contents

I Preamble 1

1 Release Notes 2

2 Introduction 32.1 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Styling Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Daily operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 PKI Appliance Overview 53.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

II Appliance Installation 6

4 PKI Appliance Unboxing 74.1 Included in delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.2 Opening the box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.3.1 Front View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3.2 Back View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.4 Taking into Operation / Powering Up . . . . . . . . . . . . . . . . . . . . . 12

5 Initial Set-up 135.1 External Erase and Factory Reset . . . . . . . . . . . . . . . . . . . . . . . . 145.2 One Time Password and SSL Fingerprint . . . . . . . . . . . . . . . . . . . . 155.3 Changing the IP Address of the PKI Appliance . . . . . . . . . . . . . . . . . 165.4 Connecting to the PKI Appliance . . . . . . . . . . . . . . . . . . . . . . . . 175.5 Logging in for the first time . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.6 Fresh Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.7 Network Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.8 Date and Time Settings (NTP) . . . . . . . . . . . . . . . . . . . . . . . . . 235.9 Management CA Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.10 Security Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.10.1 Domain Master Secret . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.10.2 Appliance Security Level . . . . . . . . . . . . . . . . . . . . . . . . 255.10.3 PKCS#11 Slot Configuration . . . . . . . . . . . . . . . . . . . . . . 265.10.4 Audit Log Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.10.5 HSM FIPS Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.11 Confirm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.12 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.12.1 Get PKCS#12 key store . . . . . . . . . . . . . . . . . . . . . . . . 315.12.2 Using legacy browser enrollment . . . . . . . . . . . . . . . . . . . . 345.12.3 Get certificate from CSR . . . . . . . . . . . . . . . . . . . . . . . . 36

5.13 Finalize Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

III Appliance Advanced 41

6 HA Setup 426.1 Scope of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.1.1 How it works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.1.2 Synchronization of key material . . . . . . . . . . . . . . . . . . . . . 42

6.1.2.1 Pre-cluster setup generation of keys . . . . . . . . . . . . . 426.1.2.2 Post-cluster setup generation of keys . . . . . . . . . . . . . 43

Use-Case: Synchronize key material . . . . . . . . . . . . . . . . . . . . . . 436.1.3 Network topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436.1.4 Cluster traffic security considerations . . . . . . . . . . . . . . . . . . 44

6.2 Continuous service availability . . . . . . . . . . . . . . . . . . . . . . . . . . 446.3 Levels of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.3.1 Stand alone instance . . . . . . . . . . . . . . . . . . . . . . . . . . 446.3.2 Hot stand-by with manual fail-over . . . . . . . . . . . . . . . . . . . 446.3.3 High availability with automatic fail-over . . . . . . . . . . . . . . . . 45

6.4 High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Use-Case: Setting up a 2 node cluster from scratch . . . . . . . . . . . . . . 45Use-Case: Setting up a 3 node cluster from scratch . . . . . . . . . . . . . . 46Use-Case: Extending a cluster from n to n+1 nodes . . . . . . . . . . . . . . 46

6.5 Backup, Restore and Update . . . . . . . . . . . . . . . . . . . . . . . . . . 476.5.1 Backing up a cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . 476.5.2 Restoring a cluster from backup . . . . . . . . . . . . . . . . . . . . 47

Use-Case: Restoring a cluster from a backup taken on node 1 . . . . 48Use-Case: Restoring a cluster from a backup taken on node 2 or node

3, PKI Appliance firmware version 2.2.0 (or older) . . . . . 48Use-Case: Restoring a cluster from a backup taken on node 2 or node

3, PKI Appliance firmware version 2.3.0 . . . . . . . . . . . 486.5.3 Updating the software (firmware/applications) on a cluster . . . . . . 49

Use-Case: Software update on a three node cluster from 2.2.0 to 2.3.0 496.6 Controlled full cluster shutdown and startup . . . . . . . . . . . . . . . . . . 50

6.6.1 Shutting down the cluster in controlled manner . . . . . . . . . . . . 50

6.6.2 Starting a fully shutdown cluster . . . . . . . . . . . . . . . . . . . . 506.7 Operational Caution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Use-Case: Changing the IP Address of the Application Interface of anode in a three node cluster . . . . . . . . . . . . . . . . . 51

Replacing a failed cluster node . . . . . . . . . . . . . . . . . . . . . . . . . 52

7 PKCS#11 Slot Smart Card Activation 537.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537.2 Installation/Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

7.2.1 "Number of users required" . . . . . . . . . . . . . . . . . . . . . . . 547.2.2 "Number/copies of user smart cards" . . . . . . . . . . . . . . . . . . 547.2.3 "Require smart cards to activate system after boot" . . . . . . . . . . 547.2.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.2.4.1 Example with default values . . . . . . . . . . . . . . . . . 557.2.4.2 Slots 0 and 1 . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.3 Application/Activation of a slot . . . . . . . . . . . . . . . . . . . . . . . . . 557.3.1 Activation on boot/slot 0 . . . . . . . . . . . . . . . . . . . . . . . . 56

PKI ApplianceInstallation Guide – Public Key Infrastructure by PrimeKey

Ver: 2.7.2

Part I

Preamble

1 (56)


1. RELEASE NOTES Ver: 2.7.2

Chapter 1

Release Notes

PKI Appliance 2.7.2 Release Notes

This is a maintenance release to 2.7.1 which mainly brings new versions of EJBCAand SignServer to the PKI Appliance.

With the new EJBCA version custom certificate extensions for CV certificates areavailable. There are also improvements on CT logs.SignServer comes with support for one click certificate renewals from withinEJBCA.

New Features:* EJBCA Enterprise 6.10.1.2 - Please check out EJBCA release notes for more

detailed information* SignServer 4.2.0 - Please check out SignServer release notes for more details

Minor tweaks and bug fixes:* TimeMonitor was not active after restoring from an old backup (<= 2.5.1)* In some cases of improper shutdown some configuration was lost. This is fixed

now.* 2-node cluster setup now possible without errors on restore from old versions* Improved error reporting for Jboss

Known Issues and Limitations:* Setting up a peer connector fails when DHE is selected

2 (56)


2. INTRODUCTION Ver: 2.7.2

Chapter 2

Introduction

This manual provides an in depth understanding of the public key infrastructure (PKI) prod-ucts and services provided by PrimeKey and is intended to serve as a guide to understandingand implementing PKI as a product and service within the PKI Appliance.

2.1 AudienceThis guide is intended for use by Information Technology (IT) professionals with an interestin implementing the PKI products provided by PrimeKey in their environment using thePKI Appliance. The guide is presented in a structured manner so that it begins with anintroduction to the subject and progressively moves into more deeper technical topics. Thisallows the guide to be useful for a wide variety of personnel from managers to integrators.The lowest common denominator between the various groups of audiences is the sharedinterest in implementing PKI using PrimeKey products.

2.1.1 Styling Conventions

The following items explain the styling conventions that are used throughout this document,together with an example below each description:

• Buttons on the GUI are represented like Create .

• Options from popup menus or values that can be choosen like RSA 2048

• Links in the GUI that need to be selected/clicked upon are displayed in blue like:Search End Entities.

• Values that has to provided in text fields are presented as: a new value.

• Group titles or GUI text that is not selectable is represented as: RA Functions.

• Informative messages provide additional explanation of the steps being performed, orthe configuration being applied. For example:

3 (56)


2. INTRODUCTION Ver: 2.7.2

i This is an informative message containing extra information.

• Warning messages are used to draw the attention to a critical or sensitive step thathas to be performed, or to critical piece of information that has to be provided. Forexample:

! This is a warning message.

• Shell listings are used to specify commands that should be run on a server in a terminal,by a specific operating system user. For example:

Run as user

df -h

2.1.2 Daily operations

Exercises are indicated by the "Use-Case" prefix as illustrated below. Exercises provide a stepby step approach to perform an activity and require the practical environment:

Use-Case: Install PKI Appliance

While following the exercises outlined in this document, the following guidelines apply:

i Unless the instructions explicitly state so, do not deviate from the instruc-tion order. All steps should be performed in the sequence that they areoutlined in. Do not jump back and forth between different exercises, unlessthe instructions explicitly state so.

4 (56)


3. PKI APPLIANCE OVERVIEW Ver: 2.7.2

Chapter 3

PKI Appliance Overview

3.1 DescriptionEJBCA Enterprise Appliance is a PKI-in-a-box and combines the flexibility, reliability andfeature set of EJBCA Enterprise software, with a secure technology stack and enterprise-grade hardware including a FIPS 140-2 Level 3 certified HSM. Through the combination ofbuilt in CA, RA and VA functionality and a variety of interfaces like OCSP, CMP, SCEP andWebServices, EJBCA Enterprise Appliance provides a unique turn-key PKI solution.EJBCA Enterprise Appliance is based on an unified and controlled technology stack whichreduces technical risks for the entire PKI project and reduces patch management effortsduring operation. Simplified management and maintenance workflows lower the setup timeand operational costs and reduce the TCO.High flexibility, performance, support for high-availability and load-balancing make the EJBCAEnterprise Appliance suitable for critical infrastructure setups within commercial and gov-ernmental organization of all sizes.

As of version 2.4.0 the EJBCA Enterprise Appliance (or PKI Appliance) exists in threedifferent product sizes, designated as S, M or L. Previous unlabeled versions are equivalentto the M size. While the L version takes advantage of recently available bigger hard disksto provide for more database space, the S version is a highly reduced version with smallerdatabase size and also a reduced speed HSM.

5 (56)


Ver: 2.7.2

Part II

Appliance Installation

6 (56)


4. PKI APPLIANCE UNBOXING Ver: 2.7.2

Chapter 4

PKI Appliance Unboxing

Congratulations! You have obtained the PKI Appliance from PrimeKey Solutions AB.Illustrated below are the items that can be found while unboxing the PKI Appliance package.

4.1 Included in delivery• One PKI Appliance.

• One set of mounting rails, a mounting instruction and a set of screws.

• Four mains cables, one pair for each Europe and American standard.

• Optionally: One PIN pad and ten smart cards.

• A Quality Assurance Test Report

• A Packing List

7 (56)



4.2 Opening the boxBy opening the box you should find a PKI Appliance Test Report signed by PrimeKeyauthorized personnel showing the quality checks that have been performed.

Figure 4.1: Opening the box.

You will find 4 cables and rack mount sliding rails (see fig. 4.2).

Figure 4.2: Components inside the box.

8 (56)



Also there is a PIN pad with 10 smart cards (see fig. 4.3).

Figure 4.3: PIN pad with smart cards.

Finally the second layer reveals the packed PKI Appliance as shown in figure 4.4.

Figure 4.4: PKI Appliance packed in the cardboard box.

9 (56)



4.3 Overview

4.3.1 Front View

Figure 4.5: Front View of the PKI Appliance

1. Four bays for customer serviceable hard disks (Solid State Disks, SSD) for database,RAID1, two disks are provided

2. SSD Slot 0

3. SSD Slot 1

4. SSD Slot 2, empty

5. SSD Slot 3, empty

6. Cooling vents. Do not obstruct!

7. Status LED row: Power (green), Hard Disk (red), Info (yellow)

8. Front display for status information and IP address configuration with menu buttons:Up, Down, Enter, Cancel

9. Front USB ports, suitable for PIN pad connection

10. Safeguarded reset button

11. Power button (ATX)

10 (56)



4.3.2 Back View

Figure 4.6: Back View of the PKI Appliance

1. Two redundant Power Supply Units (PSU)

2. PSU Alarm mute button

3. IPMI Network port, to be not used, blocked in future versions

4. Mainboard USB ports, suitable for PIN pad connection

5. Application Network Interface

6. Management Network Interface

7. Hardware Security Module (HSM). USB and serial interface to be not used

8. optional: Connector for external battery and test automation

9. Safeguarded External Erase button for Factory Reset

10. Mainboard VGA connector, not required for operation

11. Mainboard Serial connection, not operational

12. Mainboard PS/2 connection, not required for operation

13. PKI Appliance serial number

11 (56)



4.4 Taking into Operation / Powering Up1. Make sure the seal at the right side of the PKI Appliance is intact and untampered

2. Make sure the serviceable hard disks are sitting properly in their bay

3. Make sure the PSUs are properly seated

4. Connect power cord

5. Do not yet connect the network cables

6. Power on the machine, booting will take about 5 minutes

12 (56)


5. INITIAL SET-UP Ver: 2.7.2

Chapter 5

Initial Set-up

The initial setup of the PKI Appliance transfers the device from the delivery state to aproduction setup by configuring all components of the system. The initial setup routinerequires four steps:

• Performing a Factory Reset

• Setting the initial management IP address using the control panel at the front

• Obtaining the One Time Password (OTP) from the display to access WebConf

• Running the WebConf and completing the setup

We recommend to not yet connect the network cables. As a general rule of precaution,we suggest that you first configure the IP addresses before connecting the PKI Appliance toyour network. Any previously configured IP address or the default IP addresses could alreadybe assigned to another network device in your network and thus disrupt service.

The network interfaces are:

• To the very left, next to a pair of USB connections, you will find a single networksocket which is not in service. To be not used. Never.

• Of the two network ports next to each other, the left one is the interface for theApplication Interface. It’s default IP address is 192.168.5.161.

• The right one of the two network ports is the Management Interface, which defaultsto 192.168.5.160.

13 (56)



5.1 External Erase and Factory ResetA Factory Reset resets the machine into factory defaults, a defined state deleting all con-figuration files and sensitive information like cryptographic keys on the Hardware SecurityModule (HSM) or certificates in the CA database. Performing a Factory Reset is necessaryin the following cases:

• you lose access to the PKI Appliance,

• you need to reinstall the PKI Appliance,

• you need to make sure that possibly secret data needs to be erased or

• you want to switch from testing or demo to a production system.

Figure 5.1: Placement of the External Erase button.

The following steps describe the procedure to perform a Factory Reset with the PKIAppliance:

! The next step is a definite action. All sensitive data will immediately beerased from the HSM. The only possibility to restore the data is from abackup (if one exists) and Backup Key Share smart cards, where required.

1. On the back of the PKI Appliance there is a hole underneath the integrated HardwareSecurity Module (HSM) with a hidden button (see figure 5.1). This is the buttonfor External Erase. Press that button for one second using a pen while the machineis powered, switched on, finished booting and make sure you hear a confirmationsound that should be played within 15 seconds (but might take up to ten minutesunder certain circumstances, e.g. if you slipped off the button and pressed it a secondtime).

14 (56)



i It is ensured that the HSM deletes the data as soon as the button is pressed.Under certain circumstances (as described above), the feedback (audibleand PKI Appliance front display) might take longer.

2. If the machine acknowledged that you pressed the button either by the audible feedbackor by the message on the front panel display, you will have to reboot the PKI Applianceto actually execute the Factory Reset by briefly pressing the power button on the frontpanel and then confirming the reboot via the display buttons. The machine will rebootand clear all configuration files. It should be clearly stated that a clean shutdown andboot is required for the configuration to be deleted. A hard power fail will not do.

3. After rebooting, the PKI Appliance display should show a cycle of the current Man-agement Interface IP address, the initial TLS fingerprint, some additional informationlike software version and the One Time Password. Seeing the One Time Password isproof that the Factory Reset was successfull.

i As soon as OTP is displayed, the PKI Appliance is in Factory Reset state,ready for installation.

5.2 One Time Password and SSL FingerprintAfter powering up the system, the display will give you the information you need to accessthe system through your web browser (see figure 5.2). The One Time Password (OTP) isrequired to initially access the WebConf and will become invalid after the installation hasbeen successfully accomplished. Please take note of this OTP as it will be required for theweb based installation procedure.

Figure 5.2: Front Display showing the One Time Password

The shortened TLS fingerprint indicated on the display shows the first characters of thefingerprint of the TLS certificate used to secure the connection from your web browser tothe PKI Appliance WebConf (see figure 5.3). The WebConf will ask you to compare thisfingerprint with the fingerprint of the TLS certificate presented to you by the browser tomake sure that you are accessing the right machine.

15 (56)



Figure 5.3: Front Display showing the TLS Fingerprint

5.3 Changing the IP Address of the PKI ApplianceAfter a factory reset and also later during normal operation the display will show you theIP address of the Management Interface of the PKI Appliance. After a factory reset, thiswill default to 192.168.5.160 (see figure 5.4).

Figure 5.4: Front Display showing the IP Address

If the default IP address of the Management Interface of the PKI Appliance does notmatch your network configuration, you can easily change it according to your needs. However,it is preset to have a network prefix of /24 (resulting in a subnet mask of 255.255.255.0).

i As the 100.64.0.0/10 network range is used for internal networking, IP ad-dresses in this range are not allowed as external management or applicationnetwork address.

Pressing the "OK" button when the IP address is shown will allow you to change the IPaddress (see figure 5.5). The IP address will be presented with leading zeroes. The cursorwill start at the first digit of the first byte of the IP address. You can abort this operationat any time by pressing the x button.

16 (56)



Figure 5.5: Changing the IP Address

1. use the up and down buttons to adjust the digit to your target IP address.

2. then press the v button to confirm this digit

3. the cursor will move to the next digit

4. repeat steps 1 to 3 for every digit

5. when confirming the last digit with the v button, the display will ask you to confirmthe IP address. This time, the IP address will be shown without leading zeroes.

6. confirm your entry with the v button.

The chosen IP address will be committed. Please note that this operation can take up to10 seconds. After that time, it is safe to connect the first network cable to the ManagementInterface (the right one, as seen from behind).

5.4 Connecting to the PKI ApplianceThe next and last step of the initial configuration of the PKI Appliance is to run the webbased configurator. During this procedure all components of the system will be configuredaccording to the parameters you provide.

i The WebConf is designed and tested to work with Firefox 26.0+. Otherbrowsers like Chrome or Safari are working but are not officially supportedand you may observe minor incompatibilities. Internet Explorer is currentlynot officially supported and depending on the version you might not be ableto finish the configuration process successfully.

1. Navigate your browser to the IP address of the Management Interface of the PKIAppliance. A simple web page will instruct you to connect through TLS (see figure5.6).

2. Follow that link and your browser will respond with a TLS warning because the servers

17 (56)



Figure 5.6: Instruction to connect to the PKI Appliance using TLS

TLS certificate is not signed from any CA your browser knows already (see figure 5.7).

Figure 5.7: Browser TLS Warning

3. Open the I Understand the Risks section by clicking that link

4. then click the button Add Exception... :

5. Untick Permanently store this exception if you plan to install the machine now. Thecertificate will be regenerated during installation and the permanently stored certificatewould be obsolete. Confirm the Security Exception by clicking Confirm Security Exception(see figure 5.8).

i If you don’t wont to be prompted again to confirm don’t untick.

18 (56)



Figure 5.8: Confirm Security Exception

6. You will be greeted by the WebConf (see figure 5.9).

Figure 5.9: Instruction to compare and confirm the TLS certificate fingerprint

7. Check the fingerprint of the TLS certificate and compare the first characters to thefingerprint shown on the display of the PKI Appliance.

(a) Click the little padlock icon in the address bar of your browser (see figure 5.10).

19 (56)



Figure 5.10: Firefox padlock information window

(b) Click on More Information... (see figure 5.11).

Figure 5.11: Security Information

(c) Click on View Certificate . You will be shown the SHA1 fingerprint. Thefingerprint should correspond as much as was visible on the display (see figures5.12 and 5.3).

20 (56)



Figure 5.12: Certificate Information

8. If the two fingerprints match, then you can be sure to be connected to the correctmachine. Click The fingerprints are the same as in 5.9.

21 (56)



5.5 Logging in for the first timeNow you will need the One Time Password (OTP) that is displayed on the front of the PKIAppliance. This password changes every time the machine is started, until the system hasbeen installed. Click Login when you have entered the authentication code (see figure5.13).

Figure 5.13: Entering the OTP

5.6 Fresh InstallationAnytime you use the OTP to log in to an un-provisioned PKI Appliance, you will be giventhe choice to

1. Fresh install

2. Restore system from backup

3. Connect to cluster

For now we will do a fresh install, so click the Next button below Fresh install (seefigure 5.14)

5.7 Network SettingsYou will be asked to configure the network settings of the PKI Appliance. All of this can becorrected at a later point in time, if needed.

You might want to make up your mind about the network configuration beforehand: Ofthe two physical interfaces, one is designed to be a Management Interface, through whichyou can access the WebConf and the AdminGUI of EJBCA. The other interface is designed

22 (56)



Figure 5.14: Installation Choices

Figure 5.15: Network Settings

to be the Application Interface, through which the operational payload will be routed. It’sperfectly fine to set up two separate networks if you want to separate those tasks. For thetime being, the Management Interface IP address has been configured at the front paneldisplay and is preset to have a network prefix of /24 (subnet mask 255.255.255.0). On theapplication network however, you are free to chose the IP address, network prefix anddefault gateway. You will also be asked to enter the designated hostnames, if you plan tomake the PKI Appliance available through DNS name resolution.

After the installation, you will be given the possibility to change the IP address of theManagement Interface.

To confirm the configuration and proceed to the next step, click on Next: Time (seefigure 5.15).

5.8 Date and Time Settings (NTP)For many of the applications of a Public Key Infrastructure (PKI), it is very important tohave a correct date and time. You might consider using a Network Time Protocol (NTP)

23 (56)



time source. If you plan to build a cluster, you have to use NTP.

Figure 5.16: Date and Time Settings (NTP)

Proceed to the next page of the configuration by clicking the Next:Management CAbutton.

! In case that you will use NTP this is the right time to do it! If you configureit later and there is a difference between the NTP server and current systemtime, the synchronization will not happen directly. It can take up to severalhours.

5.9 Management CA SettingsThese are settings that should be carefully considered, because they cannot be altered afterthe installation. You should take the time to think of some meaningful identifier to be addedto the Additional Subject Fields, as shown in the picture. The Additional Subject DNwill be reflected in the TLS certificates that are stored in your browser and in the name ofthe backup files. If you plan on doing several test/demo installations, this is where you canbrand them.

Figure 5.17: Management CA Settings

If you have already an TLS PKI somewhere, you can opt to not generate a new Man-agement CA but use an existing Management CA. You will be prompted to upload thePEM-encoded CA certificate. In case you need the Management CA to be created now, youwill be asked to configure it:

24 (56)



• Common Name of the EJBCA Management CA

• Additional Subject Fields like organization and country

• Signature Algorithm that shall be used by the EJBCA Management CA

– SHA1withRSA

– SHA256withRSA

– SHA256withECDSA

• Signing Key Specification

– ECDSA - secp256r1 / prime256v1 / P-256

– RSA 1024

– RSA 2048

– RSA 4096

• EJBCA SuperAdmin Common Name

Continue by clicking on Next: Security .

5.10 Security SettingsThis is another page of immutable settings. The security section helps you to configure allsecurity relevant aspects of the PKI Appliance.

5.10.1 Domain Master Secret

The first step is to set a secret for your Domain Master Secret. This passphrase is usedto derive a symmetric key which is used to encrypt backup archives created by the PKIAppliance. It is your choice whether you specify it manually or whether you prefer to haveit generated by the system. If generated, you will be given the possibility to print the highlysecure Domain Master Secret. In both cases it is very important to write down the secretand keep it in a safe place. If lost, the device will not be able to be restored from a backup.Also you would not be able to extend this system to a cluster.

5.10.2 Appliance Security Level

There are three options for the Appliance Security Level:

• Soft key files

• 2 out of 3 Backup key share smart cards

25 (56)



Figure 5.18: Security Settings

• 3 out of 5 Backup key share smart cards

This option defines if and how many smart cards shall be used to protect the HSM keymaterial. As an example, if 2 out of 3 Backup key share cards is chosen, you will beasked to insert 3 smart cards during installation where on each a share of a symmetric key(the Backup Key) will be stored. The symmetric key will be used to encrypt the backups.As the Backup Key is also securely stored on the HSM you will not need to provide thesmart cards for every backup operation. Should it be necessary to restore the PKI Appliancefrom a backup you will need to provide 2 of the initially created 3 smart cards to importthe Backup Key into the HSM to decrypt and import the backup data. Likewise for the3 out of 5 Backup key share smart cards scenario.

For low security or testing scenarios it is also possible to operate the PKI Appliance with-out smart cards and use software based keys which are stored on the PKI Appliance instead.In this case, any backup of cryptographic keys (from the HSM) will not be additionally se-cured by the Backup Key Share smart cards, but only by the Domain Master Secret, thatencrypts all data in a backup file.

5.10.3 PKCS#11 Slot Configuration

The next option on this page is to change the authentication codes for the PKCS#11slots of the HSM. Automatically generated authentication codes are stored on the systemso that applications can run unattended while still offering a decent security. Manuallygenerated authentication codes allow for applications that should only be available aftermanual activation. Even higher security can be achieved by enabling smart card activationon slots. (Minimum PKI Appliance Version 2.2.0, please refer to chapter 7 on page 53for more information about smart card activated slots. Please notice that the smart card

26 (56)



activation for PKCS#11 slots is not available with HSM FIPS Mode, see below.)

5.10.4 Audit Log Storage

This option allows you to choose whether you want to store signed log records of securityoperations to the clustered storage. Default is enabled. Audit log records consume databasedisk space. For a typical installation, the creation of a single certificate issues approximately10 audit log records. For all typical installations, the audit log database table will be at leastdouble the size of the other database tables. If you disable the storage of the signed auditlog, you will still be able to receive and store the audit log records externally, over syslogshipping (unsigned, unencrypted).

5.10.5 HSM FIPS Mode

This last option offers you to load and activate the HSM FIPS Mode firmware module.It will enforce restrictions required by the FIPS 140-2 standard. This means that someknown unsecure mechanisms and algorithms will be disallowed, but also new or modernmechanisms and algorithms will not be available because they have not yet been approved.A known limitation is that the PKCS#11 slots cannot be authenticated with smart cardswhen FIPS restrictions have been requested.

To continue, click on Next: Summary to see an overview of all configuration optionsdone so far.

5.11 ConfirmIt is highly recommended that you double check everything on this summary page. Youmight even want to print this page. If you spot an error, you can easily navigate backwardswith the Previous buttons or use the breadcrumbs at the top of the screen.

i In case you have decided to use smart cards for your setup, please make surethat the PIN pad included in the delivery is connected to one of the USBports in the front of the PKI Appliance and you have a sufficient amountof smart cards at hand. The smart cards are delivered with the default PIN"123456". You will be given an opportunity to change the PIN of a smartcard after installation has finished, see chapter ?? on page ??

When you are ready to continue the installation click on Begin installation . Theinstallation will take a few minutes (see figure 5.19).

27 (56)



Figure 5.19: Confirm installation choices

5.12 InstallationThe installation process will take a few minutes. During this time you can follow the installa-tion and configuration steps shown below the progress bar which will include the configurationof the HSM, the database and the applications, like EJBCA.

i In the case you have decided to use smart cards, please mind the outputfrom the PIN pad during the installation process which will request you toinsert the smart cards and enter the PIN. You will be asked to enter thesmart cards in two steps using the k out of n schema:

1. Key generation: Insert all (n) smart cards you have chosen to use,always providing the PIN.

2. Key import (to HSM): Insert again the amount of smart cards thatis needed to restore the Backup Key (k)

28 (56)



At the end of the installation, you will find the following screen (see figure 5.20).

Figure 5.20: End of Installation

29 (56)



To manage the PKI Appliance you need to get a client side SuperAdmin TLS certificateissued by the Management CA that can be used from your browser. This certificate will beyour one and only authentication to the system, unless you configure other access meth-ods. Configuration of further users and other authentication methods are described in theWebConf chapter (see page ??).

Select the option that suits your current client environment.

1. Get PKCS#12 key store: The SuperAdmin certificate and corresponding key pair isgenerated on the PKI Appliance and manually imported into the browser.

2. Using legacy browser enrollment: The SuperAdmin key pair is generated in thebrowser and the SuperAdmin certificate is automatically imported into the browser.

3. Get certificate from CSR: The SuperAdmin key pair is generated outside the browsercontext and the SuperAdmin certificate will be created from a Certificate SigningRequest.

The certificate and corresponding key pair is a vital component of your system. Youneed to protect and backup it with the same care that you apply to the backups and dataof the PKI Appliance itself: Anyone in possession of this certificate can manipulate yourinstallation. Without this certificate, you have no access whatsoever to the PKI Appliance.

30 (56)



5.12.1 Get PKCS#12 key store

A PKCS#12 key store is a format for storing both private keys and certificates protectedby a password. By selecting this option you will be able to download such key store thatcontains both a SuperAdmin certificate and the corresponding key pair. The .p12-file thenneeds to be manually imported into the browser using the PKCS#12 protection passwordshown to you.

Start by pressing Confirm enrollment option when "Get PKCS#12 key store" isselected (see figure 5.21).

Figure 5.21: Get PKCS#12 key store - step 1

31 (56)



Next, press Get SuperAdmin PKCS#12 key store (see figure 5.22). A new tabwill open.


32 (56)



In the newly opened tab, select a Key Specification matching your organization’s securityrequirements an click Enroll (see figure 5.23). You will be prompted to save .p12-file.Download it to the local machine.


Close the newly opened tab. Back in the installation wizard tab (see figure 5.22), makea note of the PKCS#12 protection password. Use your browser’s proprietary mechanism forimporting the .p12-file using the PKCS#12 protection password before proceeding.

Once the P12 has been successfully imported, click Finalize installation .

33 (56)



5.12.2 Using legacy browser enrollment

Start by pressing Confirm enrollment option when "Using legacy browser enrollment"is selected (see figure 5.24).

Figure 5.24: Using legacy browser enrollment - step 1

Click that link labeled Get SuperAdmin certificate (see figure 5.25). A new tab willopen.


34 (56)



In the newly opened tab, click Enroll . Your browser will then generate a key pair,request the certificate from the Management CA and automatically install the certificate inyour browser (see figure 5.26). Confirm the popup and close the tab.


Back in the installation wizard tab (see figure 5.25), click Finalize installation .

35 (56)



5.12.3 Get certificate from CSR

Enrolling the initial SuperAdmin certificate using a Certificate Signing Request/PKCS#10should only be used when you can’t use any of the other methods. Creation of the CSR andinstalling the resulting certificate in such a way that it is usable for client TLS authenticationis outside the scope of this document.

Start by pressing Confirm enrollment option when "Get certificate from CSR" isselected (see figure 5.27).

Figure 5.27: Get certificate from CSR - step 1

36 (56)



Make a note of Enrollment username and Enrollment code. Click that link labeledGo to SuperAdmin enrollment page (see figure 5.28). A new tab will open.


37 (56)



In the newly opened tab, enter Enrollment username and Enrollment code from theprevious page. Select or paste the certificate signing request you want to use to issue theinitial SuperAdmin certificate. Click OK . (See figure 5.29.)


38 (56)



Download the certificate (see figure 5.30) and install it (using some proprietary method).Close the tab when done.


Back in the installation wizard tab (see figure 5.25), click Finalize installation .

39 (56)



5.13 Finalize InstallationAs the very latest step of our installation, you have to finalize the installation by clickingthe button Finalize installation . Finalizing takes some 30 seconds. The browser willreload the page and ask you to confirm that your (or which) client side certificate shall beused for authentication (see figure 5.31). If you use different Additional Subject DN forthe different installations, the matching certificate should be pre-selected. (Should you everneed to delete certificates from your browser, please keep in mind that you need to restartyour browser for these changes to take full effect).

This is also the moment where you can connect the second network cable to the Appli-cation Interface (the left one, as seen from behind) if you had not done this before.

Figure 5.31: Certificate Selection

Due to the inner workings of the PKI Appliance, configuration changes onlyget persisted after approximately one hour (or when the machine is properly shutdown/rebooted), leading to lost configuration in case of a power outage rightafter installation. This might be relevant if you are running a test installation onyour desk or in a test lab.

40 (56)


Ver: 2.7.2

Part III

Appliance Advanced

41 (56)


6. HA SETUP Ver: 2.7.2

Chapter 6

HA Setup

6.1 Scope of availabilityFor the PKI Appliance the availability is defined as being able to keep the service running withfull data integrity for the applications running on the PKI Appliance that uses the internalSQL database.

6.1.1 How it works

The cluster implementation used on the PKI Appliance uses regular network connectivityover the Application Interface for all cluster communication. This means that cluster nodesdon’t have to be placed physically close to each other as long as they have good networkconnectivity.

However, this also means that a node cannot distinguish between a node failure ofanother node and broken network connectivity to the other node. To avoid the situationwhere the cluster nodes operate independently and get diverging data sets (a so called splitbrain situation), the cluster nodes take a vote and will cease to operate unless they are partof the majority of connected nodes. This ensures that there is only one data set that isallowed to be updated at the time. In the case of a temporary network failure, disconnectednodes can easily synchronize their data to the majority’s data set and continue to operate.

6.1.2 Synchronization of key material

Key material stored in the HSM is not automatically synchronized after the cluster has beenset up. Manual synchronization is however possible.

6.1.2.1 Pre-cluster setup generation of keys

If suitable for your use-case, you could generate all keys that will be used during the instal-lations life-time after installing the first node, but before starting the cluster configurationfor the additional nodes. This way, all additional cluster nodes will be provisioned with the

42 (56)



complete key material on installation and no additional manual key synchronization will benecessary.

6.1.2.2 Post-cluster setup generation of keys

When generating new keys (or in any other way modifying the key material) after the clusterhas been setup, you need to manually synchronize the key material.

Note that applications that are connected to the shared database may malfunction ifthey try to use references to keys that are not yet synchronized. For example, if a CertificateAuthority in EJBCA is renewed with new key generation, other cluster nodes shortly afterthe renewal will try to use the new key. This will fail since the key generation was local tothe node where it was performed.

Use-Case: Synchronize key material

1. On Node 1: Generate the key pair(s) on the first node.

2. On Node 1: Go to the HSM tab of the PKI Appliance WebConf and download a "Clus-ter Key Synchronization Package" by clicking Download protected HSM backup.

3. On Node n: Go to the HSM tab of the PKI Appliance WebConf and upload thepackage.

4. Repeat step 3 for each node (n>1).

5. Configure the application to start using the new key pair(s).

Since node 1 has higher database quorum vote weight, it is generally advised to generatethe keys there to avoid a reboot and potential downtime in a two node setup.

6.1.3 Network topology

All cluster nodes should have a dedicated connection to all other nodes in the cluster.However the cluster can propagate the data as long as all nodes are connected to at leastone other node.

The network connection is done via the GRE protocol (IP protocol number 47, seehttps://en.wikipedia.org/wiki/List_of_IP_protocol_numbers). Since GRE is anIP protocol, it is not based on either TCP or UDP and has no concept of ports. It is anIP protocol by itself. That means that it can not simply be made available with a portforwarding behind a NAT (Network Address Translation). A fully transparent VPN solutionwill be required if the cluster is supposed to be installed over different locations.

If you do have network equipment that is able to encapsulate the protocol, you mightstill run into the issue of network address complications. This is easiest worked aroundby setting up the systems in a simpler network configuration (e.g. same site) and latershipment/reconfiguration.

43 (56)

https://en.wikipedia.org/wiki/List_of_IP_protocol_numbers



A cluster node will never forward traffic between two other nodes to avoid networkingloops. Compared to using the spanning tree protocol (STP), this means that a brokennetwork connection between two nodes will not trigger any downtime of other connections.

If you prefer the dynamic loop prevention behaviour, you could add managed switches infront of the Application Interfaces of the PKI Appliances. Please note that if the networktopology change prevents network traffic between the nodes for too long, your cluster nodesmight stop operation and require manual interaction. Rapid Spanning Tree Protocol (RSTP)might be an interesting alternative to STP in this case.

6.1.4 Cluster traffic security considerations

The current version of the PKI Appliance uses no protection for the cluster traffic. IPSecwill be used in a later release, but for now you need to ensure that this sensitive traffic isprotected by other means.

6.2 Continuous service availabilityTo ensure that service clients always connect to an operational node in the cluster, an externalload-balancer should be used for automatic fail-over and/or load distribution.

In the case a custom application is being developed for consumption of the servicesprovided by the PKI Appliances’ external interfaces, this could also be handled by makingthe custom application connect to any of the nodes that is found to be operational.

If lower availability and manual interaction is acceptable in case of a node failure, thiscould also be solved by redirecting a DNS name to the service.

6.3 Levels of availability

6.3.1 Stand alone instance

This is a basic single node installation of the PKI Appliance. In case of a node failure anew PKI Appliance needs to be reinstalled from a backup. All data between the time of thelatest backup and the failure will be lost. If a cold stand-by (spare) PKI Appliance is notavailable, the time of delivery of a new box needs to be taken into account when calculatingthe acceptable downtime.

6.3.2 Hot stand-by with manual fail-over

In this setup, two nodes are connected as a cluster where the first installed node has a higherquorum vote than the second node.

In the case the second node fails, the first node will continue operating but the secondnode will be set into maintenance. In the case the first node fails, the second node will ceaseto operate and will be set into maintenance. To bring back the second node into service itrequires manual interaction via the PKI Appliance administrative interface (WebConf).

44 (56)



To avoid data loss, the manual interaction is required and the secondary should only bepromoted if the first node really is dead and will be replaced.

6.3.3 High availability with automatic fail-over

This is a setup with three or more nodes. In case of a node failure, the remaining nodes willstill be able to form a cluster through a majority quorum vote and continue to operate. Ifthe PKI Appliance that has failed is still switched on it will be set into maintenance.

The first cluster node always has a slightly higher quorum vote than the rest of the nodes.In a setup of an even (4 or more) number of nodes where the nodes are divided over twosites, the site that has the first node will continue to operate if the connectivity between thesites fails.

6.4 High Availability

Use-Case: Setting up a 2 node cluster from scratch

1. Make a fresh install according to the normal installation procedure or restore a nodefrom backup.

2. If possible, generate all keys in the HSM that will be used during the installationslife-time to avoid manual key synchronization later.

3. Go to the cluster tab on the initial node in the PKI Appliance WebConf and add aconnection to where the next node’s Application Interface will be.

4. From the same tab, download the setup bundle for the second node.

5. Factory reset the second node and connect to the web based installer

6. Select Connect to cluster and upload the setup bundle.

7. At this point, both network cables need to be connected to the second node. Startthe installation procedure.

8. After installation completes, you should be able to manage the new node using thesame credentials as the first one.

If the first node has been used for a while before the second node was connected, youmight need to wait until the data is fully synchronized, even after the cluster connection hascompleted. When the Local node state in the WebConf’s Status tab shows Ok, the nodeis ready for use.

45 (56)



Use-Case: Setting up a 3 node cluster from scratch

1. Make a fresh install according to the normal installation procedure or restore a nodefrom backup.

2. If possible, generate all keys in the HSM that will be used during the installationslife-time to avoid manual key synchronization later.

3. Go to the Cluster tab on the initial node in the PKI Appliance WebConf and add thetwo connections to where the next nodes’ Application Interface will be.

4. From the same tab, download the setup bundle for the two new nodes.

5. Factory reset the second node and connect to the web based installer

6. Select Connect to cluster and upload the setup bundle for node 2.

7. At this point, both network cables need to be connected to node 2. Start the instal-lation procedure.


9. Even if a full synchronization between the first and second node is still running at thispoint, you can proceed with the cluster connection of the third node.

10. Factory reset the third node and connect to the web based installer

11. Select Connect to cluster and upload the setup bundle for node 3.


If the first node has been used for a while before the two new nodes were connected, youmight need to wait until the data is fully synchronized, even after the cluster connection hascompleted. When the Local node state in the WebConf’s Status tab shows Ok, a node isready for use.

Use-Case: Extending a cluster from n to n+1 nodes

1. Go to the cluster tab on all of the existing (n) nodes in the PKI Appliance WebConfand add a connection to where the next node’s Application Interface will be.

2. From the same tab on one of the nodes, download the setup bundle for the new node(n+1).

3. Factory reset the new node (n+1) and connect to the web based installer

4. Select Connect to cluster and upload the setup bundle.

46 (56)



5. At this point, both network cables need to be connected to the new node. Start theinstallation procedure.

6. After installation completes, you should be able to manage the new node (n+1) usingthe same credentials as the first one.

When the Local node state in the WebConf’s Status tab shows Ok, the new node is readyfor use.

6.5 Backup, Restore and UpdateIn the domain of High Availability/Clustering, the topics of backup, restore and update haveto be handled differently as compared to stand alone instances of the PKI Appliance to notdisrupt operation.

6.5.1 Backing up a cluster

Although that you have set up a High Availability Setup to prevent any outages, you shouldalways take full-out scenario into consideration. In this case, and only in this case, you willhave to recover your cluster from a backup. From operational perspective, it might makesense to decide to take backups only from node 3 (which is designed to be at a disasterrecovery site off-location) to reduce load and network traffic on the nodes at the main site.However, it is only with PKI Appliance version 2.3.0 that we properly support recoveringwith a backup taken on node 3. Even then, the procedure to recover a full-out disaster ismore complicated if the system is to be restored from a backup of node 3 or node 2 ratherthan node 1.

If you can afford, we recommend to set up a automated backup schedule on all of yournodes to make sure to be able to recover everything, out of every situation, even if perhapsa failure takes a long time to be discovered.

Generally speaking, a backup always contains all information of a cluster node (config-uration and database), including its node identity. For example, a backup file taken fromnode 3 will not just create any node of a cluster, but exactly node 3 when restored. A node2 or node 3 is always configured to not run alone after a boot, but only in conjunction witha node 1 or if manually forced into primary, to be repeated after every reboot. Therefore,having a backup of node 1 is always preferable when you need to recover your cluster froma full-out scenario.

6.5.2 Restoring a cluster from backup

A backup file of a cluster node should only be used in the highest emergency of a full-outscenario. If at least one node remains operational, the cluster should always be reestablishedfrom the last good node. Pick your case from the following list of Use-Cases:

47 (56)



Use-Case: Restoring a cluster from a backup taken on node 1

A backup file of any cluster node should only be restored in a case of utmost emergency. Ifyou really have to, the first step in recovering your cluster is to restore the backup of thenode 1 to the machine designated to be node 1. Please refer to chapter ?? (on page ??)for a description on how to restore a backup to a PKI Appliance. The machine should comeup operational, with other cluster nodes/cluster connections configured, but not connectedto them. Make sure that the assigned IP addresses are matching according to your plans.You can now go ahead, download the cluster setup bundles and start connecting the otherremaining nodes to your node 1 as to reestablish high availability.

Use-Case: Restoring a cluster from a backup taken on node 2 or node 3, PKIAppliance firmware version 2.2.0 (or older)

A restore of a backup taken on node 2 or node 3 with a PKI Appliance software version2.2.0 or older is currently not supported. Please contact PrimeKey Support or your localPrimeKey Partner for support.

Use-Case: Restoring a cluster from a backup taken on node 2 or node 3, PKIAppliance firmware version 2.3.0

A backup file of any cluster node should only be restored in a case of utmost emergency. If acluster needs to be recovered from backup, it is highly recommended to do so with a backupfile that has been created on node 1. If you really have to, the first step in recovering yourcluster from a node 2 or node 3 backup file is to restore the backup to the according machine.A backup file from node 2 should be restored to the PKI Appliance designated to be node2, likewise a backup file from node 3 should be restored to the PKI Appliance designated tobe node 3. Please refer to chapter ?? (on page ??) for a description on how to restore abackup to a PKI Appliance. After reboot, the WebConf will be reachable and operational,but the database will refuse to start up in this situation, hence the applications will not yetbe operational. (The button Force into Active (formely "Force into Primary") )that theWebConf offers only starts the database, it does not yet start the applications).

The second step of recovering your cluster is to reestablish your node 1. Make surethat the assigned IP address is matching according to your plans. You can now go ahead,download the cluster setup bundle and start connecting the PKI Appliance designed to benode 1 to the Appliance that you just restored from backup. Node 1 should come upoperational. You might need to force node 1 into primary.

The next step is to connect the remaining third node (node 2 or node 3, depending onwhether you started the operation with node 3 or node 2). To do so, add the cluster nodeto the configuration of your two nodes, download either cluster setup bundle and set up thethird PKI Appliance. It should come up fine, operational on database and application.

Once that you have connected all three nodes to each other, you will have to reboot thecluster node that you initially restored from backup. It will now come up with database andapplication operational.

48 (56)



6.5.3 Updating the software (firmware/applications) on a cluster

Updating the software of the PKI Appliance will always require a reboot. A reboot of a PKIAppliance in a cluster should always be scheduled with care as to not accidentally degradecluster performance. It is a common mistake to ease up on the operational caution when itis known that some technical measures are in place to take care of outages and thus giveaway any safety margins. In a cluster, software update should be applied on a single node ata time. Only if the node you are currently working on is completely done with the updateand confirmed to be back up and running should you proceed to updating the next node.

Starting with version 2.2.0, the PKI Appliance firmware is to be updated separately fromthe applications installed on the platform of the PKI Appliance. You are supposed to upgradeboth the firmware and the application, starting with the firmware.

A PKI Appliance on a version older than 2.2.0 can not simply be customer-upgraded dueto major architectural changes. Please contact PrimeKey Support or your local PrimeKeypartner for support.

For procedures on how to update a cluster on PKI Appliance version 2.3.0 to an evennewer version, please refer to the even newer documentation delivered with the new softwareversion.

Use-Case: Software update on a three node cluster from 2.2.0 to 2.3.0

To update a three node cluster from PKI Appliance version 2.2.0 to 2.3.0, please proceedwith the following steps:

1. Before starting any configuration changes on a cluster node, you should assert that thenode has been running fine up to now. This is the only way to know for sure whetheryou actually broke anything if the procedure does not succeed as expected.

2. You might also want to make a last manual backup of the PKI Appliance

3. Make sure this cluster node is declared as not operational, (e.g. disabling in loadbalancing frontend), so that:

• no other operator does any maintenance on any other node while we deliberatelyreduce redundancy on the cluster,

• nobody relies on the availability of this node during maintenance downtime,• and no alarm is raised if this node gets unavailable.

4. Start the software update procedure on this node by updating the PKI Appliancefirmware first, then updating the COS applications. This should generally be the sameprocedure as described in ??: Install firmware, reboot, install application.

5. After the cluster node has been rebooted, check that the node is operating correctly.

6. After you asserted that this node is up and running, verify that the entire cluster is ingood shape, i.e. that all of the cluster nodes of your cluster confirm that your clusteris back up and running with redundancy.

49 (56)



7. Announce this cluster node to be operational back again or whatever you need to undofrom step 3.

8. Continue with updating your cluster by applying the same steps on the next clusternode, restarting at step 1.

6.6 Controlled full cluster shutdown and startupThis section describes how to do a controlled shutdown of the whole cluster and get backto a fully running state.

6.6.1 Shutting down the cluster in controlled manner

When shutting down an N node cluster, start with the highest node number and wait untilthe node is fully shutdown before proceeding with the next one. This ensures that the quorumis kept as long as possible and in the end node 1 is the most up to date node.

6.6.2 Starting a fully shutdown cluster

Start by identifying the node that had an OK database status last before the shutdown. Ifyou performed a controlled shutdown as described in 6.6.1, node 1 is guaranteed to have themost up to date data. Since the cluster uses synchronous replication, a power outage thattakes down all nodes forming the quorum allows you to start with any of these nodes. If youhave shutdown the nodes in some other order or a minority of nodes had been disconnected,you need to keep track of which server was holding the quorum last (had database statusOK in WebConf).

1. Power up all nodes.

2. Once the node that has the most up to date copy of the clustered data has started,promote the node using Force into Active (formerly "Force into Primary").

3. Wait until all N nodes are fully started and database status is OK on each node.

4. If the node you promoted was any other than node 1, reboot this node and wait untilits database status is OK.

6.7 Operational CautionThe cluster will now continuously respond to requests, synchronize the data, and evaluatethe health of the cluster to ensure availability on one hand, but also data integrity on theother hand. As described earlier, a node will rather stop working than to risk a split brainsituation. A split brain situation develops when two nodes believe they are lone survivors,continue to serve requests, causing two different database sets.

50 (56)



To prevent accidental degradation of the cluster health, some precautions need to betaken. A planned network reconfiguration could be mistaken to be an emergency by thecluster, for example.

Maintenance operations on the cluster such as rebooting, updating, network reconfigu-ration, ... should be restricted to only one node at a time, with ample time for the node toreconnect and synchronize after the task is completed. Before you proceed to the next node,make sure that your cluster is back to full health.

Use-Case: Changing the IP Address of the Application Interface of a node in a threenode cluster

In a PKI Appliance cluster, the internal communication is being transferred over the Appli-cation Interface. Hence, if you need to change the IP address of the Application Interface,cluster communication will fail at first and you will have to take some manual configurationsteps to bring back the node into play:

1. Before starting any configuration changes on a cluster node, it is good practice toassert that the node has been running fine up to now. This is the only way to knowfor sure whether you actually broke anything if the procedure does not succeed asexpected.

2. You might also want to make a last manual backup of the PKI Appliance.

3. We’ll assume here that you have announced this cluster node as being not operational(e.g. disabled in a frontend load balancer) for the time of the change.

4. Now start the actual change by changing the Application Interface IP address on thecluster node in WebConf, see chapter ?? ?? on page ??.

5. Navigate your browser to the Cluster tab of the WebConf on all of the other clusternodes.

6. Wait for the cluster node to appear offline/not connected in the cluster connectionstable, the IP address should now be in an editable input field.

7. On every of the other cluster nodes, correct the application IP address of the clusternode in the cluster table.

8. Confirm the operation by hitting Apply . It could be that you have to wait a coupleof seconds before you are allowed to click that button.

9. After the cluster reconfiguration has finished, all cluster nodes should be connected toall of the other cluster nodes.

10. When everything works as expected, you should not forget to bring back the node intothe load balancer.

51 (56)



Replacing a failed cluster node

To replace a failed cluster node, follow the same procedure as you would for adding thecluster node for the first time. See chapter 6.4 Use-Case: Extending a cluster from n to n+1nodes on page 46 for more detailed information. Restoring the node from a backup will notwork because the database content in the backup file will be outdated.

52 (56)


7. PKCS#11 SLOT SMART CARD ACTIVATION Ver: 2.7.2

Chapter 7

PKCS#11 Slot Smart CardActivation

7.1 IntroductionAll sensitive cryptographic material of the PKI Appliance is stored on a Hardware Secu-rity Module (HSM). This HSM protects your key material against physical attacks. Thekeys required by the PKI Appliance and your infrastructure are organized in so-called slots,commonly used with the cryptographic API PKCS#11. To operate on these keys, theseslots must be activated with some authentication code. Depending on your requirementsfor availability, usability and security, you can select whether those authentication codesshould be stored on the PKI Appliance or not. This can be chosen per slot. Slots withstored authentication codes can be auto-activated for immediate availability. The generatedand automatically stored authentication codes are of very high quality. This choice can bechanged even later during the operation of the PKI Appliance.If even manually entered authentication codes do not meet the security requirements, thereis an option for a two-factor authorization: It is possible to additionally require an activationwith smart cards for one or more slots. This choice has to be done during installation.

7.2 Installation/ConfigurationPKCS#11 slot smart card activation can be enabled per slot but only during the installationof the PKI Appliance. To do so, untick (Automatically generated) AuthenticationCode for the slot you want to give more security. You will then be given the possibility to tickSmart card activated for that slot. Then you will see some more options available for thegeneral slot smart card activation settings. You still have to define an authentication codeper slot. You can either chose something trivial like 1234 since you are relying to externalsecrets anyways, or you can make it even more secure by defining a real secret authenticationcode which will be required additionally upon activation.

53 (56)



7.2.1 "Number of users required"

It can be chosen how many smart cards should be required to activate a slot. This way avery important application can be secured even further. However, there is no quorum (like"3 out of 5") available. If Number of users required: 5 has been chosen, then 5 differentuser credentials will be generated and written to 5 different smart cards, all of which needto be present when activating a slot. The default setting of the PKI Appliance is to createonly one user credential to be required.

7.2.2 "Number/copies of user smart cards"

! Unlike the backup key share on the smart cards, the user credentials can notbe copied from card to card. A lost, broken or blocked smart card can notbe replaced. Therefore the PKI Appliance offers to create sufficient copies,once and for all.

The default setting of the PKI Appliance is to create 2 smart cards with the same usercredential.

7.2.3 "Require smart cards to activate system after boot"

For highest security concerns, smart card activation can also be enabled for PKCS#11 slot0, which contains the key that is used to sign the audit log. Since EJBCA produces an auditlog entry for every single action, it needs access to slot 0 for every single action, includingstart-up. This effectively means that EJBCA will not be reachable after a system startupunless slot 0 has been successfully activated by smart card.

7.2.4 Procedure

For every slot activation user that has been chosen, the following procedure will first runduring the installation:

• The user credentials are generated in memory.

• For every copy that has been chosen, the user credentials will be written to a smartcard. It is required to enter the PIN (default PIN on delivery: 123456) and acknowledgewith "OK".

• The user credentials (only public key) are read into the HSM, it will only be requiredto press the OK button.

After the installation, it is strongly advised to change the PINs of the smart cards throughthe WebConf.

54 (56)



7.2.4.1 Example with default values

The procedure with an PKI Appliance Security Level of "2 out of 3" and slot smart cardactivation on slot 7 with default values 1 user and 2 copies will look like this:

• Backup key shares handling

– One audible alert (bee-beep)– Generation of the backup key and writing to three cards (with PIN and OK)– Reading of the backup key from two cards (with PIN and OK)

• Handling of one slot activation user

– Generation of user credentials– One audible alert (bee-beep)– User credential being written to one card (with PIN and OK)– One audible alert (bee-beep)– User credential being written to one card (with PIN and OK)– One audible alert (bee-beep)– Creation of the user within the HSM by reading the public key, (only OK)

7.2.4.2 Slots 0 and 1

If the installation is configured to have smart card activation on slot 0 and slot 1 (ManagementCA) Require smart cards to activate system after boot the installation procedure will beextended by more PIN pad operations since the installer needs access to these slots to createthe keys needed for operation, audit log signature and Management CA respectively.These extensions will be activation procedures as described in the next section.

7.3 Application/Activation of a slotWhenever the application will attempt a "Login" to the slot (as when activating a Crypto-Token in EJBCA), the PKI Appliance will automatically and immediately request the smartcard(s) to be inserted to the PIN pad. This can be noticed by a small audible alert (bee-beep). The PKI Appliance physical front display will give a short hint at which slot is beingactivated and user card is required to be inserted.

! The user cards will always be required in ascending order, always startingwith User 1.

Whenever some PKCS#11 slot activation with smart card goes wrong, the internal PKIAppliance mechanism will restart all applications, which in turn requires that all slots needto be activated again.

55 (56)



7.3.1 Activation on boot/slot 0

If Require smart cards to activate system after boot has been chosen during installation, onevery system start/boot, the PKI Appliance will first require the successful activation of slot0 before it can continue with start up. Smart card and PIN have to be entered within onehour after system start.

56 (56)

pki appliance installation guide

Documents