helm operations manual - yorkshire & humber care record€¦ · 2fa a web presentation tier...

Helm Operations Manual

Synanetics Confidential

Contents

0 Document Control ........................................................................................................................... 5

0.1 Version History ........................................................................................................................ 5

0.2 References .............................................................................................................................. 5

0.3 Glossary ................................................................................................................................... 6

1 Helm Background ............................................................................................................................ 8

1.1 Document Objectives .............................................................................................................. 9

2 Helm Solution Overview and Description ..................................................................................... 10

2.1 Schematics ............................................................................................................................ 10

2.1.1 Helm Overview .............................................................................................................. 10

2.1.2 Logical Technology Model............................................................................................. 11

2.2 Helm Components ................................................................................................................ 12

2.2.1 Network Flow Between Components ........................................................................... 12

2.2.2 Persistence Tier ............................................................................................................. 13

2.2.3 EtherCIS ......................................................................................................................... 13

2.2.4 OpenEHR Service ........................................................................................................... 14

2.2.5 QEWD-Courier ............................................................................................................... 14

2.2.6 PulseTile ........................................................................................................................ 20

2.3 External APIs used by Helm .................................................................................................. 21

2.3.1 NHS Login (Citizen Identity Service) .............................................................................. 21

2.3.2 YHCR (Yorkshire Health Care Record) ........................................................................... 21

2.4 Security, Users and Roles ...................................................................................................... 22

2.4.1 User Access Control ...................................................................................................... 22

2.4.2 Helm User Roles ............................................................................................................ 22

2.4.3 Helm User Login ............................................................................................................ 22

2.4.4 Support Users ................................................................................................................ 23

2.5 System Maintenance ............................................................................................................ 24

2.5.1 About ............................................................................................................................. 24

2.5.2 System Backup .............................................................................................................. 25

2.5.3 System Restore ............................................................................................................. 25

2.5.4 Operating System Updates and Patches ....................................................................... 26

2.5.5 System Upgrade Procedures ......................................................................................... 26

2.5.6 Configuration Management .......................................................................................... 26

2.5.7 SSL Certificate replacement .......................................................................................... 26

2.5.8 Graylog .......................................................................................................................... 26


2.6 Application Maintenance ...................................................................................................... 27

2.6.1 Snapshots ...................................................................................................................... 27

2.6.2 Deploying New Qewd Service/Updating Existing Qewd Service .................................. 28

2.6.3 Adding New Institutions to Helm .................................................................................. 28

2.6.4 Updating PulseTile UI .................................................................................................... 28

2.6.5 Debugging Qewd Issues ................................................................................................ 28

2.6.6 EtherCIS Updates .......................................................................................................... 28

2.6.7 Database Administration .............................................................................................. 29

2.6.8 Application Log Maintenance ....................................................................................... 29

2.7 Monitoring and Alerting........................................................................................................ 30

2.7.1 About ............................................................................................................................. 30

2.7.2 Monitoring and Alerting Infrastructure ........................................................................ 30

2.7.3 Monitored Components................................................................................................ 30

2.7.4 Component State .......................................................................................................... 30

2.7.5 Infrastructure Alerts ...................................................................................................... 30

2.7.6 Supporting Software Alerts ........................................................................................... 30

2.7.7 Application Alerts .......................................................................................................... 30

2.7.8 Monitor Dashboards ..................................................................................................... 31

2.7.9 Alert Consumers ............................................................................................................ 31

2.8 Operational Procedures ........................................................................................................ 32

2.8.1 About ............................................................................................................................. 32

2.8.2 System Component Restart .......................................................................................... 32

2.8.3 Helm UI Overview ......................................................................................................... 35

2.9 Business Continuity / Disaster Recovery ............................................................................... 47

2.9.1 Scenarios ....................................................................................................................... 47

2.9.2 Procedures .................................................................................................................... 47

2.10 Software ................................................................................................................................ 48

2.10.1 Software Repositories ................................................................................................... 48

2.10.2 Key Configuration .......................................................................................................... 48

2.11 Helm Servers ......................................................................................................................... 50

2.11.1 Helm-Admin (helmprod-admin, helmstaging-admin)................................................... 50

2.11.2 Helm-HSCN-Proxy (helmprod-hscn-proxy, helmstaging-hscn-proxy) ........................... 51

2.11.3 Helm-DMZ (helmprod-dmz, helmstaging-dmz) ............................................................ 52

2.11.4 Helm-QEWD (helmprod-qewd, helmstaging-qewd) ..................................................... 53

2.11.5 Helm-EtherCIS (helmprod-ecis, helmstaging-ecis) ....................................................... 54


2.11.6 Helm-Logging (helmprod-logging, helmstaging-logging) .............................................. 55

2.11.7 Helm-Db (helmprod-db, helmstaging-db) ..................................................................... 56

2.12 Support Knowledgebase ....................................................................................................... 57

2.12.1 Known Issues ................................................................................................................. 57

2.13 Appendix ............................................................................................................................... 58

2.13.1 Application Component Schematics provided by the Ripple Foundation .................... 58

2.13.2 UK Cloud handover recommendations ......................................................................... 63

2.13.3 Network Topology on UKCloud ..................................................................................... 65


0 Document Control

0.1 Version History

Date Author Comments

13/06/2019 Richard Brown, Ian Core V1.1. Some sections need further work, areas that have been highlighted as ‘Placeholder’

0.2 References

Reference Comments

Helm _Ripple Stack Handover Material -Repos_Directories_MSStartupNotes_2019May02.pdf

Supplied by the Ripple Foundation

UKCloud Solution Handover Leeds CC.pdf Supplied by UKCloud


0.3 Glossary

Term Explanation

Helm Person Held Record Solution

NHS National Health Service

YHCR Yorkshire U Humber Health Care Record

PulseTile Open Source Modular UX/UI framework for the Helm presentation tier

QEWD Open Source NodeJS framework

EtherCIS Open Source Clinical Data Repository

PostgreSQL PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and technical standards compliance. It is designed to handle a range of workloads, from single machines to data warehouses or Web services with many concurrent users.

YottaDB YottaDB is a Multi-Language NoSQL database engine. It is currently in production at some of the largest real-time core banking applications and electronic health record deployments.

React React is a JavaScript library for building user interfaces. It is maintained by Facebook and a community of individual developers and companies. React can be used as a base in the development of single-page or mobile applications, as it is optimal for fetching rapidly changing data that needs to be recorded.

Docker Coupled software-as-a-service and platform-as-a-service products that use operating-system-level virtualization to develop and deliver software in packages called containers

Container Lightweight discrete software packages that run under the Docker engine

Microservice Microservices are a software development technique—a variant of the service-oriented architecture architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services are fine-grained and the protocols are lightweight.

openEHR Technology for e-health, consisting of open specifications, clinical models and software that can be used to create standards, and build information and interoperability solutions for healthcare.

FHIR Fast Healthcare Interoperability Resources. HL7® Fast Healthcare Interoperability Resources is a next generation standards framework that leverages the latest web standards and applies a tight focus on implementation and was developed by healthcare standards developing organization, Health Level Seven International

HSCN Health and Social Care Network. The Health and Social Care Network (HSCN) provides a reliable, efficient and flexible way for health and care organisations to access and exchange electronic information.


LTHT Leeds Teaching Hospitals NHS Trust. One of the largest and busiest acute hospital trusts in the UK

DMZ DMZ (demilitarized zone), also sometimes known as a perimeter network or a screened subnetwork, is a physical or logical subnet that separates an internal local area network (LAN) from other untrusted networks, usually the internet.

SSH Secure Shell

OIDC OpenID Connect is a simple identity layer on top of the OAuth 2.0 protocol, which allows computing clients to verify the identity of an end-user based on the authentication performed by an authorization server, as well as to obtain basic profile information about the end-user in an interoperable and REST-like manner. In technical terms, OpenID Connect specifies a RESTful HTTP API, using JSON as a data format. OpenID Connect allows a range of clients, including Web-based, mobile, and JavaScript clients, to request and receive information about authenticated sessions and end-users. The specification suite is extensible, supporting optional features such as encryption of identity data, discovery of OpenID Providers, and session management.

API Application Interface

NGINX ‘Engine X’ Nginx is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache.

https://en.wikipedia.org/wiki/OAuth#OAuth_2.0

https://en.wikipedia.org/wiki/OAuth#OAuth_2.0

https://en.wikipedia.org/wiki/Client_(computing)

https://en.wikipedia.org/wiki/REST

https://en.wikipedia.org/wiki/RESTful

https://en.wikipedia.org/wiki/HTTP

https://en.wikipedia.org/wiki/API

https://en.wikipedia.org/wiki/JSON

https://en.wikipedia.org/wiki/JavaScript_engine

https://en.wikipedia.org/wiki/Extensibility


1 Helm Background

Helm is a Person Held Record aiming to help people to better manage and control their own care and wellbeing, and help to prevent further health issues in the following ways:

It is an innovative platform for viewing, controlling and contributing to their own health and

wellbeing information.

It will be the first time the public have been able to have access to a system that joins up all

their medical information from the NHS with council services and more, with one single

login.

Citizens will have the ability to add information they believe and/or their care professional

has asked them to record to help manage a health condition and improve their wellbeing.

It will revolutionise access to personal data, starting with medical information and growing

from there.

Easy to use; interactive; ability to interface with apps and devices; personalise-able and with

granular privacy settings; secure; supported.

It is built on an open platform with a set of principles, which include open standards, and is

vendor and technology neutral. Allowing it to scale and expand both geographically and in

terms of connectivity.

It will encourage a marketplace to create apps and functionality that will work with the open

platform because they conform to the standards set. Opening the health and care system to

SMEs that struggle to enter and innovate in this sector.

This could in time allow citizens to be the conduit of the information they can share, which

allows researchers, universities etc fantastic opportunities to access currently untapped

data.

In the future, it will become the hub for all public services that rely on citizens to contribute

to and manage data and information.


1.1 Document Objectives

This document is a reference for technical support staff responsible for the day to day running of the Helm solution.


2 Helm Solution Overview and Description

2.1 Schematics

2.1.1 Helm Overview

The Helm platform, on a logical viewpoint consists in the following in scope units:

An authentication unit managing patients’ credentials and deals with authentication using

2FA

A web presentation tier supporting the display, formatting and acquisition of patient data

An integration tier that capture data from external sources and format it to further persist it

An openEHR (Technology for e-health, consisting of open specifications, clinical models and

software that can be used to create standards, and build information and interoperability

solutions for healthcare.) data persistence unit maintaining clinical data in compliance to

openEHR standards

An admin interface/gateway enabling administrators to perform tasks using SSH (Secure

Shell Protocol) and a limited Web interface to configure and interact with the log server

A logging unit that capture logs from the platform active units, consolidate the records and

provide a querying interface.


2.1.2 Logical Technology Model

Logical Schematic showing both Production and Staging environments


2.2 Helm Components

2.2.1 Network Flow Between Components

System components are deployed internally as depicted above

● The data flow (in green) is essentially using HTTP (Hyper Text Transport Protocol) and JDBC

(Java Database Connectivity - for the DB transactions).

● The admin flow (in red) allows to access SSH consoles of the various instances. Access is

granted after a client X.509 certificate authentication is successful. The log server

administration is performed via a Web console (hence the http proxy).

● The log traffic (in blue) is funnelled to the log server via a specific collector protocol (graylog

beat)

All network traffic with external network (either public or HSCN (Health and Social Care Network)) is relayed by proxies acting as DMZ. No direct access to any components (proxies excepted) from an external network is possible.


2.2.2 Persistence Tier

2.2.2.1 PostgreSQL - Holds the persistence tier, namely a postgreSQL database (v10+) for use with EtherCIS

(see here), this holds clinical data (e.g. top 3 things). The approach here is based on a

data model mixing both relational and document (json – JavaScript Object Notation)

data. Currently, this unit consists of a single database server instance

2.2.3 EtherCIS - Holds EtherCIS REST API to store clinical data to the persistence tier (i.e. top 3 things).


2.2.4 OpenEHR Service

2.2.4.1 About - For a definition of OpenEHR see here.

- Service for interaction between Qewd infrastructure and EtherCIS.

- Allows patient to access and edit their top 3 things.

- Allows institutions (e.g. LTHT, Leeds Teaching Hospital Trust) to pull data about patient’s top 3

things.

Service location: - /home/centos/qewd-helm/helm/openehr_service

Replicated: No Ports: - 8003

Configuration location: - /home/centos/qewd-helm/helm/openehr_service/configuration

Notes: - The client_id and client_secret for each client in the oidc-provider must be copied to the

openehr.sites section of the global_config.json file so that institutions can access patient data via

the /hscn/ route.

- The openEHR service makes a request to check the token of institutions, which is different to

how PHR user access is managed.

2.2.5 QEWD-Courier

2.2.5.1 About

For a definition of QEWD (see here) The QEWD courier implementation consists of several microservices running under the Docker engine Qewd Courier setup is a mixture of both statically defined routes (i.e. configured directly in the /configuration folder) and routes imported from microservices in import mode. Note that each time you add a static route the orchestrator will need to be restarted. Note that each time a new route is added to a service that exports its routes both the orchestrator and the service the route has been added will need to have their import status reset (see later in this section). When the routes are imported into the orchestrator both the orchestrator and the service the routes are imported from will terminate on completion and need to be brought back up.

2.2.5.2 General Information


The Start-up script for all docker containers is located at /home/centos/qewd-helm/start.sh

2.2.5.3 Orchestrator

What is it for: - Provides gateway into Qewd Microservices network

Replicated: No Service location: - /home/centos/qewd-helm/helm/orchestrator

Configuration location: - Services configuration - /home/centos/qewd-helm/helm/configuration/config.json

- Route configuration - /home/centos/qewd-helm/helm/configuration/routes.json

Ports: - 8000

Notes: - Orchestrator logs all requests into Qewd environment into YottaDB (see here). Is it

possible to access these logs?

- All Qewd instances have an administration panel accessible via /qewd-monitor


2.2.5.4 Initialisation Service The Initialisation Service is a conductor service that handles certain ordered operations that must be conducted when a user first logs into Helm. These include: - Looking up the currently configured Policies for Helm and checking if the user has provided

consent to all policies. - Returning Policies to sign in the case the user has not yet accepted the terms. - Passing signed consents to the fhir_service once the user has accepted them. - Initialising the user in EtherCIS. Ports: - 8006 Replicated: No Configuration location: - Services configuration - /home/centos/qewd-helm/helm/configuration/config.json - Route configuration - /home/centos/qewd-helm/helm/configuration/routes.json - Service configuration - /home/centos/qewd-helm/helm/configuration/global_config.json - Notes: The Policies served by the initialisation service are configured in the global_config.json file in the initialisation_service.policies property.


2.2.5.5 FHIR service What is it for: - Handles requests to the System of Systems (YHCR Yorkshire Health Care Record, see here)

for patient demographics data and converting to PulseTile format - For a brief explanation of FHIR see here. - Handles retrieving Policy documents for Helm and the Consents provided by users so they

may access the system.

Replicated: No Configuration location: - Service Qewd configuration - /home/centos/qewd-

helm/fhir_service/configuration/config.json

- Service internal configuration - /home/centos/qewd-

helm/fhir_service/configuration/fhir_service.config.json

- Search configuration - /home/centos/qewd-

helm/fhir_service/configuration/fhir_service.search.json

- Routes - /home/centos/qewd-helm/fhir_service/configuration/routes.json

Ports: - 8085

Notes: - The fhir_service is configured as a standalone Qewd microservice, its routes are

imported by the Orchestrator on first run. On completion of import both the

orchestrator and fhir_service will require a restart.

- In order for the import mechanism to work, the Docker container must be started with

environment variable mode=”microservice”

- In order to authenticate with the YCHR servers the service must generate a token for

each user logging into Helm, a private key must be installed at /home/centos/qewd-

helm/fhir_service/configuration/privateKey.key, YHCR will have a corresponding public

key generated by a Certificate Signing Request.

- The grant_type must be configured to "urn:ietf:params:oauth:grant-type:jwt-bearer" in

/home/centos/qewd-helm/fhir_service/configuration/fhir_service.config.json.

- The client_id and client_secret provided for the YHCR must be provided in

/home/centos/qewd-helm/fhir_service/configuration/fhir_service.config.json.

- The rejectUnauthorized property in /home/centos/qewd-

helm/fhir_service/configuration/fhir_service.config.json can be used to allow the system

to access servers with self-signed certificates (when set to false) for staging/test

environments. For production purposes this property should be set to true, or omitted

completely.

- The /home/centos/qewd-helm/fhir_service/configuration/fhir_service.search.json file

can be used to alter search requests to the YHCR system, allowing for differences in how

some FHIR searches.


2.2.5.6 OIDC Client

What is it for: - Handles login for PHR users (via NHS login)

- For a brief explanation of OIDC see here.

- Acts as an authentication service for Helm services

Replicated: No Ports: - 8001

Service folder: /home/centos/qewd-helm/oidc-client Configuration location: - /home/centos/qewd-helm/oidc-client/configuration

Notes: - Login provided by NHS login, configuration for login mechanism is located in

/home/centos/qewd-helm/oidc-client/configuration/oidc.json

- A private key is required for token signing for NHS login which is configured by the

private_key_file_path property.

- The oidc client is a standalone microservice that is not coupled to the rest of the Helm

microservices directly

- Service requires routes to be imported on first starting up the service (same as FHIR

service).

- While the Oidc Client serves as the auth_service for the Helm infrastructure, there is an

auth_service folder within /home/centos/qewd-helm/helm:

“This folder exists to provide the onOrchResponse event hook for the /api/initialise

route / API

The reason is because the Authentication MicroService (auth_service) is now provided

by the separate, standalone "oidc-client" MicroService, whose startup knows nothing

about the Orchestrator.

So we need to add this stubbed folder path into the Orchestrator so that it adds the

onOrchResponse to itself so it gets triggered correctly”


2.2.5.7 OIDC Provider

What is it for: - Allows access to Helm Patient data by institutions through HSCN (e.g. LTHT).

Service folder: /home/centos/qewd-helm/oidc-provider Ports: - 8080

Configuration location: - /home/centos/qewd-helm/oidc-provider/configuration

Notes: - Service is used only so institutions (e.g. LTHT) can acquire an access token to access

openehr_service.

- Clients are configured in the /home/centos/qewd-helm/oidc-

provider/configuration/data.json file.


2.2.6 PulseTile

2.2.6.1 About PulseTile is a user interface and usability framework for the integrated digital care record platform for EHR management.

An electronic health record (EHR), or electronic medical record (EMR), refers to the systematized collection of patient and population electronically stored health information in a digital format. These records can be shared across different health care settings. Records are shared through network-connected, enterprise-wide information systems or other information networks and exchanges. EHRs may include a range of data, including demographics, medical history, medication and allergies, immunization status, laboratory test results, radiology images, vital signs, personal statistics like age and weight, and billing information.

2.2.6.2 Technology Stack

2.2.6.2.1 React 15.6.2 React is a front-end library. It is used for handling the view layer for web and mobile apps. ReactJS enables the creation of reusable UI components. It is currently one of the most popular JavaScript libraries and has a strong foundation and large community behind it.

2.2.6.2.2 Redux Redux is a predictable state container for JavaScript apps. Its purpose is to enable the writing applications that behave consistently and run in different environments.

Service location:

- /usr/share/nginx/html

Replicated: No

Ports: 80 Configuration location:

- There is no configuration for the PulseTile installation.

Notes: - The PulseTile project can be built from the source repository (using npm run build from the

root directory of the project).


2.3 External APIs used by Helm

2.3.1 NHS Login (Citizen Identity Service)

NHS Login (Citizen Identity Service) Overview

Once authenticated in NHS Digital Citizen ID, a Citizen ID id_token is generated. This token is added as a claim into a JWT (JSON web token) created by QEWD authentication microservice. The created JWT holds various details on the current user including its NHS id, defined role etc. The JWT is then used to verify and use the credentials in other services. JWT verification is using a shared secret . This key is propagated to the services implementing token verification.

2.3.2 YHCR (Yorkshire Health Care Record) The YHCR (System of Systems) is a solution that connects multiple regional systems for data aggregation so it can be presented in a FHIR format. Helm links into the YHCR to retrieve demographics information and transform this into a format that can be use by the PulseTile technology ready for display in the UI.


2.4 Security, Users and Roles

2.4.1 User Access Control For a glossary of terms, see here.

All access to the system for users is under the control of NHS Digital Citizen ID . This solution makes it possible to implement an authentication access using the logon ID/password created during a user’s registration process which is subject to identity verification. This service provides legitimate client with an OAuth2 / OpenID Connect framework. The details of the implementation of the NHS Digital Citizen ID are considered out of scope for the purposes of this document.

2.4.2 Helm User Roles

Currently one role available in Helm, the PHR (Person Held Record) user. PHR User - A public health record user, a member of the public may access the system and retrieve

information about their health record.

- They may not view details of the health records of other users.

- They may view/edit/delete their top 3 things.

Institutions (e.g. LTHT) can access patient data (top 3 things) via REST request.

2.4.3 Helm User Login

For PHR User Role, user may login via NHS login page, token will be supplied by NHS login page on redirect back to Helm and is stored as a claim in the user’s Qewd Session Jwt. - No onboarding of new users necessary as their credentials are managed by NHS login.

- A user without a token is either taken directly to the NHS login page or is redirected

after landing at Helm.

- User logs in via NHS login and is redirected back to Helm with their NHS login token to

/auth/token endpoint of the oidc_client service.

- Oidc client extracts some information from the token and builds a token for use with

Helm/Qewd.

- Patient is redirected back to PulseTile home page.

- Details of logged in patient are checked against openEHR (if the patient doesn’t exist in

openEHR they are created at this point.

- A request is made to get patient demographics from fhir_service.

- Patient should be able to view their home page, with any top 3 things available and their

demographics visible.

Institutions such as LTHT can login to Helm through the HSCN network via the oidc-provider service. - Institution sends client_id & client_secret to oidc-provider service within Helm in

exchange for an access token (REST request).


- Institution then makes request for patient data (top 3 things) via

/api/hscn/:site/top3Things/:patientId, providing the access token they acquired in the

previous step.

2.4.4 Support Users

At this time support staff use a single login ‘centOS’ via SSH using public keys to access the

servers for support purposes. This needs to be further discussed as there is a potential audit

shortfall with this approach.


2.5 System Maintenance

2.5.1 About

2.5.1.1 System Software Installed

Name Role Software Installed[]

helmprod-admin Admin server Nginx 1.12.2

helmprod-db Database Server PostgreSQL 10.5

helmprod-dmz Bastion Nginx 1.12.2

helmprod-ecis ECIS Server Java 1.8 (ecis) ethercis-1.30

helmprod-hscn-proxy HSCN Proxy Squid 3.5.20 Nginx 1.12.2

helmprod-logging Log Server ElasticSearch 5.6.12 GrayLog 2.4 Java 1.8

helmprod-qewd QEWD Server Docker CE 18.06. QEWD

helmstaging-admin Admin server Nginx 1.12.2

helmstaging-db Database Server PostgreSQL 10.5

helmstaging-dmz Bastion Nginx 1.12.2

helmstaging-ecis ECIS Server Java 1.8 (ecis) ethercis-1.30

helmstaging-hscn-proxy HSCN Proxy Squid 3.5.20 Nginx 1.12.2

helmstaging-logging Log Server ElasticSearch 5.6.12 GrayLog 2.4 Java 1.8

helmstaging-qewd QEWD Server Docker CE 18.06. QEWD

myhelm-myhelm Web Server PHP 5.4.16 Apache 2.4.6 MariaDB 10.3.9

Software installed per server type


2.5.2 System Backup

2.5.2.1 Application components

2.5.2.1.1 Approach The approach for backing up application components would be to perform full server virtual machine snapshots – once a week. In addition to the above, there will be a procedure necessary for performing ad-hoc backups for the purposes of upgrade planning and execution. The above approach needs to be discussed and finalised with UKCloud. After this the full

set of procedures can be documented.

2.5.2.2 Database components

2.5.2.2.1 postgreSQL The backup mechanism for the postgreSQL database is determined by the requirements of the solution due to the flexibility postgreSQL provides. For example, if a point in time restore of the database is required then there would need to be a regular backup schedule in conjunction with WAL (write ahead logs). What are the solution requirements for backing up and restoring the postgreSQL

database? Eg. Point in time, restore to last backup.

2.5.2.2.2 YottaDb The purpose of the YottaDB is to act as a caching mechanism for the qewd components. There is no requirement to back this up on a regular basis, should there be a problem with it, the database files can be replaced as new with the versions in the git source code repository.

2.5.2.3 Backup Procedures In order to document the backup procedures, there is a dependency on the finalised

approach see postgreSQL section above.

2.5.2.4 Backup Location In order to document the backup location, there is a dependency on the finalised


2.5.2.5 Backup Schedule In order to document the backup schedule, there is a dependency on the finalised


2.5.3 System Restore

2.5.3.1 Restoring postgreSQL In order to document the restore procedures, there is a dependency on the finalised


2.5.3.2 Restoring the application The approach for restoring application components would be to perform a restore of the full server virtual machine snapshots.


The above approach needs to be discussed and finalised with UKCloud. After this the full

set of procedures can be documented.

2.5.4 Operating System Updates and Patches Synanetics are responsible for ensuring that the underlying operating system is up to

date with respect to updates and patching.

The formal policy for doing this needs to be agreed as the standard questions such as

where these will be tested, what will be tested (eg. Full set of regression tests) are yet to

be finalised.

2.5.5 System Upgrade Procedures

2.5.5.1 Approach The formal policy for upgrading the system needs to be agreed as several questions are

currently outstanding, for example;

Backup pre and post upgrade, what will be backed up (everything or just changed

components)

Route to production through available environments for example; Development ->

Staging -> Production

Post implementation testing

2.5.6 Configuration Management Configuration is currently held in public repositories which does present a risk if

someone commits changes into them. Synanetics policy for configuration management

for this solution needs to be finalised.

Configuration files, Keys and Certificates (eg. YHCR, NHSLogin)

Is there an option to use a UKCloud function to this end?

2.5.7 SSL Certificate replacement The formal policy for SSL certificate replacement needs to be agreed as several

questions are currently outstanding, for example;

Servers in scope for certificate replacement; dmz, myHelm, admin servers

2.5.8 Graylog The formal policy for Graylog upgrades needs to be finalised


2.6 Application Maintenance

2.6.1 Snapshots Helm-DMZ: - Snapshot before PulseTile deployment (for rollback purposes)

- Snapshot after PulseTile deployment and testing (disaster recovery)

Helm-Proxy: - Snapshot before configuration change (for rollback purposes)

- Snapshot after configuration change (disaster recovery)

Helm-Db: - Regular (daily/weekly) db backup then snapshot.

- Db backup and snapshot before deployments.

Helm-Qewd: - Snapshot before code deployment (for rollback purposes)

- Snapshot after code deployment (disaster recovery)

- Backups of Orchestrator YottaDB and snapshots periodically

Helm-EtherCIS: - Snapshot before updating EtherCIS (for rollback purposes)

- Snapshot after updating EtherCIS (disaster recovery)


2.6.2 Deploying New Qewd Service/Updating Existing Qewd Service

Setup maintenance page

Set proxy and DMZ not to route traffic into system

The details of the above need to be finalised

Backups and snapshots of Helm-Qewd and Helm-DB

Stop Qewd services

Deploy code and configure

Bring back up Qewd Orchestrator for route imports

Testing of updated system

Revert to previous snapshot/code + config version on deployment failure

Route traffic back into helm system

Remove maintenance page

2.6.3 Adding New Institutions to Helm Placeholder for instructions for this from a Synanetics perspective

2.6.4 Updating PulseTile UI Setup maintenance page

Set DMZ not to route traffic into system


Backups and snapshots of Helm-Qewd and Helm-DB

Deploy code

Testing of updated system

Revert to previous snapshot/code version on deployment failure

Route DMZ traffic back into helm system

Remove maintenance page

2.6.5 Debugging Qewd Issues Reenable access to qewd-monitor via nginx. Interface into yottaDB

Accessing logs


2.6.6 EtherCIS Updates EtherCIS deploy and scripts


2.6.7 Database Administration This console is subject to the security provisions described in STIGS (Security Technical Implementation Guides) PostgreSQL 9.x Security Technical Implementation Guide . The DB admin console is provided by psql which supports

SSL

Certificate based authentication

defining roles that provide a strict distinction between operations that affect the

information structure and those that allow access to the data stored, the latter being

prohibited to non-medical personnel.

The DB allows a strict isolation of the resources

System resources allowed in principle to operators

Database resources reserved for application access and, in case of emergency, to medical

staff with computer skills.

Service resources to perform connection setup, operation, and backup operations.

Application resources to perform console access controls and other application permission

definitions.

PostgreSQL administration console is accessible only through the SSH admin host and subject to strong multi-factors authentication.

2.6.8 Application Log Maintenance

The fine details of log maintenance of the solution are yet to be fully determined. This

section of the document acts therefore as a placeholder.


2.7 Monitoring and Alerting

2.7.1 About

The fine details of how the monitoring and alerting of the solution works are yet to be fully

determined. This section of the document acts as a placeholder. Where shortfalls exist,

there will be specific questions documented designed to resolve the shortfall.

2.7.2 Monitoring and Alerting Infrastructure The logging server has been provisioned for the purposes of monitoring and alerting. The software provisioned for this purpose is known as ‘Graylog’ Graylog is highly flexible with many configurable items and methods of processing and presenting information.

The configuration of Graylog for this purpose is not yet finalised so this section acts as a

placeholder

Graylog DNS entry so that dashboards can be exposed over the internet. An alternative

means by which to expose this to support staff is required.

2.7.3 Monitored Components

Placeholder; Each logical component of the Helm solution and their respective artefacts

should be incorporated into the monitoring solution where possible.

2.7.4 Component State

2.7.4.1 Monitor Portal for each qewd component Placeholder

2.7.4.2 Docker Logs Placeholder

2.7.4.3 Java monitor for EtherCIS Placeholder

2.7.5 Infrastructure Alerts Placeholder; Each logical infrastructure component of the Helm solution and their respective

artefacts should be incorporated into the monitoring solution where possible.

2.7.6 Supporting Software Alerts Placeholder; Where the supporting software provides instrumentation that is useful for

monitoring then this is the place to document it.

2.7.7 Application Alerts Placeholder; Each logical component of the Helm solution and their respective artefacts

should be incorporated into the monitoring solution where possible.


2.7.8 Monitor Dashboards Graylog provides a rich feature set for development of dashboards and visualisations to

illustrate system state and highlight potential problems and / or threats. Development of

these dashboards would be necessary and a means by which to expose this to support staff

is required.

2.7.9 Alert Consumers Graylog provides a rich feature set for the purposes of escalating alerts. The policies,

procedures and alert specifics need to be finalised.


2.8 Operational Procedures

2.8.1 About

2.8.2 System Component Restart

2.8.2.1 Restart postgreSQL Restart PostgreSQL using the following commands: SSH to the relevant Helm-DB server (staging/production) via the admin server.

Check which PostgreSQL service is running using:

systemctl list-units|grep postgresql This should show something similar to: postgresql-10.service You may then use: sudo service postgresql-10.service reload To restart the PostgreSQL service.

2.8.2.2 Restart EtherCIS Restart EtherCIS using the following commands: SSH to the relevant Helm-Ecis server (staging/production) via the admin server. Execute: sudo -i To enter a root shell and cd into /home/ethercis EtherCIS may be stopped by executing: ./ecis-server stop And started again by executing: ./ecis-server start

2.8.2.3 Restart Docker

Restart Docker using the following commands: SSH to the relevant Helm-Qewd server (staging/production) via the admin server.

Docker may be restarted using the following command: sudo systemctl restart docker


2.8.2.4 Restart Docker Components Restart Docker using the following commands: SSH to the relevant Helm-Qewd server (staging/production) via the admin server. You can list the running containers by executing the following command: sudo docker container ls You should see the following containers running:

- initialisation_service

- auth_service

- fhir_service

- oidc

- orchestrator

- openehr_service

If any of the above containers are not running, they may be brought back online using the following commands: Orchestrator:

sudo docker run -d --name orchestrator --rm --net qewd-net -p 8000:8080 -v

~/qewd-helm/helm:/opt/qewd/mapped -v ~/qewd-

helm/yottadb/orchestrator:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-

server:1.0.0

Auth Service:

sudo docker run -d --name auth_service --rm --net qewd-net -p 8001:8080 -v

~/qewd-helm/oidc-client:/opt/qewd/mapped -e mode="microservice" -v ~/qewd-

helm/yottadb/authentication:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-

server:1.0.0

OpenEHR Service:

sudo docker run -d --name openehr_service --rm --net qewd-net -p 8003:8080

-v ~/qewd-helm/helm:/opt/qewd/mapped -e microservice="openehr_service" -v

~/qewd-helm/yottadb/openehr:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-

server:1.0.0


FHIR Service:

sudo docker run -d --name fhir_service --rm --net qewd-net -p 8005:8080 -v

~/qewd-helm/fhir_service:/opt/qewd/mapped -e mode="microservice" -v ~/qewd-

helm/yottadb/fhir:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-server:1.0.0

Initialisation Service:

sudo docker run -d --name initialisation_service --rm --net qewd-net -p

8006:8080 -e microservice="initialisation_service" -v ~/qewd-

helm/helm:/opt/qewd/mapped rtweed/qewd-server:1.0.0

Oidc Provider:

sudo docker run -d --rm --name oidc -p 8080:8080 -v ~/qewd-helm/oidc-

provider:/opt/qewd/mapped -v ~/qewd-

helm/yottadb/oidc_provider:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-

server:1.0.0

The commands can be found on the Helm-Qewd server in the file /home/centos/qewd-helm/start.sh file.


2.8.3 Helm UI Overview

2.8.3.1 User Interface (UI) The below sections provide an overview of the current functionality provided by Helm

2.8.3.1.1 Terms and Conditions

2.8.3.1.2 Login


2.8.3.1.3 Patient Summary Home Page

The Patient Summary view presents information as above, demographic data in the header bar. The Home section contains the PulseTile entry point into the information categories, in this instance because there is only the ‘Top Three Things’ then this is all that is displayed, the title of each being displayed in the tile. Clicking through to this takes the user to the ‘Top Three Things About Me’ UI.


2.8.3.1.4 User Profile / Logout

2.8.3.1.5 Top Three Things

2.8.3.1.5.1 Summary List View


2.8.3.1.5.2 Detailed View

On entry into the ‘Top Three Things about Me’ tile, further information for each category is expanded. Data can be updated and saved as necessary.


2.8.3.1.6 User Tour / Navigation Guidance


2.8.3.2 Failure Scenarios Generally, the first port of call for investigations into issues with Helm would be the logs created by the various microservices running in the docker containers. Access to the information in these is key to diagnosing issues. These can be accessed via docker using the following commands For individual containers;

sudo docker logs <container_name>

Multiple containers with timestamps;

sudo docker-compose logs -f -t

Attaches current process to the running logs of all running containers. WARNING: This can be verbose! Some of the below scenarios have been unable to be tested, however have been documented for completeness. In cases where containers fail to start repeatedly, it is possible to start the container in interactive mode to view the output before the container exits. To do this start the containers using the “-it” flag:

sudo docker run -it …

The rest of the startup command can be found in the start.sh file outlined in the Restart Docker Components section. It is also possible to increase the log levels emitted by the containers by using the debug environment variable. See overleaf.


For example to enable debug logging for the fhir_service execute the following command:

sudo docker run -d --name fhir_service --rm --net qewd-net -p 8005:8080 -v

~/qewd-helm/fhir_service:/opt/qewd/mapped -e DEBUG=”*” -e

mode="microservice" -v ~/qewd-

helm/yottadb/fhir:/root/.yottadb/r1.24_x86_64/g rtweed/qewd-server:1.0.0

Note the -e DEBUG=”*” environment variable flag, this will increase the verbosity of logging for the container, this can also be used with the “-it” flag to view verbose logs in interactive mode.


2.8.3.2.1 Unable to reach Helm

If the Helm site is unavailable there could be some network related issue or the docker microservices could be stopped. In the first instance, check that the microservices are running using the ‘sudo docker ps’ command. An expected output of this would be;

CONTAINER ID IMAGE COMMAND CREATED

STATUS PORTS NAMES

6fba5bbea683 rtweed/qewd-server "npm start" 4 days

ago Up 17 seconds 0.0.0.0:8091->8080/tcp docker_oidc-

provider_1

6c3b461dc4aa nginx "nginx -g 'daemon of…" 5 days

ago Up 18 seconds 0.0.0.0:80->80/tcp docker_dmz_1

902856aaa521 rtweed/qewd-server "npm start" 5 days

ago Up 18 seconds 0.0.0.0:8080->8080/tcp

902856aaa521_docker_orchestrator_1

e812ddd79c79 rtweed/qewd-server "npm start" 5 days


e812ddd79c79_docker_openehr_service_1

63abf75f5cf6 rtweed/qewd-server "npm start" 5 days


docker_initialisation_service_1

32e43aa273b4 rtweed/qewd-server "npm start" 5 days


32e43aa273b4_docker_fhir_service_1

f0b5953b2338 rtweed/qewd-server "npm start" 5 days

ago Up 17 seconds 8080/tcp

docker_oidc_client_1

If nothing is returned, then bring up the docker microservices using the restart commands outlined in the section Restart Docker Components.


2.8.3.2.2 UI (User Interface) Non-responsive For cases where the User Interface is consistently non-responsive then the docker logs of the various components would be analysed for anomalous conditions. The dmz for example holds the front-end web logs, an example as below;

172.27.0.1 - - [12/Jun/2019:09:04:05 +0000] "GET

/api/patients/null/synopsis/top3Things/latest HTTP/1.1" 200 2987

"http://helm-local.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)

AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

172.27.0.1 - - [12/Jun/2019:09:04:05 +0000] "GET

/api/patients/9999999801/top3Things/ethercis-4efe935c-bf1f-466f-afcf-

9210921245af HTTP/1.1" 304 0 "http://helm-local.com/" "Mozilla/5.0 (Windows

NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/74.0.3729.169 Safari/537.36"

172.27.0.1 - - [12/Jun/2019:09:04:05 +0000] "GET




Chrome/74.0.3729.169 Safari/537.36"

172.27.0.1 - - [12/Jun/2019:09:04:05 +0000] "GET




Chrome/74.0.3729.169 Safari/537.36"

172.27.0.1 - - [12/Jun/2019:09:06:27 +0000] "GET

/api/patients/null/synopsis/top3Things/latest HTTP/1.1" 200 2987



172.27.0.1 - - [12/Jun/2019:09:07:34 +0000] "GET /null HTTP/1.1" 404 555



172.27.0.1 - - [12/Jun/2019:09:07:39 +0000] "GET /null HTTP/1.1" 404 555



Specific things to look for would be the HTTP status codes, the endpoint URI that is returning the status code particularly where there are errors (5xx series, eg 503, 504, 500 or 404 file not found as examples), the size of requests and time taken for the entry too (not in the example above).

2.8.3.2.3 UI not rendered correctly For this case it would be necessary to determine if this was widespread or for a single service user. The various docker logs would need to be examined for anomalous conditions like the above. UI specific issues can often be diagnosed using Chrome Developer Tools, load the webpage in Chrome and press Ctrl+Shift+I.


From the developer tools you can inspect elements in the web page and investigate the styling applied to those elements. You can also view network traffic from the network tab to view requests and responses to the server, debug javascript issues from the sources tab and inspect cookies and local storage from the application tab.

2.8.3.2.4 Unable to update and save information In the case where information cannot be saved then the running of the EtherCIS java container should be checked. The log files for EtherCIS can be found at /var/opt/ecis. For System of Systems related data issues, the logs for the microservice fhir_service should also be checked for errors using:

sudo docker logs fhir_service

2.8.3.3 NHS Login

2.8.3.3.1 Authorisation not received Placeholder

2.8.3.3.2 Authorisation rejected Placeholder

2.8.3.4 FHIR Interaction

2.8.3.4.1 Authorisation received; demographics unavailable Placeholder

2.8.3.4.2 Authorisation and Demographics received; UI not rendered Check the developer tools of the browser for any console or network errors, you should also be able to check what document elements have been rendered from the developer tools.

2.8.3.4.3 Policy not accessible from YHCR As Helm requires that the user has consented to the Policies, the service will be unavailable until this issue has been remedied. The following should be checked on the Helm-Qewd server: Check the global_config.json file to see what Policies are configured in the initialisation_service section of the file. You should also then check that those policies are available in the YHCR. Check both the initialisation_service and fhir_service are running. Check the logs of the fhir_service to make sure the policies are being pulled from the YHCR.


2.8.3.5 etherCIS

2.8.3.5.1 EtherCIS unavailable The EtherCIS layer runs via a Java JVM and is underpinned by postgreSQL. This can be checked as follows on the on helmprod-ecisthis or on helmstaging-ecis depending on environment;

ps -ef |grep java

This command will output all java running processes on the host machine. If an EtherCIS JVM is not present, follow the instructions for starting EtherCIS see here.


2.8.3.6 postgreSQL

2.8.3.6.1 postgreSQL unavailable The following command can be run on centOS to determine status;

systemctl list-units|grep postgresql

The output of which should be like the below, dependant on status; postgresql-10.service loaded active running PostgreSQL 10 database server To restart use the following command (the postgreSQL string used to restart should be what was outputted from the previous command;

service postgresql-10.service restart


2.9 Business Continuity / Disaster Recovery

2.9.1 Scenarios

2.9.1.1 About Placeholder. The specifics of the measures in place for Helm Business Continuity and

Disaster Recovery are yet to be finalised.

2.9.1.2 Loss of external services

2.9.1.2.1 YHCR Placeholder

2.9.1.2.2 NHS Login Placeholder

2.9.1.2.3 HSCN Placeholder

2.9.1.3 Loss of UKCloud Infrastructure

2.9.1.3.1 Network Placeholder

2.9.1.3.2 Servers Placeholder

2.9.1.3.3 Storage Placeholder

2.9.2 Procedures

2.9.2.1 Application Restore Placeholder

2.9.2.2 Database Restore Placeholder


2.10 Software

2.10.1 Software Repositories

Helm Project https://github.com/LeedsCC/Helm-PHR-Project/issues

PulseTile - https://github.com/PulseTile/PulseTile-RA/tree/helm

Qewd-Courier - https://github.com/QEWD-Courier/QEWD-Courier (develop is default/most

up to date branch)

Oidc Client - https://github.com/robtweed/oidc-client

Oidc Provider - https://github.com/robtweed/oidc-provider

Fhir Service - https://github.com/richardcbrown/QC-MPI-Microservice

EtherCIS repo - https://github.com/ethercis/ethercis (v1.3)

Code for service to be moved to a central location.

2.10.2 Key Configuration

The following configuration files should be backed up to private source (not stored in public repositories) and properly version controlled. They will need to be reinstalled on rollback/disaster recovery.

2.10.2.1 Helm-Hscn-Proxy:

Nginx Reverse Proxy - /etc/nginx/conf.d/default.conf

Squid Forward Proxy - /etc/squid/squid.conf.default

2.10.2.2 Helm-DMZ:

Nginx reverse proxy - /etc/nginx/conf.d/default.conf

2.10.2.3 Helm-Qewd:

Qewd microservice configuration - /home/centos/qewd-

helm/helm/configuration/config.json

Qewd microservice routes - /home/centos/qewd-helm/helm/configuration/routes.json

Qewd service specific configuration - /home/centos/qewd-

helm/helm/configuration/global_config.json

Fhir service configuration - /home/centos/qewd-helm/fhir_service/config.json

Fhir service routes - /home/centos/qewd-helm/fhir_service/routes.json

Fhir service internal configuration - /home/centos/qewd-

helm/fhir_service/fhir_service.config.json

Fhir service search configuration - /home/centos/qewd-

helm/fhir_service/fhir_service.search.json

Fhir service privateKey (for YHCR token) - /home/centos/qewd-

helm/fhir_service/privateKey.key


Oidc client service configuration - /home/centos/qewd-helm/oidc-

client/configuration/config.json

Oidc client routes - /home/centos/qewd-helm/oidc-client/configuration/routes.json

Oidc client service specific configuration - /home/centos/qewd-helm/oidc-

client/configuration/oidc.json

Oidc client private key (for NHS token) - /home/centos/qewd-helm/oidc-

client/configuration/private_key.pem

Oidc client public key (for NHS token) - /home/centos/qewd-helm/oidc-

client/configuration/public_key.pem

Oidc token field extractor (for NHS login) - home/centos/qewd-helm/oidc-

client/configuration/extract_idToken_fields.js

Oidc provider service configuration - /home/centos/qewd-helm/oidc-

provider/configuration/config.json

Oidc provider service specific configuration - /home/centos/qewd-helm/oidc-

provider/configuration/oidc.json

Oidc provider install data (institution clients) - /home/centos/qewd-helm/oidc-

provider/configuration/data.json


2.11 Helm Servers

Operating System Access - All accesses to the operating system (CentOS 7) are done using SSH.

2.11.1 Helm-Admin (helmprod-admin, helmstaging-admin)

2.11.1.1 About - System administration, admins must log in or tunnel through the admin server to reach internal

servers.

Locations: - Staging:

51.179.212.165

192.168.51.12

- Live:

192.168.50.14

51.179.212.163

Replicated: No Can be accessed by SSH directly: Yes


2.11.2 Helm-HSCN-Proxy (helmprod-hscn-proxy, helmstaging-hscn-proxy)

2.11.2.1 About - Forward and Reverse proxy server between Helm infrastructure and HSCN network

- Access into Helm network from HSCN configured by NGINX

- Access to HSCN network from Helm configured by Squid


o 192.168.54.11 (SSH)

o 192.168.254.11

o 10.200.82.70

- Live:

o 192.168.254.11

o 192.168.54.11 (SSH)

o 10.200.82.80

Replicated: No Can be accessed by SSH directly: No Configuration: - Nginx Reverse Proxy - /etc/nginx/conf.d/default.conf

- Squid Forward Proxy - /etc/squid/squid.conf.default

Notes: - Access into Helm from HSCN is necessary to allow IDCR users (e.g. LTHT) to pull data entered

by PHR users.

- Nginx configuration should be adjusted like the configuration of the DMZ server reverse

proxy.


2.11.3 Helm-DMZ (helmprod-dmz, helmstaging-dmz)

2.11.3.1 About - Bastion server between internet and Helm internal servers

- Serves static pages i.e. PulseTile build

- Proxies requests to Helm internal servers


o 192.168.54.10

o 51.179.212.178

- Live:

o 192.168.54.10

o 51.179.212.201

Replicated: No Can be accessed by SSH directly: Yes Configuration:

- PulseTile build - /usr/share/nginx/html

- Nginx reverse proxy - /etc/nginx/conf.d/default.conf

Notes: - Staging currently allows for ssl_protocols TLSv1 – this was for LTHT testing but it has been

noted this is not longer necessary

- Primary routes within nginx are:

o / - serves static pages from PulseTile

o /api – proxies requests to Qewd microservices via the Orchestrator microservice at

http://192.168.55.11:8000

- Additional routes are provided to proxy to microservices directly, these may be used to

access monitoring capabilities included in Qewd (/qewd-monitor). It has been noted that

access to these capabilities should be blocked under normal operations and enabled only for

short periods of time while debugging issues as they are only protected by password which

may or may not be configured as the default password for Qewd-Monitor, a Task should be

created to change these passwords and make sure these routes are disabled under normal

operations.

http://192.168.55.11:8000/


2.11.4 Helm-QEWD (helmprod-qewd, helmstaging-qewd)

2.11.4.1 About - Holds Qewd-Courier microservices within Docker.


o 192.168.55.11

- Live:

o 192.168.55.11

Replicated: No Can be accessed by SSH directly: No (via admin server) Configuration: Qewd-Courier - /home/centos/qewd-helm


2.11.5 Helm-EtherCIS (helmprod-ecis, helmstaging-ecis)

2.11.5.1 About What is it for:

- Holds EtherCIS REST API to store clinical data (i.e. top 3 things).


o 192.168.51.12

- Live:

o 192.168.51.12

Replicated: No Can be accessed by SSH directly: No (via admin server) Configuration: - EtherCIS configuration file - /home/ethercis/ethercis-serverd – this file contains all the

information about configuration settings for EtherCIS.

Notes: - The log files for EtherCIS can be found in the /var/opt/ecis directory.


2.11.6 Helm-Logging (helmprod-logging, helmstaging-logging)

2.11.6.1 About - Contains Graylog to hold logs produced by Helm system.

- Does not currently have any logs being sent to it from the system.


o 192.168.50.15

- Live:

o 192.168.50.15

Replicated: No Can be accessed by SSH directly: No (via admin server)


2.11.7 Helm-Db (helmprod-db, helmstaging-db)

2.11.7.1 About

- Holds postgresql database for use with EtherCIS, holds clinical data (e.g. top 3 things).


o 192.168.53.13

- Live:

o 192.168.53.13

Replicated: No Can be accessed by SSH directly: No (via admin server)


2.12 Support Knowledgebase

2.12.1 Known Issues

2.12.1.1 Missing authorization header On restart of the docker microservices, the following can be seen periodically in the oidc_provider_1 log. This stops Helm from being functional. In order to rectify, the microservices should be restarted.

oidc-provider_1 | OpenId Connect Server Loader starting with

params:

oidc-provider_1 | {

oidc-provider_1 | "error": "Missing authorization header",

oidc-provider_1 | "disconnect": true

oidc-provider_1 | }

oidc-provider_1 | undefined


2.13 Appendix

2.13.1 Application Component Schematics provided by the Ripple Foundation

2.13.1.1 Integration and Data Normalisation Tier


2.13.1.2 EtherCIS


2.13.1.3 Database Persistence


2.13.1.4 Administration


2.13.1.5 Logging


2.13.2 UK Cloud handover recommendations After the deployment phase is completed, refinements and adjustments to the infrastructure will be required in order to ensure its longevity and fitness for purpose. The following are some areas that should be assessed with higher priority.

2.13.2.1 Backup Implementing a regular, secure backup and restore strategy should be a top priority. There is currently no backup procedure in place, and all services are at risk until this is resolved. At a minimum, regular snapshots of instances should be taken, and a process be put into place to manage the lifecycle of these snapshots. Snapshots can be created inside the UKCloud for OpenStack web portal. Automated snapshot workflows can be created using the OpenStack API or the OpenStack command-line client interface.

2.13.2.2 Monitoring Basic monitoring of instances can be performed inside the UKCloud for OpenStack web portal, including uptime and current running status. It is highly recommended that a monitoring system be implemented that covers application specific metrics, as well as CPU, storage, and networking usage inside the running instances.

2.13.2.3 Resource Monitoring System resources such as CPU utilisation, disk space, and network usage should be monitored using standard tools. Examples of these tools are: • Icinga / Nagios • Sysstat • SAR • Grafite Monitoring data should be integrated into a system dashboard, such as as Grafana or Kibana.

2.13.2.4 Performance Monitoring Quality of Service baselines should be established and integrated into the monitoring strategy. Critical metrics vary by application, but examples include ping times, storage IOPs, and access logs.

2.13.2.5 Service Monitoring Service monitoring from an external provider should be implemented. Examples of this type of service: • PingDom • Uptime Robot

2.13.2.6 Lifecycle Management Processes should be in place to manage the deployment, patching, and decommissioning of all services. Administrative tasks such as system updates, security scans, and resource monitoring should be assigned to owners and carried out according to best practices. Regular patching and auditing are essential to the stability and security of the deployment. Updates and system upgrades should be carried out in a staged workflow, providing the ability to reliably test any changes in the staging environment before pushing to production.

2.13.2.7 High Availability Each service is currently a single point of failure. It is highly recommended that the architecture be evolved to implement high availability for all services. Recommended architectures for compute resources, proxies, and database services should be


developed, and a migration plan put in place to realise those new architectures.

2.13.2.8 Disaster Recovery Procedures should be developed for automated redeployment of the Helm application upon loss of service. Ideally, this would create a second geographic target for deployment, along with provisions to access recent snapshots or backups of the running environment. This redeployment should be tested regularly.


2.13.3 Network Topology on UKCloud The actual Helm deployment on UKCloud is done according to OpenStack capabilities. The strategy is to simulate actual fine grained subnet partitions by using network filters (implemented as Security Groups). The following section provides some details on the topology and filtering mechanisms.

2.13.3.1 Network Architecture Network is deployed as OpenStack virtual networks. The administration of this environment is simplified using UK Cloud web console. Helm network consists of the following subnets/segments:

● Internet: provides the routing to the external public network, its gateways are materialized

by 2 routers: helmstaging-router and helmprod-router to bridge respectively staging and

production to the external public network.

● HSCN: as above to route traffic to/from HSCN with staging and production

● Helmstaging-network: hosts all instances related to staging deployment

● Helmprod-network: hosts all instances related to production

Several instances (e.g. virtual hosts) have floating IPs enabling bridging with external networks:

● Hscn-proxy with HSCN

● Admin and dmz with Internet

https://docs.openstack.org/stein/


This is summarized in the following table

external address(es) (via routers)

Network Name subnets CIDR

10.200.82.84 helmprod-proxy helmprod-proxy 192.168.254.0/24

51.179.212.165 (admin) 51.179.212.178 (dmz)

helmstaging-network helmstaging-db 192.168.53.0/24 helmstaging-ecis 192.168.51.0/24 helmstaging-qewd 192.168.55.0/24 helmstaging-admin 192.168.50.0/24 helmstaging-dmz 192.168.54.0/24

51.179.212.201 (dmz) 51.179.212.163 (admin)

helmprod-network helmprod-ecis 192.168.51.0/24 helmprod-qewd 192.168.55.0/24 helmprod-db 192.168.53.0/24 helmprod-admin 192.168.50.0/24 helmprod-dmz 192.168.54.0/24

10.200.82.68 helmstaging-proxy helmstaging-proxy 192.168.254.0/24

myhelm-network myhelm-subnet 192.168.101.0/24

51.179.210.205 51.179.212.162 51.179.212.215

internet

HSCN


2.13.3.2 Network Traffic Logical View The following is an attempt to visualize the network rules between the different components of the architecture. To simplify this exercise, only staging is being represented as production is identical on this viewpoint.

Since the network topology is compartmentalized using OpenStack Neutron Namespaces, the partitioning is somewhat similar to multi-tenancy between staging and production (hence the apparent usage of identical conflicting CIDRs) as networks are actually belonging to totally independent domains. Further, the network traffic is controlled via filtering rules that mimic physically independent subnets with defined routing. This can be depicted as follows:

helm operations manual - yorkshire & humber care record€¦ · 2fa a web presentation tier...

Documents