PCT TEST METHODOLOGY FOR PLANT SUPERVISORY

CONTROLLERS FOR BUILDING AUTOMATION

Kumar Abhishek (Senior Engineer)

Honeywell Technology Solution Lab Pvt Ltd

TABLE OF CONTENTS

Contents

Executive Summary__________________________________________________________________2

Introduction________________________________________________________________________2

The Problem and Solution_____________________________________________________________2

Differentiation______________________________________________________________________3

Test Execution checklist_______________________________________________________________4

Memory parameters capture procedure__________________________________________________4

What more needs to be done__________________________________________________________4

Advantages and disadvantages or limitations______________________________________________4

Details of third-party controllers________________________________________________________4

Real or Virtual devices?_______________________________________________________________4

How much load should we test with?____________________________________________________4

Setup Configurations_________________________________________________________________4

Setup Diagrams_____________________________________________________________________4

Performance graphs__________________________________________________________________4

Conclusion/Analysis__________________________________________________________________4

References & Appendix_______________________________________________________________4

Author Biography____________________________________________________________________4

Executive Summary

A series of performance and memory usage tests using a BACnet network, multiple BACnet devices, each with multiple points, and industry-standard BMS SCADA software resulted in the development of a methodology test engineers can use to verify the quality of their BMS products.

The tests conducted for this paper suggest that sustaining a high level of heap usage for a long period of time may allow testers to observe an embedded supervisory controller’s performance in terms of memory leaks, CPU usage, System memory, etc.

Introduction

Performance testing is a major and important Process Control Technology (PCT) for producing a quality product that can sustain a business in the marketplace.

This paper introduces best practices to use when performance testing third-party LonWorks and BACnet controllers to verify their compatibility with BMS (Building Management System) SCADA (Supervisory Control And Data Acquisition) software.

Analysing the behavioural changes embedded supervisory controllers experience as various devices and points are added can help users understand the types of devices that are best supported by BMS SCADA software.

The performance test also defines the memory configuration parameters required to analyse memory consumption levels in embedded supervisory controllers.

This study used a BACnet network and supported devices for developing the test methodology.

The Problem and Solution

For any new product, performance testing is a challenging task. Many parameters need to be configured very carefully prior to executing the test. The ability of the test to record subtle behavioural changes requires long periods of time in an undisturbed environment.

If any mistake occurs during the test, the entire test must begin again from beginning. Mistakes are time consuming and tedious.

THE PART OF PROBLEM THIS PAPER ADDRESSESIn our case, we re-ran the performance test several times not because of configuration issues, but to discover the best method for executing a test capable of achieving the best quality product in terms of performance and memory usage.

This process can be followed to test any plant controllers from any vendor for compatibility and performance with any BMS SCADA software, and to analyse the behaviour of the embedded supervisory controllers in terms of memory consumption and leakage.

PROPOSED SOLUTIONTo establish the quality of the various third-party devices (such as BACnet devices), each embedded supervisory controller went through the performance and compatibility test using various devices and BMS SCADA.

The report also contains the complete end-to-end process to configure the setup with prior and post tests, including the capturing of various memory parameters at various levels to find out the memory consumption while gradually adding devices.

Differentiation

This report will be very helpful for new users who have never been involved in performance testing before. It can help them set up test parameters prior to starting the test as well as to identify when, and at what level, system performance degrades or begins to consume unacceptably large quantities of memory.

Test Execution checklist

Test compatibility with BMS SCADA software using embedded supervisory controllers. Discover various BACnet MSTP (Multiple Spanning Tree Protocol) devices using a

BACnet Network running BMS SCADA software. Discover points from the BACnet devices. Create logic and derive the BACnet points using simulation. Test read and write operations using BACnet points from BMS SCADA software

points and vice versa. Drive BACnet points using BMS SCADA software schedules. Generate trend logs, alarm notification, and BMS SCADA software alarms using

BACnet devices. Analyze the data packets transferred between the BMS SCADA software and the

BACnet devices along with analyzing network response times using the third-party tool: VTS, Wireshark.

Continue the test for a long run of activity to confirm the compatibility and sustainability of devices over a long period of time.

Memory parameters capture procedure

To set up the capture of memory usage, these steps need to be followed very carefully:

Disable the embedded supervisory controller’s automatic start feature. Disable the embedded supervisory controller’s restart-on-failure feature. Set the trigger to an interval with one (1) minute. This starts logging the data for heap

and system memory used. Create a numerical writable point with an interval history extension (@ 20 sec update

intervals), and link if from the embedded supervisor controller to the system’s CPU service (from the station side). This is for taking the CPU usage data.

Take the no load reading when the embedded supervisory controller station with the BACnet driver discovers devices. Take the reading at the discovery pane ONLY.

After taking the no load reading, discover the BACnet devices and add only one device to the station database, discover points, add 10 points, and configure trend logs, alarm notifications etc.

Once the system usage (CPU and system memory) stabilizes, invoke GC (Garbage Collection) and immediately capture the reading of CPU, heap used and system used from the station resource.

After GC, allow the setup to run for one minute, then allow it to run for 10 minutes undisturbed.

Take the reading for the previous 10 minutes and keep the setup idle again for a couple of hours before adding the next set of devices.

Follow the above procedure for adding the next set of devices like 5, 15, 25, 35….. & so on

What more needs to be done

The same performance test needs to be re-executed for any major build that to be qualified for beta release. Build-on-build comparisons are to be done to analyse the behaviour changes of builds. Dedicated device configurations. [ABHISHEK: The previous sentence is a fragment. What else do you want to say here?]

Advantages and disadvantages or limitations

The main advantage of following the methodology defined in this paper is to make the product very robust, sustainable, compatible (with various third-party devices) and reliable prior to release to the market.

This test was performed with limited third-party devices. Investment in various types of third-party BACnet devices is needed to expand the testing scope.

Details of third-party controllers

All performance tests used all real, third-party BACnet devices that had been verified for compatibility with the embedded supervisory controller.

1. EAGLE Controller EAGLE is CentraLine’s Ethernet-based, freely-programmable Building Automation controller offering a combination of BACnet IP, BACnet MS/TP, and LONWORKS® communication.

http://www.big-eu.org/uploads/tx_teproddb/catalog_pdf/en0z0970- ge51r0214_Honeywell_Product_catalogue.pdf

2. Remutech I/O Fieldbus modules Modbus rtu / BAC-net MS/TP, RS485

http://www.romutec.eu/Products/I-O-Fieldbus-modules-Modbus-rtu-BAC-net-MS-TP-RS485.html

http://www.romutec.de/Produkte/Feldbus-Module-I-O-Module-Remote-Direct-Control/RDC.html?c=fo_categorylists.php&CatID=261

RDC 621, RDC 601, RDC 633, RDC 641 & RDC 643: Remote Direct Control Series 600. About the RDC device series digital and analog signals at various points can be recorded and processed in the building. The monitoring and control of inputs and outputs is realized in automatic mode via the RS485 interface. Several protocols available to bind DDC systems.

http://www.romutec.de/Produkte/Feldbus-Module-I-O-Module-Remote-Direct-Control/RDC.html?c=fo_showproduct.php&ItemID=1639

3. Alerton VLC 853: Alerton's BACtalk VLCs are high-performance, fully programmable logic controllers designed to control central-plant systems, air handling units, any type of terminal unit, and various control and process equipment.

Every VLC is BACnet-compliant. As a native BACnet controller that requires no proprietary chipsets to integrate seamlessly with a BACnet system. All inputs are 10-bit resolution. VLC-853—Eight 10-bit inputs, five binary outputs, and three 8-bit analog outputs.

VLC 444: Alerton's VLC-444 is a BACnet field controller designed for next-generation fan-coil units that include variable speed fan control and other features typically found in EMC or EC-DC fan-coil units. In international markets, EMC fan coils generate significant energy savings and meet local legislation that mandates low energy consumption from terminal equipment. Currently, the EMC fan-coil is the easiest way to ensure compliance with that legislation.

VLC-1600: The Alerton® BACtalk® VLC-1600 is a high-performance, fully programmable input monitoring device. With 16 high-resolution inputs, it is perfectly suited for applications with high input density, where it can augment the input capabilities of other controllers. The VLC-1600 has no control outputs. VLC-1600—16 inputs.

https://alerton.com/en-US/Pages/Category.aspx?cat=ECC-Alerton&category=Field%20Controller

4. Trend IQeco31The IQeco31 is a terminal unit controller for use with BACnet over MSTP. It can communicate with other IQecos over the BACnet MSTP network, and with Trend networked devices by way of a BINC. It has 10 I/O channels and can be supplied as a fixed or programmable unit, with Trend written strategies, custom strategies or no strategy.

https://partners.trendcontrols.com/trendproducts/cd/en/ecatdata/pg_iqeco31_24v.html

5. Spyder® BACnet Programmable ControllersThe PUB and PVB controllers are part of the Spyder family. These controllers are BACnet MS/TP network devices designed to control HVAC equipment. They provide many options and advanced system features that allow state-of-the-art commercial building control. Each controller is programmable and configurable through software.

https://customer.honeywell.com/resources/techlit/TechLitDocuments/63-0000s/63-2689.pdf

Real or Virtual devices?

Although the performance test used all real, third-party BACnet devices, one BMS SCADA software controller, which acts as BACnet device, was used to verify software compatibility with the embedded supervisory controller.

How much load should we test with?

Once the CPU usage reaches to 80% of load, keep the setup idle for a long run to analyse and monitor the system degradation, any memory leakage, failure time line, etc.

Setup Configurations

This is configuration at the BMS SCADA software level with the embedded supervisory controller including device details (model, number of device added, baud rate etc.), check points (mandatory parameters), and load details for each device.

Setup Diagrams

Fig. 2: Setup diagram -1

Fig. 3: Setup diagram -2

Performance graphs

The various memory parameters have been captured with incremental load like heap used (GC-Garbage Collection, average and peak), system memory usage (GC, average and peak), CPU (GC, average and peak).

Heap Used (GC- Garbage Collection, Average & Peak):

1 5 15 25 35 45 55 65 75 85 95 105 1150

10

20

30

40

50

60

70

25 26

34

42

34 35 37 38 40 42

59

49 50

Heap Used in MB (after GC)

# Device

Hea

p Us

ed

1 5 15 25 35 45 55 65 75 85 95 105 1150

20

40

60

80

100

120

140

73.9 78.688.2

79.590.9 89.4 87.9

97.9 101.6

86.1

109.2117.6

101.5

# Device

Heap

Use

d

Heap Used in MB (Average Value)

1 5 15 25 35 45 55 65 75 85 95 105 1150

20

40

60

80

100

120

140

160

180

117.9130.0 132.1 129.0 131.0 133.9 130.5

138.4 142.8133.4

152.9 149.8 144.9

Heap Used in MB (Peak Value)

# Device

Heap

Use

d

System Memory (GC, average and peak):

1 5 15 25 35 45 55 65 75 85 95 105 115902

904

906

908

910

912

914

916

918

907 907

909 909910 910

911 911 911912 912

917 917

# Device

Syst

em M

emor

y

System Memory Used in MB (After GC)

1 5 15 25 35 45 55 65 75 85 95 105 115900

902

904

906

908

910

912

904.9905.6

907.1 907.4 907.7 907.9908.5

909.2 909.2 909.4 909.6910.3

910.8

# Device

Syst

em M

emor

y

System Memory Used in MB (Average Value)

1 5 15 25 35 45 55 65 75 85 95 105 115902

903

904

905

906

907

908

909

910

911

912

905.0905.6

907.1907.6 907.7 907.9

908.5909.2 909.2 909.4 909.6

910.3910.9

# Device

Syst

em M

emor

ySystem Memory Used in MB (Peak Value)

CPU (GC, average, and peak):

1 5 15 25 35 45 55 65 75 85 95 105 1150

5

10

15

20

25

30

35

10 10

1315

22

27

16

19 19

27

31 31 32

# Device

Curr

ent C

PU

Current CPU Usage in % (After GC)

1 5 15 25 35 45 55 65 75 85 95 105 1150

10

20

30

40

50

60

70

80

90

10.5 9.5

21.9

35.0

47.652.4

59.764.8

69.874.6 76.1 78.9

83.2

# Device

Curr

ent C

PU

Current CPU Usage in % (Average)

1 5 15 25 35 45 55 65 75 85 95 105 1150

20

40

60

80

100

120

17.021.0

32.0

48.0

69.0 72.0 75.0 78.0

100.0

87.0

100.0 100.0

84.0

# Device

Curr

ent C

PUCurrent CPU Usage in % (Peak Value)

Conclusion/Analysis

The test team observed that keeping the heap usage high for a longer period of time that allowed in these tests would all us to observe the performance behaviour in terms of memory leaks, CPU usage, System memory, etc.

References & Appendix

http://www.stitcs.com/en/lonworks/bacnet%20vs%20lonworks.pdf

http://www.usa.siemens.com/intelligent-infrastructure/assets/pdf/smart-building-white-paper.pdf

https://www.tridium.com/~/media/tridium/library/documents/collateral/white%20papers/BMS SCADA software%20appliance.ashx?la=en

http://www.bacnet.org/Bibliography/DMF-7-96/DMF-7-96.htm

https://alerton.com/en-US/support/whitepapers/Documents/MK-WP-BACNETTRAFFIC.pdf

http://www.slideshare.net/susantsahani/bacnet-white-paper

Author Biography

The author Mr. Kumar Abhishek has overall 7+ years of experience in Honeywell Technology Solution Lab Pvt Ltd in the field of Building Management Systems – HVAC systems where involved in Software Development Life Cycle including Integration testing with different hardware, Functional, Regression, Performance, Reliability Testing using Agile Methodology & Automation Testing. He holds the Master’s in Mechatronics from Vellore Institute of Technology & Bachelor’s in Mechanical from North Maharashtra University.

KUMAR ABHISHEKSENIOR ENGINEER

Kumar.abhishek@honeywell.comhttps://www.linkedin.com/in/kumar-abhishek-b3b5081b/

THANK YOU!