Isode

Steve Kille

CEO Isode Ltd

14th February 2019

MEASURING PERFORMANCE OF MESSAGING PROTOCOLS FOR HF RADIO

Isode

Contents

• Background and Goals

• Messaging Protocols

• Test Approach

• Analysis

Isode

Why Bulk Throughput Matters

▪ Optimizing data throughput over HF is a hard problem▪ Optimizing for low latency (e.g., XMPP covered in other talk today) is relatively easy

▪ If you are sending mission critical data over bulk data (e.g., a photograph), best throughput leads to fastest delivery▪ Top WBHF (240kbps) with 100% link utilization is 30 secs/MByte

▪ Sometimes throughput is not good:▪ DSTL reported (Bristol HFIA Sep 2018) file transfer throughput of approximately 1% modem

speed

▪ This triggered the questions which led to this talk

▪ Question 1: What is a fair way to measure throughput?▪ Ideally want an approach which is easy to apply to any HF system

▪ Question 2: What is a reasonable target?▪ Talk looks at measurements of messaging protocols

▪ Messaging is the primary bulk application used over HF

▪ Measurement details in Isode white paper: “Measuring Performance of Messaging Protocols for HF Radio”

▪ Message protocol information in white paper “Messaging Protocols for HF Radio”

Isode

Easy Measurement of Bulk Throughput

▪ Use random binary data (e.g., Photo)▪ Important, as we want to test protocols not bulk data compression algorithms

▪ Transfer the data and calculate transfer rate by measuring▪ Volume of data transferred (in one or more blocks)

▪ Time of transfer

▪ Compare to link (modem) speed▪ Easy if fixed modem speed is used

▪ Use average if modem speed varies

▪ Calculate percentage utilization by comparing the two speeds

▪ Easy approach: attach photo(s) to an email and send▪ Reasonable estimates can be made without any special tools

▪ This gives a good way to compare systems

4

Isode

ACP 142

▪ Multicast Protocol designed for constrained bandwidth networks▪ STANAG 5066 operation for use over HF

▪ Supports EMCON operation

▪ Option to carry two formats of message▪ STANAG 4406 Annex E

▪ NATO Standard format for Messages at 20 kbps and slower

▪ RFC 8494 “Multicast Email (MULE) over Allied Communications Publication (ACP) 142”

▪ SMTP over ACP 142

▪ Allows support of Military Messaging using RFC 6477

▪ STANAG 4406 Annex E used in measurements▪ MULE expected to give almost identical number

5

Isode

SLEP

▪ SIS Layer Extension Protocol (SLEP) (S5066-APP3)▪ Open protocol specification developed by Isode

▪ Talk given at Bristol BLOS Comms (Sep 2018)

▪ SLEP defines a messaging protocol to carry▪ STANAG 4406 Annex E; or

▪ MULE (RFC 8494)

▪ Provides all ACP 142 functionality except Multicast and EMCON▪ Gives better point to point performance

▪ Useful in point to point networks (ALE) that will not benefit from Multicast

▪ We plan to measure MULE over SLEP▪ To publish in white paper update

▪ Implementation not quite ready in time for this talk

▪ Slides share anticipated results

6

Isode

CFTP

▪ Compressed File Transfer Protocol (CFTP)▪ Simple point to point SMTP protocol defined in STANAG 5066

▪ Reasons why inferior to ACP 142 and SLEP▪ No Delivery Status Notifications

▪ Important for HF

▪ Needed for end to end tracking

▪ 7bit, so BINARYMIME transfer not possible

▪ Some overhead for all attachments (seen in measurements)

▪ Large overhead for compressible attachments (e.g., Word Document)

▪ No priority support or overtaking

▪ So not suitable for military messaging

7

Isode

Measurement Architecture

▪ Isode Test Client submits messages

▪ M-Switch uses the selected protocol▪ Recipient address leads to right

protocol

▪ Icon-5066 provides end to end STANAG 5066

▪ MoRaSky simulates Modem (Waveform), Radio and Ionosphere▪ Parameters chosen according to

test

▪ Receiving end delivers Messages

▪ Viewed in Microsoft Outlook

8

Isode

Measurement Tool

▪ Screenshot shows result of typical test run in Outlook

▪ Subject of message makes it easy to get data from test run

▪ Test Tool controls▪ Number of messages

▪ Size of random binary data

▪ Interval between messages (for testing latency)

9

Isode

Basic Measurements

• 9600 bps STANAG 4539 short interleaver

• Clear Link

• For large messages over 90% link utilization can be achieved

• CFTP bulk performance slightly worse due to 7 bit encoding

• Message protocol overhead increases as payload size decreases

• Makes ACP 142 multicast overhead clear

• SLEP expected to always gives best performance

10

Protocol Size Utilization Message Protocol Overhead

STANAG 5066 Overhead

ACP 142 1 Kbyte 46% 45.6% 7.5%

CFTP 1 Kbyte 55.5% 37% 7.5%

ACP 142 10 Kbyte 83% 9.5% 7.5%

CFTP 10 Kbyte 82% 10.5% 7.5%

ACP 142 100 Kbyte 89.5% 3% 7.5%

CFTP 100 Kbyte 88% 4.5% 7.5%

ACP 142 1 Mbyte 91% 1.5% 7.5%

CFTP 1 Mbyte 89.5% 3% 7.5%

Isode

Slower Speeds

• 10 kByte messages• Largest sensible at 75bps

• STANAG 5066 overhead increases due to choice of C_PDU segment size appropriate to speed • Default Icon-5066

configuration options

• 800 bytes at 9600

• 300 bytes at 1200

• 75 bytes at 75bps

• Good performance at very low speeds is a key benefit of protocols directly using STANAG 5066

11

Protocol Speed Utilization Message Protocol Overhead

STANAG 5066 Overhead

ACP 142 75 bps 60% 7% 33%

CFTP 75 bps 60.5% 6.5% 33%

ACP 142 1200 bps 79.5% 9% 11.5%

CFTP 1200 bps 79% 9.5% 11.5%

ACP 142 9600 bps 83% 10% 7%

CFTP 9600 bps 82% 11% 7%

Isode

Faster Speeds

• STANAG 5069 used

• Faster speeds slightly better due to using max C_PDU segment size

• Top speed slightly slower, as modem simulation reflects some extra delays in the modem we modelled it on

12

Protocol Speed Utilization Message Protocol Overhead

STANAG 5066 Overhead

ACP 142 9.6 kbps 91% 1.5% 7.5%

CFTP 9.6 kbps 89.5% 3% 7.5%

ACP 142 57.6 kbps 93% 1.5% 5.5%

CFTP 57.6 kbps 90.5% 4% 5.5%

ACP 142 240 kbps 92.5% 1.5% 6%

CFTP 240 kbps 90% 4% 6%

Isode

Impact of Errors

• Bit Errors on modem output simulated

• Levels chosen to reflect typical operationl targets

• STANAG 5066 ARQ layer deals well with errors

13

Protocol Error Rate Utilization Message Protocol Overhead

STANAG 5066 Overhead

ACP 142 Clear 89.5% 3.5% 7%

CFTP Clear 88% 5% 7%

ACP 142 BER 10-6 88% 3.5% 8.5%

CFTP BER 10-6 86% 5% 9%

ACP 142 BER 10-5 72% 3% 25%

CFTP BER 10-5 64% 4.5% 30.5%

Isode

Messaging Latency

• Measured message transfer times

• 1 kByte payload

• STANAG 4539 9600 bps Short

• CAS-1 Link established

• Results consistent and reasonable

14

Protocol Min Max Mean

ACP 142 12 secs 13 secs 12.6 secs

CFTP 12 secs 13 secs 12.6 secs

Isode

STANAG 5066 Performance Notes

▪ STANAG 5066 overhead dominates for larger transfers

▪ For higher speeds, protocol overhead is about 4%▪ Increases as C_PDU segment size is decreased (choice is a trade-off)

▪ Window Size important▪ Even at 9600bps clear, “STANAG 5066 Large Windows Support” (S5066-EP5) needed for best

performance

▪ More important when errors introduced and higher speeds

▪ Turnaround overhead about 3% with 127.5 second transmissions▪ Shorter transmissions increase the overhead (e.g., 6% with 60 second transmission)

▪ Choice of maximum transmission length a key tuning consideration

15

Isode

Architecture vs Real Measurements

▪ Important to understand “Above Modem” architecture performance

▪ Test this by connecting modems back to back (perfect link) with fixed speeds

▪ Overheads due to link level, intermediate layers, and applications

▪ The architectures here give over 90% link utilisation

▪ This is a baseline comparison that should be made for any HF system▪ Anticipate that some IP architectures being proposed will not get anywhere near this

▪ Real systems will use variable speed▪ Need to determine average modem speed

▪ Real systems will lose data▪ If you are not losing data (90% utilisation) you are transmitting to slowly

▪ If you are getting (much) more than 50% frame loss (45% utilization) you are transmitting too fast

▪ Gives target link utilization range (45-90%)

16

Isode

Conclusions

▪ A good architecture for bulk traffic can:▪ Achieve over 90% link utilization over a perfect HF link

▪ Work reasonably at 75bps

▪ Architectures that do not achieve this should be avoided

▪ Operational link utilization for bulk traffic should be in the range 45-90%

▪ The best protocols for messaging are ACP 142 and SLEP▪ Can be used with STANAG 4406E or MULE (RFC 8494)

▪ It is time to retire CFTP

17

Isode