syzygy engineering key technologies regarding distributed and/or collaborating satellite systems -...
TRANSCRIPT
SYZYGY Engineering
Key Technologies Regarding Distributed and/or Collaborating
Satellite Systems- Protocols
Will IvancicSYZYGY Engineering
[email protected](440) 503-4892
© 2014Syzygy Engineering – Will Ivancic
SYZYGY Engineering
Protocols
2
Protocols are tools for communication
Some should not be used.
Some work OK
Some are designed for the job at hand
One size does not fit all.
SYZYGY Engineering
Beware on inappropriatecomparisons (Marketing)
Good Performance Poor Performance
SYZYGY Engineering
4
An Example of TCP Steady State Performance
Chart is from “Why not use the Standard Internet Suite for the Interplanetary Internet?”
By Robert C. Durst, Patrick D. Feighery, Keith L. Scott
Does this infer that all Internet Protocols are not suitable for
Interplanetary operations?
SYZYGY Engineering
Reliable, High-Rate Transport Protocols for Highly Asymmetric
Private Links
5
SYZYGY Engineering
6
Theoretical Steady State Throughput
Increasing Delay
Delay Tolerant
TCP Performance equitation is from Mathis, M. et al, "The Macroscopic Behavior of the Congestion AvoidanceAlgorithm",Computer Communications Review, volume 27, number 3, July 1997.
• Don’t Use TCP for long delays. However, one can still use IP and a rate-base protocol.
• TCP does not operate well on highly asymmetric links due to Acknowledgement congestion.
SYZYGY Engineering
DTN Layering (RFC 5050)
Application
DTN Bundle Agent
Co
nve
rgen
ce L
aye
r
UP
D
LT
P
TC
P
Blu
eto
oth
US
B
AO
S
IP? IP
Convergence
Layer
(This is the transport
protocol piece)
Heterogeneous
Network
SYZYGY Engineering
“PROTOCOLS FOR STORE-AND-FORWARDMESSAGE SWITCHING VIA MICROSATELLITES”
J. W. Ward and H. E. Price
• A suite of protocols specifically optimized for use on store-and-forward microsatellite communications missions.
• Design the spacecraft software so that there was one generic data entity that it knew how to deal with and deal with well
• Leave all routing decisions to the ground segment.
• A client identifies a message which it requires and sends a broadcast request to the server.
• The client will receive some or all of the datagrams from its request. Upon receiving these (or other) datagrams, the client places them at the indicated byte offset in the appropriate file.
• When a client wishes to have the server retransmit missing datagrams, the client composes and transmits a repeat request datagram. If some of the desired datagrams are not received, then the clent sends the request again.
• The client repeats this process until the message is complete.
9
Generic SNACK GET Concept
Receiver Sender
REQUEST
METADATA
DATA 1
DATA 2
DATA 3
DATA 4
STATUS optional / resume transfer
DATA 3frame lost
DATA 5
STATUS I need 3 again
STATUS All received OK9
10
Generic SNACK PUT Concept
Receiver Sender
METADATA
DATA 1
DATA 2
DATA 3
DATA 5
DATA 3frame lost
DATA 4
STATUS I need 3 again
STATUS All received OK
SYZYGY Engineering
Selective, Negative Acknowledgment (SNACK) Protocols – not all inclusive
11
Consultative Committee for Space Data Systems (CCSDS) File Delivery Protocol (CFDP) - CCSDS 727.0-B-2:
• Selective Negative Acknowledgement• Designed for Space Applications (Highly Asymmetric Links, Suspend/Resume timers during
known outages)• Private Links (No Congestion Control)• Designed for File Transfers (really and application and transport protocol)• Reliable or Unreliable file transfer • Routing Capabilities (Preconfigured Next Hop – Static Routing)
Licklider Transport Protocol (LTP)• Based on CFDP• Distinguishes between important and unimportant data• Timers work together with communication schedules and can be suspended whenever a
scheduled link outage occurs • Needs to be informed about link layer availability, round-trip time and communication schedule,
basically requiring a management information base, Thus, Highly stateful
Saratoga (used buy SSTL daily)• Based on “Protocols For Store-and-forward Message Switching Via Microsatellites” • CFDP proved overly complex • High-speed, UDP-based, peer-to-peer protocol, providing error-free guaranteed delivery of files, or
streaming of data.• No specified timers means no timeouts, so Saratoga is ideal for very long propagation delay
networks (such as deep space)• Able to cope with high forward/back network asymmetry (>850:1)
SYZYGY Engineering
Store, Carry and Forward (SCF)(DTN or Others)
End-to-end (e.g,. SCP, FTP):
Must wait for complete path
SCF:
Incremental progress without end-to-end
path
SCP/FTP LatencySCP/FTP Throughput
SCF ThroughputSCF Latency
SYZYGY Engineering
DTN(Implies TCP cannot be used for
convergence layer)
SYZYGY Engineering
Requires In-Order Delivery
Does not Guarantee In-Order Delivery
Need In-Order Delivery App
Aggregation (or Not?)
16
64K Bundles (De-aggregation)
64K Bundles (Re-aggregation)
Bundles Table Explosion
Req
uiers T
op
olo
gical A
dd
ressing
DTN in a Nutshell
17
DTN Source
DTN Bundle
Forwarding Agent
DTN Destination
Where are you
headed?
Knoxville, Wheeling,
then Fort Pitt.
Can you take this letter to my brother
at Fort Pitt?
Yes Mam, I sure can.
Dave Smith
@ Ft. Pitt
Dave Smith
@ Ft. Pitt
Dave Smith
@ Ft. Pitt
Dave Smith
@ Ft. Pitt
This is
bundling
This is “opportunistic”
bundle-agent
Discovery
SYZYGY Engineering
DTN in a Nutshell
18
DTN Bundle
Forwarding Agent
Dave Smith
@ Ft. Pitt
Dave Smith
@ Ft. Pitt
DTN Bundle
Forwarding Agent
DTN Destination
SYZYGY Engineering
Why DTN?
• Cons– Added complexity– Not needed for single hop
systems– Adds Overhead
• Can be significant overhead depending on the bundle size
• Pros– Flexibility– Reliability– Robustness– Automates routing
for multi-hop, multi-path network topologies
19
SYZYGY Engineering
Delay/Disruption Tolerant Networking (DTN)
Store
Forward
DTN
• Opportunistic
•Low delay
•Possibly no need to schedule assets
DTN
• Long delays• Need to schedule assets
DTN is really an application overlay that has
aspects of scheduling, data transport and routing.
DTN Environments Opportunistic
(intermittent
connectivity,
short delay)
Sched
uled
(long d
elay
)
Opportunistic(intermittent connectivity, short delay)
SYZYGY Engineering
Single-Hop Architecture
• CFDP• LTP• Saratoga• Others• No Need for DTN
Complexity– DTN is most useful for
multi-hop networks
ForwardStore
SYZYGY Engineering
Delay/Disruption Tolerant Network (DTN)
• Goal: A standardized store and forward protocol and routing protocol
• Designed for extreme environments– Large transmission link delays – Extended periods of network partitioning– High per-link error rates making end-to-end reliability difficult– Routing capable of operating efficiently in the following
environments• Frequently disconnected• Pre-scheduled or Opportunistic link availability• Heterogeneous underlying network technologies
(including non-IP-based internets)• The architecture operates as an overlay network
23
SYZYGY Engineering
DTN Layering (RFC 5050)
Application
DTN Bundle Agent
Co
nve
rgen
ce L
aye
r
UP
D
LT
P
TC
P
Blu
eto
oth
US
B
AO
S
IP? IP
Convergence
Layer
Heterogeneous
Network
SYZYGY Engineering
Known Issues with CurrentBundling Specifications
• No IRTF generated Request For Comments (RFCs) are standards.– RFC5050 is an Experimental Specification not a Standard
• Requires loose time synchronization• No reliability checks on the header• No way to terminate routing loops• Naming is addressing
– Flat name space makes scalability hard if not impossible– No identified way to aggregate routes
• Dynamic Routing is incomplete• Discovery is incomplete• Bundle Security Protocol
– Specification is overly complex– Cases have been identified were it is broken– Security does not work with reactive fragmentation– Key exchange and key management is an open issue
25
SYZYGY EngineeringDTN ON THE INTERNATIONAL
SPACE STATION
SYZYGY Engineering
ISS History
• Design dates back to 80s and early 90s – pre-Internet era
• Phase 0– Commanding via low rate S-Band uplink
• Uplink bandwidth is limited resource that must be managed.– Payload data downlink via Ku-Band at 50 Mbps– Highly asymmetric links and both may not be on simultaneously
• Phase I– KuBand uplink at 3 Mbps– Orbital communication adapter (introduced Internet
• 3 Mbps Up, 6.5 Mbps down IP in CCSDS packets• KuBand Upgrade
– 30 Mbps Up, 300 Mbps Down Ku-Band– IP onboard with IP encapsulated in CCSDS up/down
SYZYGY Engineering
NASA ISS DTN Project
• ISS Disruption Tolerant Networking (DTN) Architecture for flight, ground, and test/simulation
• Increased reliability of payload data transfers between ISS and remote payload control centers during Acquisition of Signal (AOS) / Loss of Signal (LOS) transitions
• Increased automation of Payload Developer (PD) requests for data transfers
• Mechanism to alleviate extensive support to plan payload transfers around AOS/LOS and operator required transfers
• Mechanism to use standard, publicly available protocols, avoiding the use of costly custom protocol implementations
• Opportunity to gain valuable experience using DTN, which is the expected communication protocol of choice for future space exploration
SYZYGY Engineering
Current Payload Operations
• Uplink/Downlink request • Entire files have to be re-transmitted when transfer errors occur • Manual transfer by operations required for file transfers across
AOS/LOS • 24x7 continuous support operations to ensure access to science
data • Data recorded to Outage Mass Memory unit requires operators to
playback • Custom file transfer protocols utilized. Currently each payload,
tool, support equipment uses a custom method or protocol to transfer data reliably between on-orbit and ground – Ex. Alpha Magnetic Spectrometer (AMS) and custom AMS
Crew Operations Post (ACOP) system to store and buffer data in a proprietary implementation
SYZYGY Engineering
DTN Benefits to Payload Communications
• Provides capability to automate operations and ensure science delivery with little regard for link or facility outages
– MSFC, MCC-H, and ISS DTN nodes will store user file uplinks/downlinks and forward bundles as Kuband becomes available
• Reduce real-time support to access and downlink science data
• DTN stores data during LOS and automatically initiates transfer upon AOS
• A download transfer can span Ku-Band AOS periods without any special scheduling or scripting
• Reduces need for duplicate storage and extra retrieval actions
SYZYGY Engineering
DTN Benefits to Payload Communications
• Reliable data transfer for ISS during LOS/AOS cycles
– Automatic verification of bundle receipts, retransmissions reduced
– When transmission errors occur only the bundles that have errors are retransmitted
– Maximizes use of bandwidth by reducing the amount of data that has to be retransmitted
• Allows Payload Developers to use DTN protocols for their own applications
• Efficient use of downlink stream through DTN Quality of Service (QoS)
• Tolerance for high network latency (600 ms Round Trip Time (RTT) delay is typical on Ku link)
SYZYGY Engineering
Current/Future ISS DTN Use Cases
• Current Users of DTN on ISS:– Various “developmental” DTN configurations have been in use on ISS
in support of payload activities for several years – DTN is or has been used in support of the following Payloads – BioServe Commercial-Grade Bioprocessing Apparatus (CGBA) – DARPA Synchronized Position Hold Engage and Reorient
Experimental Satellites (SPHERES) Smartphone – ESA Quickstart and OPSCOM I
• Additional Payloads discussing use of DTN include: – Multiple User System for Earth Sensing (MUSES) – ESA’s METERON (Multi-Purpose End-To-End Robotic Operation
Network)– Human Exploration Telerobotics (HET) Surface Telerobotics – JAXA Kibo KaBand Rain fade mitigation – Other payloads are becoming interested
Boulder POCC
CGBAGSE
Huntsville
HOSC ESOC
ESA ColumbusUS Destiny
TDRSTDRS
ISS payload
LANCGBA
ESA QuickStart-A and METERON topologiesESA QuickStart-A and METERON topologies
C&T / H&S
DTN
DTN-lapGSE
DTN Laptop
DTN Node
DTN Pass-
though
QuickStart-AQuickStart-A METERON METERON
Boulder POCC
CGBAGSE
Huntsville
HOSC ESOC
ESA ColumbusUS Destiny
TDRSTDRS
ISS payload
LANCGBA
METERNGSE
METERNLaptop
SYZYGY Engineering
Spheres
DTN Tunnel
BP
SYZYGY Engineering
Summary
• The implementation of DTN on ISS will provide a standard method of communication for payloads that is reliable, autonomous and more efficient than current techniques, resulting in better utilization and more science return from ISS
• The use of DTN by payloads will significantly ease Payload Development of both onboard and ground communication systems and could reduce payload operator costs
SYZYGY Engineering
Implementation of DTN for Large File Transfers from
Low Earth Orbiting SatelliteWill Ivancic
NASA Glenn Research Center
216-433-3494
36
SYZYGY Engineering
Secure Autonomous Integrated Controller for Distributed Sensor Webs
37
VMOC
NOCNOCNOC
6
Stored data transferred to ground
Sensor
1
Seismic Sensor alerts
VMOC
5
Space Sensor acquires data (e.g. image)
44
4
4
Network Control Center Configures
Spacecraft via VMOCVMOC negotiates for ground station
services
VMOC negotiates for ground station
services
2 2
VMOC negotiates for Space Assets
3
3
Network Control Center Configures
Ground Assets
Network Control Center Configures
Ground Assets
Stored data transferred to ground (Large file transfer over multiple
ground stations)
7
SYZYGY Engineering
General MobilityRegistration and Discovery
38
Internet Alice
(Mobile Node)
Headquarters
(Location Manager)
Bob
(Corresponding Node)
Where is Alice’s
Location Manager?
I am in
Cleveland, Ohio
Hello Alice
Hello Bob,
I am in Cleveland, Ohio
What is the Weather like in
Cleveland?
© 2014Syzygy Engineering – Will Ivancic
For disconnection, this is a hard problem
Home
Agent
(GRC)
Battlefield Operations
(Vandenberg AFB)
Segovia NOC
2nd Ground Station
VMOC-2
(GRC)
SSTL
VMOC-1
Open Internet
VMOCDatabase
Experiments
WorkstationSatellite
Scheduler
& Controller
Rate Mismatch
Problem
Desire is to buffer locally
while in sight of the satellite
then redistribute to the VMOC
<<- Time <<-
Large File Transfer
Over Multiple Ground Stations
- The Problem -
Ideal LARGE Image Transfer – Multiple Ground Stations
Mobile Router
Note, Mobile Router appears to reside on Home Agent’s Network
File Transfer
Using Mobile-IPv4
(Triangular Routing)
Home Agent
VMOC
Open Internet
VMOCDatabase
Satellite
Scheduler
& Controller
Ground
Station 3
Ground
Station 2Ground
Station 1
->> Time ->>
Large File Transfer
Over Multiple Ground Stations
- DTN is a Potential Solution -
DTN Bundle Agent
Intermediary
DTN Bundle Agent
IntermediaryDTN Bundle Agent
Intermediary
DTN Bundle Agent
Sink
SYZYGY Engineering
United Kingdom – Disaster Monitoring Satellite
• The United Kingdom -Disaster Monitoring Constellation (UK-DMC) satellite is an imaging satellite– One of 5 (or 6 or 7 as constellation grows)– Commercial Money Making Operation
• You can request an image (and pay)
• Polar Orbit approximately once every 100 minutes• Satellite is in view of any one ground station for 8 to
14 minutes – hence disruption.• Round Trip Time Delay is ~ 100 msec, thus delay is
not the issue here (unlike for deep space).
41
SYZYGY Engineering
UK-DMC Characteristic
• Onboard experimental Payload, Cisco router in Low Earth Orbit (CLEO) – Not Used for DTN Testing
• Three Solid State Data Recorders– 1 with a StrongARM Processor– 2 with Motorola MPC8260 PowerPC (We use one of these)
• RTEMS operating system (POSIX API, BSD sockets)• Storage Capacity 1 GByte RAM• Operating System Image limit is 0.5 Mbyte
• Uplink is 9600 bits per second• Downlink is 8.134 Mbps• Datalink – Frame Relay/HDLC• Network Protocol – IPv4 (could easily run IPv6)• Transport Protocol (Saratoga version 0 over UDP)
– Saratoga version 0 is existing SSTL transport– Saratoga version 1 is what is in the Internet Drafts
• Enhances version 0 to make it more widely usable 42
SYZYGY Engineering
Saratoga
• Simple High Speed File Transfer Protocol– Replaces CFDP
• Most of the features of CFDP not needed
• CFDP implementation was to slow to fully fill SSTL downlinks
– Implemented for highly asymmetric links
• Asymmetry up to 850:1 for S-Band transmitters
• Asymmetry up to 8333:1 for X-Band transmitters– Negative acknowledge rate-based protocol– Uses UDP at the network layer– Sends Beacon to allow ground station that the space/ground link is
up.
43
SYZYGY Engineering
UK-DMC Implementation
44
Only Bundling and Forwarding
Implemented
Full DTN Protocol Implemented
Surrey Satellite Tech Ltd
(SSTL)
Saratoga
Client
Test 4
SYZYGY Engineering
UK-DMC Flight Code
• Main Satellite Control if via Onboard Computer• Imaging has separate Flight Code residing in Solid State Data Recorder
– RTEMS based– Major Functions
• Control Area Network (CAN) bus interface• Commanding from the Onboard Computer is via CAN bus• Added command for MD5
• Image Capture and Storage• Optional MD5 calculation (added by NASA – Wes Eddy)
• Memory Wash• Bundling Shim (added by NASA – Wes Eddy)• File Transfer (Saratoga in Spacecraft)
• Modified to handle Bundling Shim (Metadata plus offset)
45
SYZYGY Engineering
Ground Station Code
• File Transfer (Saratoga on Ground)– GRC independent PERL implementation that passes
DTN bundles to DTN2 bundle agent• DTN2
– Modified to accept bundles from Saratoga• Named pipe-based convergence layer adapter
– Modified (fixed) early version of DTN2 to operated with very large bundles
• Patch is in current DTN2 implementation• Bundle to File Application
– Single Bundle• Removes Metadata and creates file
– Multiple Fragments• Combines Multiple bundle fragments into a single file
46Put the protocol intelligence and complexity on the ground.
Bundles on UK-DMC
47
Payload
DTN Metadata
Proactive Fragmentation
Metadata - N
Proactive Fragmentation
Metadata - 0
70 - 80 bytes
70 - 80 bytes
70 bytes
70 - 80 bytes
No
t to
Sca
le
150 Mbytes
80 MB
70 MB
For our testing purposes
N=2
(150 MB/80MB)
SYZYGY Engineering
Multi-Terminal Large File Transfers using DTN
• September 30 and October 1, 2009 successfully demonstrated multi-terminal large file transfers using DTN and ground stations in Alaska followed by Hawaii (approximately 80 minute separation)– Demonstrated proactive fragmentation– Demonstrated Store and Forward of ground infrastructure
• Ground station held bundles until routes were established– Demonstrated reactive fragmentation between Hawaii ground station
bundle agent and GRC bundle agent. – Configuration
• UK-DMC acquired a 150 Mbyte image.• DTN bundling code default set to 80 Mbytes for proactive
fragmentation.• September 30 and October 1, 2009 successfully demonstrated multi-terminal
large file transfers using DTN and ground stations in Hawaii followed by Alaska (approximately 5 minutes between passes but effectively overlapping handover)
48
SYZYGY Engineering
49
Sensor data example: The Cape of Good Hope and False Bay.
False colours – red is vegetation.
Taken by UK-DMC satellite on the morning of Wednesday, 27 August 2008. (Downloading this image also demonstrated optional “DTN bundle” use.)
SYZYGY Engineering
Ground Stations and UKDMC Contact Times
50
Multi-Terminal Large File Transfers using DTN
51
September 30 / October 1 Tests
SYZYGY Engineering
Observations
• TCP convergence layer transmission between Hawaii ground station and Cleveland destination was problematic.– Cause has yet to be determined.
• Without reactive fragmentation, these tests would have failed.– If bundle security protocol (BSP) bundle authentication block
(BAB) was used, reactive fragmentation would have failed.– If per-hop reliability checks via the BSP payload
confidentiality block (PCB), or even some other per-hop reliability check, were used, reactive fragmentation would have failed.
• Conclusion: It is desirable to be able to perform reactive fragmentation and still be able to utilize some form of hop-by-hop reliability as well as bundle security.
52
Open Internet
USN
Alaska
bundling-AK
DTN Network(August 17 and 24, 2010)
USN
Hawaii
bundling-HI
USN
Australia
bundling-AU
Universal
Space Network
Intranet
JAMSS/NICT
Koganei, Japan
x.x.34.226
bundling-jamssSSTL
Surrey, England
bundling-SSTL
NASA GRC
Open Net
192.55.90.165
bundling-dtnbone NASA GRC
Closed Net
bundling-grc1
= Hole in Firewall
UK-DMC1
dtn:uk-dmc
DTN Bundle Source
DTN Bundle
Destination
IPsecVNP
Multi-Hop DTN for Security
Reasons
SYZYGY Engineering
August 24, 2010 Passes
54
SYZYGY Engineering
Multi-Terminal Testing with NICT/JAMSS
• Passes over Koganei and SSTL occurred on August 24, 2010 from 22:02 to 22:38. – Both proactive bundle fragments were downloaded to
the Koganei ground station and automatically transferred to NASA Glenn.
– A bundle fragment and the last Syslog file were also downloaded at the SSTL ground terminal.
• A duplicate bundle fragment was received at GRC. DTN2 noted the duplicate and only stored one copy.
55
Successfully showed use of multiple terminals for large image transfer and international interoperability between USA (NASA) UK
(SSTL) and Japan (JAMSS/NICT).