data centre support for the igs-rt pp w. söhne, h. habrich, g. weber federal agency for cartography...
TRANSCRIPT
Data centre support for the IGS-RT PP
W. Söhne, H. Habrich, G. WeberFederal Agency for Cartography and Geodesy,
Frankfurt am Main, Germany
Outline
Introduction IGS-RT PP
• Data and product file centre• File types under consideration
Technical aspects• Data volume• Up- and downloading• Completeness • Limitations
Policy aspects• Upload policy
Conclusions & open questions
Introduction
Storing of highrate files nothing new for IGS• At CDDIS back to 2001, doy121• Initiative from the LEO needs• At IGN• Others
Term “highrate” is a relative one• Usually means 1 Hz sampling rate• But higher sampling rates in use depending on the
application
Currently ~110 stations at CDDIS Clear statement from IGS concerning the
needs of long-term archive still missing
IGS-RT PP (1)
One activity of the IGS-RT PP: Data and product file centre
File types under consideration• Navigational data (broadcast ephemeris)• Observational data
- highrate RINEX files- hourly and daily RINEX files
• Derived files (products)- Orbits or orbit corrections- Clocks or clock corrections- Troposphere & ionosphere parameters
IGS-RT PP (2)
Why generation of highrate files?• Currently, considerable number of users are
“feeding” near real-time applications • For future scientific analyses
What is “near real-time”?• Typically hourly applications, with 20-30 minutes
computation time two times per hour computation possible
Why 15 minutes files?• Reduction of file size compared to an hourly
highrate file• “Convention”
IGS-RT PP (3)
Is that interval small enough? Are there other intervals of interest, e.g. 5 minutes files?• Advantages
- Better distribution of uploading over time
- Other intervals can easily be created/derived
- File naming convention fits
• Disadvantages- Interval size not known in IGS
- Number of files growing
Technical aspects: data volume
Data volume of highrate observational files• Example BUTE115A45.08O: 1 Hz 15 minutes
GPS+GLONASS RINEX file, # obs. types 8, 20 SV in view: 2.4 Mb
• Hatanaka+compress: 170 Kb 5.7 Gb per station and year for the compressed
files >> 500 Gb per year• Selection of IGS stations for high-rate storing?
Data volume of product files• Clocks: 1 Hz, 1 hour: 2.5 Mb, compressed 170 Kb
5.7 Gb per solution and year• …
Technical aspects: up- and downloading
Data upload of observational files• Example 250 stations, 24 files per day (hourly):
6000 files uploaded * 100 Kb 0.6 Gb per day• Example 110 stations, 96 files per day (highrate):
10560 files uploaded * 170 Kb 1.7 Gb per day
Critical aspects are• Is the data centre able to handle all incoming files
within, e.g., 5 minutes after the full hour?If not spreading of upload over a certain time span?!
• Is the data centre able to handle the parallel download requests in peak periods?
Technical aspects: completeness (1)
“Completeness” covers the aspects• Number of observation types• Number of epochs• Number of SVs per epoch• Ratio observed / predicted number of observations
Completeness can be affected by• Outages of single stations data streams• Outages of the broadcasting system• Handling of unhealthy SVs• Limitation of supported observation types
Technical aspects: limitations (1)
“Limitations” cover the following aspects• Resolution of
- Phase
- Code
• Support of - GPS L2C, L5
- GLONASS
- Galileo
- SBAS
Technical aspects: completeness (2)
Different levels of comparison of original and accumulated files possible• Character by character
- Difficult due to roll-over phase values
• Completeness - Epochs missing?
- SVs missing?
• Parallel analysis- Differences between the results
Technical aspects: completeness (3)
Hourly file BUTE115A.08O (TEQC) Highrate file BUTE115A00.08O (BNC)
Technical aspects: limitations (2)
ALME: file from TEQC vs. file from BNC (RTCM 2.3) after RNXSMT (BSW5.0)
Technical aspects: limitations (3)
WTZR: file from TEQC vs. file from BNC (RTCM 3.0) after RNXSMT (BSW5.0)
Technical aspects: completeness (4)
579
80110
5856
62
99,9 -100,0%
99,5 - 99,6%
50,0 - 90,0%
100,0%
99,8 - 99,9%
90,0 - 99,0%
< 50,0%
99,6 - 99,7%
99,0 - 99,5%
99,7 - 99,8%
168
50010
21
35 2199,9 -100,0%
99,5 - 99,6%
50,0 - 90,0%
100,0%
99,8 - 99,9%
90,0 - 99,0%
< 50,0%
99,6 - 99,7%
99,0 - 99,5%
99,7 - 99,8%
0 1
001 0
4
14
2 2 99,9 -100,0%
99,5 - 99,6%
50,0 - 90,0%
100,0%
99,8 - 99,9%
90,0 - 99,0%
< 50,0%
99,6 - 99,7%
99,0 - 99,5%
99,7 - 99,8%
Completeness of 24 hourly high-rateRINEX files from RTIGS streamscoming in via udpRelay
Completeness of 233 hourly high-rateRINEX files from RTCMv2 streamscoming in via NTRIP/TCP
Completeness of 711 hourly high-rateRINEX files from RTCMv3 streamscoming in via NTRIP/TCP
Technical aspects: completeness (5)
Missing SVs G06, G21, G26, G30
Streaming interrupted Longer sections w/o GLONASS
Policy aspects (1)
Station A
Station B
Station C
Station N
Data centre 1
Data centre 2
Data centre 3
User a
User b
User c
User m
Mirroring
Mirroring
„Traditional“ situation transferring hourly files, daily files, ephemeris files
file creation at the station
Policy aspects (2)
Possible situation using real-time data streams for RINEX file creation
Station A
Station B
Station C
Station N
Broadcaster 1 User a
User b
Data centre 1
transferring files transferring streams
or
file creation possible at the data centre, at the users site, at a third party
Conclusions & open questions (1)
Derivation of highrate RINEX files from real-time streams suitable tool
Current interval for highrate files 15 minutes – is that small enough?
Recommendation #1: Storing of 1 Hz 15 minutes files, at least for the long-term archive
Derivation of daily files from streams instead of ftp transfer, at least for new or proposed stations?
RT-IGS PP CfP found five candidates (BKG, CDDIS, GA, KASI, Univ. of Padova)
Conclusions & open questions (2)
Limitations acceptable? If yes, how to point the users to these limitations effectively?
Derivation of RINEX files from real-time streams possible for everyone – needs some regulation or clarification: Who is allowed to upload files derived from real-time data streams?
Recommendation #2: highrate file creation and upload to the GDC in one hand, ideally at the broadcaster’s side