october 8, 2010 - session 2refag2010 paris1 distributed processing systems for large geodetic...

October 8, 2010 - session 2 REFAG2010 Paris 1

Distributed processing systemsfor large geodetic solutions

IAG WG 1.1.1“Comparison and combination of precise orbits

derived from different space geodetic techniques”

Henno Boomkamp


Introduction• Objectives of IAG WG 1.1.1

1. Study systematic errors between orbits based on different data2. Improve models and solution strategies3. Develop the means to allow the above

• Key solution mechanism: simultaneous analysis– simultaneous (re-)processing of geodetic datasets is either free of

biases, or allows estimation & analysis of inter-system biases

• Concrete targets– Analysis of all GPS / GNSS data in a single solution Dancer– Analysis of all geodetic data in a single solution Digger


POD m

odule

Digger (… reprocessing)- data types: all- long arc (years)- long latency (1 week)- key technology: BOINC

Digger (… reprocessing)- data types: all- long arc (years)- long latency (1 week)- key technology: BOINC

Consistency among techniques

Dart (Dancer-RTK)- data types: GNSS only- very short arc (RT)- zero latency- key technology: BURST

Dart (Dancer-RTK)- data types: GNSS only- very short arc (RT)- zero latency- key technology: BURST

Real-time access to accurate ITRF

ITRF

Dancer- data types: GNSS & VLBI- short arc (48 hrs) - short latency (30 min)- key technology: JXTA

Dancer- data types: GNSS & VLBI- short arc (48 hrs) - short latency (30 min)- key technology: JXTA

Large scale, direct access ITRF

Three WG projectsEffort: Dancer 90 %

Digger 9 % Dart 1 %


The trouble with GPS• Limited processing capacity of AC

– Current IGS approach would require hundreds of Analysis Centres …

• Most data not available at short latency– Reduces statistical quality ITRF– Reduces relevance of ITRF

20,000 permanent GPS receivers

5,000 public

400ITRF

• If we want to run a conventional batch LSQ solution for all receivers– Distribution over many computers is inevitable– Geographical separation of the computer cluster is inevitable

• If we want to run a conventional batch LSQ solution for all receivers– Distribution over many computers is inevitable– Geographical separation of the computer cluster is inevitable


10 GPS sites = 10 PC = 10 AC


Dancer overviewDancer brings the analysis to the data rather than vice versa • LSQ solution implemented as a peer-to-peer process on the internet• Based on existing JXTA P2P software (SUN Microsystems)

Natural separation of analysis is by receiver, not by AC• Geographical distribution of data is at the level of receivers• Solution becomes scalable in the number of stations• 99% of estimated parameters can be pre-eliminated at receiver level

… the required computers are readily available!• Every permanent receiver is connected to a local or remote computer• Most of these computers do not do anything apart from RINEX ftp• Processing capacity is perfectly collocated with the data owners


Separation of LSQ in tasks per receiver (1)Local parameters are pre-eliminated at the receiver:

N

A

N

A

NtN

t

tN

y

y

yy

x

x

xx

M

M

M

B

B

BBBBA

2

1

2

1

2

1

2

1

21

.

00

0

0

00Ax

global (orbits, sat clocks, pole)

jx

local j = 1 … N

N

jjAA

1

N

jAjA yy

1

AtjjjjjjjA

tj xByMxyxMxB

1

N

jjjjAjA

N

j

tjjjj yMByxBMBA

1

1

1

1

N

jjA

N

jjjA

N

jj yxBxA

111

With 10,000 receivers:Total solution• 30 million parameters• 170 million observationsAfter pre-elimination• 90,000 global parametersSingle process• 15,000 non-zero eq.

With 10,000 receivers:Total solution• 30 million parameters• 170 million observationsAfter pre-elimination• 90,000 global parametersSingle process• 15,000 non-zero eq.


Global normal equation represents the average equation of all receivers

Dancer averages diagonal D before solution, and the vector afterwards

Same solution , thanks to distributive property of multiplication with

Separation of LSQ in tasks per receiver (2)

N

jj

N

jj bxM

11

N

1

N

1bxM~~

N

jj

N

jj FD

NFDMM

N 11

~1~~~1

jjjj xbxFD

~xxx

N

N

jj

~1

1

xFbDxbxFD ~~~~~~ 1

x

jjjjjjj xFbDxbxFD

1~~

x 1~ D


Averaging N vectors without a central serversquare dance algorithm

• N/2 pairs can be formed by toggling one bit of each number 1...N

011010110 111010110• Pair-wise exchange of vectors: both computers find the same sum

x11010110 first bit has now become irrelevant…

• New exchange pairs are formed by toggling the second bit

011010110 101010110

xx1010110 first two bits are now irrelevant; etc…

• Some additional operations are necessary:– folding nodes are introduced to make N an exact power of two

– N splits into 50% core nodes and 50% spare nodes for contingencies

After log2N exchange cycles, all N computers have the same vectorAfter log2N exchange cycles, all N computers have the same vector


core node data volume (1-way)

0.0

50.0

100.0

150.0

200.0

250.0

1 10 100 1000 10000 100000

48h / 30s

24h / 30s

48h / 300s

24h / 300s

arc length/ epoch rate

MB

N


Sched beta

Dancer project status


Summary• Rigorous LSQ solutions for all GPS receivers are possible

– workload can be distributed over (some) existing hardware• Dancer has no data centres, analysis centres, combination centres, product centres, central bureau…

– anonymous participation avoids political issues of data access– differences regional vs. global reference frames disappear

• GNSS receivers become smart receivers– Dancer process can be embedded on future receiver hardware– Smart receiver generates products, not (just) observations

• Other distributed processes to follow• DIGGER simultaneos reprocessing of all geodetic techniques• DART RTK layer on top of Dancer for global access


www.GPSdancer.com

.. .

..

. . . .

... .

..

. ..

.

..

.

..

.

.

..

.

..

.

.

.

.

.

.

.

.

.

for • further information• more details on solution mathematics• download latest version of the software• check project status• e-mail contacts and web links

for • further information• more details on solution mathematics• download latest version of the software• check project status• e-mail contacts and web links

Dancer screenshot

october 8, 2010 - session 2refag2010 paris1 distributed processing systems for large geodetic...

Documents

data centres

solution strategiesdevelop

total solution

data ownersseparation

n computers

gps gnss data

single solution danceranalysis

receiver level