frame selection algorithms for collaboratively tele-operated robotic cameras
DESCRIPTION
Frame Selection Algorithms for Collaboratively Tele-Operated Robotic Cameras. Conventional robotic webcamera. Collaboratively controlled robotic webcamera. One Optimal Frame. Frame Selection Problem: Given n requests, find optimal frame. Requested Viewing Zones. Optimal Satellite - PowerPoint PPT PresentationTRANSCRIPT
1
Frame Selection Algorithms for Collaboratively Tele-Operated
Robotic Cameras
2
Conventional robotic webcamera
Collaboratively controlled robotic webcamera
3
4
5
Frame Selection Problem: Given n requests, find optimal frame
One Optimal Frame
6
Requested ViewingZones
Optimal Satellite Frame
Oct. 27, 2003
7
Satellite Imaging
• 2.44 Billion Market in 2001
• Increasing 14% per year since 1999
• Major clients– Government / Military– Oil Exploration– Weather Prediction– Agriculture
Ikonos, 1999
8
Related Work• Facility Location Problems
– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]
• Rectangle Fitting, Range Search, Range Sum, and Dominance Sum– Friesen and Chan [93] – Kapelio et al [95]– Mount et al [96]– Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Zhang [02]
9
Related Work
• Similarity Measures – Kavraki [98]
– Broder et al [98, 00]
– Veltkamp and Hagedoorn [00]
• CSCW, Multimedia – Baecker [92], Meyers [96]
– Kuzuoka et al [00]
– Gasser [00], Hayes et al [01]
– Shipman [99], Kerne [03], Li [01]
10
Problem Definition• Assumptions
– Camera has fixed aspect ratio: 4 x 3– Candidate frame c = [x, y, z] t
– (x, y) R2 (continuous set)– Resolution z Z
• Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = larger frame = lower resolution
(x, y)3z
4z
11
Problem Definition
Requests: ri=[xli, yt
i, xri, yb
i, zi], i=1,…,n
(xli, yt
i) (xri, yb
i)
12
Optimization Problem
n
iii
zyxcrcsS
1],,[
),(max
User i’s satisfaction
Total satisfaction
13
Problem Definition• “Satisfaction” for user i: 0 Si 1
Si = 0 Si = 1
= c ri c = ri
14
• Measure user i’s satisfaction:
)1),/min(()/(
1,)(
)(min
)(
)(),(
zzap
cResolution
rResolution
rArea
rcAreacrs
iii
i
i
ii
Coverage-Resolution Ratio Metrics
Requested frame ri
Area= ai
Candidate frame c
Area = a
pi
15
Comparison with Similarity Metrics
• Symmetric Difference
• Intersection-Over-Union
SDcrArea
crAreaIOU
i
i
1)(
)(
)(
)()(
crArea
crAreacrAreaSD
i
ii
Nonlinear functions of (x,y), Does not measure resolution difference
16
Optimization Problem
n
iii
zyxcrcs
1],,[
),(max
17
Co-Opticon Problem Versions
• Fixed Resolution Exact Algorithm
• Variable Resolution Exact Algorithm
• Approximate Algorithm for Arbitrarily-Shaped Requested Frame
• Distributed Algorithms
D. Song, A.F. van der Stappen, and K. Goldberg, Exact and Distributed Algorithms for Collaborative Camera Control, the Fifth International Workshop on Algorithmic Foundations of Robotics. Nice, France, Dec 15~17, 2002.
18
),(),( yxpyxs iii
),( yxpi
Requested Frame ri Candidate
Frame c
)1,/min()/(),( zzapcrs iiii
(for fixed z)
Objective Function Properties
19
• si(x,y) is a plateau
• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical
Objective Function for Fixed Resolution
4z x
y
3z
4(zi-z)
20
Objective Function• Total satisfaction:
n
iii
n
iiii
yxpyxS
zzapcS
1
1
),(),(
)1),/min(()/()(
for fixed z
Frame selection problem: Find c* = arg max S(c)
21
S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines.
Objective Function Properties
4z x
y
3z
4(zi-z)
3z y
si
3z
(z/zi)2
3(zi-z)
x
si
4z 4z
(z/zi)2
4(zi-z)
22
Plateau Vertex Definition• Intersection between boundaries
– Self intersection:– Plateau intersection:
y
x
23
Plateau Vertex Optimality Condition
• Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof:– Along vertical boundary, S(y) is a 1D piecewise
linear function: extrema must occur at x boundaries
y
S(y)
24
Fixed Resolution Exact Algorithm
Brute force Exact Algorithm:
Check all plateau vertices (n2) plateau vertices(n) time to evaluate S for each (n3) total runtime
25
Improved Fixed Resolution Algorithm
• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(n) – O(n) 1D problems– O(n2) total runtime
O(n) 1D problems
y
S(y)
x
y
26
Speed comparison
Random inputs
Curve B:
Brute force approach
Curve V:
using line sweeping
27
More Improvements for Fixed Resolution
• Har-Peled, Koltun, Song, and Goldberg. [03] – Exact algorithm O(n3/2 log3 n)
– Near Linear Approximation Algorithm O(NlogN)• N = O(nE)
• E = (log(1/ε)/ε)2, where ε is the approximation bound
S. Har-Peled, V. Koltun, D. Song, and K. Goldberg, Efficient Algorithms for Shared Camera Control, In Proceedings of the 19th ACM Symposium on Computational Geometry, 2003.
28
Co-Opticon Problem Versions
• Fixed Resolution Exact Algorithm
• Variable Resolution Exact Algorithm
• Approximate Algorithm for Arbitrarily-Shaped Requested Frame
• Distributed Algorithms
Dezhen Song, A. Frank van der Stappen, and Ken Goldberg, An Exact Algorithm Optimizing Coverage-Resolution for Automated Satellite Frame Selection, (To appear) IEEE International Conference on Robotics and Automation (ICRA) 2004
29
Virtual Corner• A two-requested frame case
– Requested frame:
y
x
30
Virtual Corner • Virtual corner definition
– Real corner:– Extended edge intersections:
y
x
31
Recall: Plateau Vertex Definition• Intersection between boundaries
– Self intersection:– Plateau intersection:
y
x
32
Virtual Corner and Plateau Vertex• Intersection between boundaries
– Candidate frame:– Frame intersection:
y
x
33
Virtual Corner and Plateau Vertex• Intersection between boundaries
– Candidate frame:– Virtual corner:
y
x
34
Variable Resolution Exact Algorithm
• Lemma: At least one optimal frame has its corner overlapped with virtual corner.– O(n2) Virtual corners– One 3D problem→ O(n2) 1D sub problems
r6
r2
r5
r3
x
y
r4
r1
O0.00
0.40
0.80
1.20
1.60
0 20 40 60 80 100 120 140 160z
S(z)
Candidate frame
35
• Overall complexity– O(n2) 1D problems– O(n) sub 1D problems– O(n) to compute
polynomial coefficient for each sub 1D problem
s(z) = g0z-1+g1+g2z +g3z2
– O(1) to compute the max s(z) for each polynomial
– O(n4) in total 0.00
0.40
0.80
1.20
1.60
0 20 40 60 80 100 120 140 160
S(z)
z
36
0.00
0.40
0.80
1.20
1.60
0 20 40 60 80 100 120 140 160
Improved Variable Resolution Exact Algorithm
• Incremental computing– Computing polynomial
coefficients• O(n) for first smooth
segment, • O(1) for additional• Introduce sorting cost
– O(n log n) for each virtual corner
– O(n3logn) total
S(z)
z
37
Improved Variable Resolution Exact Algorithm
• Diagonal Sweeping– No need to do
sorting for each virtual corner
– O(n) to get new sorted sequence
– Total complexity O(n3)
x
y
O(a)
O
y
r1 r2
x(b)
x(c)
y
x(d)
O O
r1 r2
r1 r2
Order of VCs
y
r1 r2
38
Speed comparison
Random inputs
0
10
20
30
40
50
60
20 30 40 50 60 70 80
Seco
nds
O (n 4 )
O (n 3 logn )
O (n 3 )
n
Brute force approach
Using Incremental computing
Using incremental computing and
diagonal sweeping
39
Co-Opticon Problem Versions
• Fixed Resolution Exact Algorithm
• Variable Resolution Exact Algorithm
• Approximate Algorithm for Arbitrarily-Shaped Requests
• Distributed Algorithms
D. Song, K. Goldberg, and A. Pashkevich, ShareCam Part II: Approximate and Distributed Algorithms for a Collaboratively Controlled Robotic Webcam, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2003.
40
Arbitrarily-Shaped Requested Frame
Requested frames
41
Approximation Algorithm
n)dd
whgO(
spacing resolution :
spacing lattice :
z2
zd
d x
y
d
Compute S(x,y) at lattice of sample points:
w, h : width and height,g: size range
42
Approximation Bound Definitionc* : Optimal frame
: Optimal at lattice (Algorithm output)
c~
)~()( * cscs
?)(
)~(1
*
cs
cs
43
Derive Approximation Boundc* : Optimal frame
: Optimal at lattice (Algorithm output)
c~
: Smallest frame at lattice that encloses c*
c
)ˆ()~()( * cscscs
?)(
)ˆ(
)(
)~(1
**
cs
cs
cs
cs
44
Derive Approximation Boundc* : Optimal frame
: Smallest frame at lattice that encloses c*
c
?)(
)ˆ(*
cs
cs
• fully enclose c*
What is the ratio between their objective functions if one candidate frame is enclosed by the other?
c
45
Approximation Bound
Requested frames
46
Approximation Bound
c
Requested frames
Candidate frame
z
zcrscs
zz
zzap
crs
n
iin
iii
i
iii
ii
1
1
),()(
/
)/)(/(
),(
47
Approximation Bound
ca
cb
Requested frames
Candidate frames
b
n
iib
a
n
iia
zzcs
zzcs
/)(
/)(
1
1
b
a
a
b
z
z
cs
cs
)(
)(
48
Approximation Bound
b
a
a
b
z
z
cs
cs
)(
)(
ca
cb
Requested frames
Candidate frames
49
Derive Approximation Boundc* : Optimal frame
: Smallest frame at lattice that encloses c*
c
z
z
cs
csˆ)(
)ˆ( *
*
What is the resolution ratio between a candidate frame and the smallest frame on the lattice that encloses it?
50
Approximation Algorithm
ca
cb
za
a
b
a
z
dz
z
z
z
dd
2
3set
dz: Lattice spacing in z axis
d
d
51
Approximation Algorithm
– Run Time: – O(n / 3)
c* : Optimal frame
: Optimal at lattice (Algorithm output)
c~
: Smallest frame at lattice that encloses c*
c
)ˆ()~()( * cscscs
)(
)ˆ(
)(
)~(1
** cs
cs
cs
cs
zdz
z
2...
min
min
1
52
Speed Comparison
Random inputs
0
10
20
30
40
50
60
20 30 40 50 60 70
ε=0.2
ε=0.1
ε=0.05
ε=0.035
Exact algorithm
ε=0.2
Time
(Sec.)
#Requests
53
Enclosing the Optimal
c* : Optimal frame
: Frame at lattice that encloses c*
c
z
z
z
z
cs
csˆˆ)(
)( min*
*
54
Cutting in Feasible Set
xy
z)',','(' zyxc
x
y
c’
Screen Space Solution Space
Φ
Φc’
55
Branch and Bound on Lattice
y
z
xkdz
dz
kdz
Layer 1
Layer 2
Layer 3
Survived nodes
Deleted nodes
56
Speed Improvements
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0 0.02 0.04 0.06 0.08 0.1
n=5
n=20
n=40
ε
Grid Basicusing time nComputatio
BnBusing time nComputatio
57
Improvement: Functional Box Sums
• Efficient reporting of
n
i i
ii
n
ii rArea
crAreacscS
11 )(
)()()(
ir
c
jr
kr
[Zhang et al 2002]
58
Origin-Involved Functional Box Sums
=
_ _ +
59
Dominance Sums
13iω
ir(14,8)
(1,3) (9,3)
c)3()1( yxωi
)3()9( yxωi
)(cS )38()114(13
)38()914(13
520
60
Dominance Sum Queries• Data structure:
– ECDF-tree • Guttman (84)
–
–
• ‘Simple’ updates when increasing zoom level
)log( onconstructi 2 nnO)(logquery 2 nO
)log)/1(( 23 nnO
61
Co-Opticon Problem Versions
• Fixed Resolution Exact Algorithm
• Variable Resolution Exact Algorithm
• Approximate Algorithm for Arbitrarily-Shaped Requested Frame
• Distributed Algorithms
62
Distributed Algorithms
63
Distributed Algorithms
• Fixed Resolution Algorithms O(n2)– Server O(nlogn)– Client O(n)
• Approximate Algorithm O(n/3)– Server O(n+1/3)– Client O(1/3)– Robustness to dropouts…