a sequential online 3d reconstruction system using dense

000

001

002

003

004

005

006

007

008

009

010

011

012

013

014

015

016

017

018

019

020

021

022

023

024

025

026

027

028

029

030

031

032

033

034

035

036

037

038

039

040

041

042

043

044

045

046

047

048

049

050

051

052

053

054

055

056

057

058

059

060

061

062

063

064

065

066

067

068

069

070

071

072

073

074

075

076

077

078

079

080

081

082

083

084

085

086

087

088

089

090

091

092

093

094

095

096

097

098

099

100

101

102

103

104

105

106

107

WACV#****

WACV#****

WACV 2015 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

A Sequential Online 3D Reconstruction System Using Dense Stereo Matching

Anonymous WACV submission

Paper ID ****

Abstract

This paper proposes a sequential online 3D reconstruc-tion system using dense stereo matching for a non-expertuser, which can sequentially reconstruct accurate and dense3D point clouds when the new image is captured. The pro-posed system is based on a novel processing pipeline of se-quential online 3D reconstruction with two key techniques:(i) camera parameter estimation of Structure from Motion(SfM) and (ii) dense stereo correspondence matching us-ing Phase-Only Correlation (POC). The user can confirmthe reconstruction result and add supplementary images tothe system in order to reconstruct a complete 3D model asneeded. Through a set of experiments, the proposed sys-tem exhibits efficient performance in terms of reconstructionaccuracy and computation time compared with the conven-tional system.

1. Introduction

Recently, with the rapid development of 3D printers,which are now available for consumer use, 3D printing ofan object has been receiving much attention. There are stillproblems for non-experts to utilize the 3D printing technol-ogy. One of major problems is to generate a 3D model of atarget object in the real-world environment. Most of prac-tical 3D measurement systems employ laser scanning orstructured light projection to generate accurate 3D models.These systems require special measurement equipment de-pending on the target objects and may be liable to be expen-sive. Also, the application of these systems is limited, sincethe measurement equipment is relatively large. Hence, aneasy-to-use 3D modeling system is indispensable for prac-tical use in daily life.

The topic of reconstructing a 3D model from a set ofimages has attracted much attention as one of approachesto easy-to-use 3D modeling in the field of computer vision.[20, 22, 5]．A dense and accurate 3D modeling algorithmfrom still images has been proposed by Furukawa et al. [9]and a web-based application of 3D modeling from photoshas also been available [2, 1]. Such 3D modeling systems

consist of the offline processes, since all the input imagesare needed to start the 3D modeling process. If the complete3D model is not generated, the supplementary images haveto be taken by the user and all the processes of 3D modelingare performed again. Therefore, the user needs more timeto generate a 3D model and the technological knowledge tounderstand the viewpoint required to generate a complete3D model.

The 3D modeling algorithm with sequential 3D recon-struction from images is comfortable for non-expert users,since the user easily confirms the 3D reconstruction resultand understands the viewpoint required to generate a com-plete 3D model. So far, online 3D modeling algorithmswhere the input for these algorithms is a video sequencehave been proposed [19, 23, 17, 26, 10, 27]. It is hard fornon-expert users to take a high-quality video sequence soas to reconstruct a 3D model of a scene compared with stillimages, since the image quality of video sequence is lowerthan that of still images and special hardware devices suchas storage, processor, etc. are needed to process a largeamount of images. The most difficult problem for non-expert users is to take a video sequence without blur dueto hand movement. Recently, the 3D modeling system onmobile phones has been proposed by Tanskanen et al. [25].For the purpose of real-time 3D reconstruction on mobilephones, the system employs Sum of Absolute Differences(SAD) for depth map computation, whose accuracy is notenough to reconstruct an accurate 3D model from images.Also, there is no quantitative evaluation of reconstructionaccuracy for the system in [25]. To realize an easy-to-use3D modeling system, the 3D model has to be sequentiallyreconstructed from still images, while there is no dense 3Dmodeling algorithm with sequential 3D reconstruction fromstill images to the best of our knowledge.

This paper proposes an easy-to-use sequential 3D recon-struction system which combines camera parameter estima-tion of Structure from Motion (SfM) [24, 11] and dense cor-respondence matching using Phase-Only Correlation (POC)[21]. When a new image is taken, the proposed system esti-mates its camera position using SfM and reconstructs dense3D points using POC from images whose camera position is

1

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

WACV#****

WACV#****


known. Through a set of experiments, the proposed systemexhibits efficient performance in terms of reconstruction ac-curacy and computation time compared with the conven-tional method [9].

The main features of the proposed system are summa-rized as follows:

• The effective pipeline for a sequential online 3D recon-struction system is employed, i.e., the proposed systemexecutes 3D reconstruction as well as image acquisi-tion. A user can confirm the reconstruction result andadd supplementary images to the system in order toreconstruct a complete 3D model as needed.

• The input for the proposed system is still images, sincehigh-quality images can be taken by a consumer digitalcamera even for non-expert users.

• The proposed system employs a POC-based corre-spondence matching method, which allows us to re-construct a high-quality 3D model.

• The proposed system does not need any special equip-ment for camera calibration. It is convenient for non-expert users to use the 3D reconstruction system.

2. System Overview

The proposed system reconstructs accurate and dense 3Dpoint clouds from still images taken by a moving camera ina short time and displays the reconstructed 3D point cloudsat the same time. The proposed system consists of a con-sumer digital camera and a computer as shown in Fig. 1.The user can select the type of the system as usage. In thecase of Fig. 1 (a), convenience for image acquisition can beimproved by using the digital camera or the memory cardwith Wi-Fi, which transfers images to the computer. In thecase of Fig. 1 (b) and (c), the use of a smartphone or atablet computer makes it possible to realize a portable sys-tem, since both a camera and a general-purpose processorare embedded in the system. The input images to 3D re-construction systems have to be in focus and be without anyhalation. The images captured with existing consumer digi-tal cameras satisfy the above conditions, since the aperture,shutter speed and focal length of these cameras are auto-matically configured by their automatic focus and exposurefunctions. Hence, the user can capture suitable images for3D reconstruction without any technological knowledge.

The proposed pipeline for the sequential online 3D re-construction system consists of (i) image acquisition, (ii)pose estimation, and (iii) dense reconstruction as shown inFig. 2. First, the user captures the image Ii from the arbi-trary viewpoint. Second, the system finds correspondencebetween the image Ii and the previous image Ii−1 using

Computer

Digital camera

(a) (b) (c)

Object Object

Smartphone

Object

Tabletcomputer

Figure 1. Examples of 3D reconstruction system: (a) consumerdigital camera and computer, (b) smartphone, and (c) tablet com-puter.

feature-based correspondence matching such as SIFT. Ac-cording to correspondence between I i and Ii−1, the cam-era parameter of Ii is estimated using SfM [24, 11]. Thestereo image pair for the i-th camera position consists ofthe image Ii and the image Ij(i > j) which is close to Ii.The system rectifies stereo images Ii and Ij and finds thedense correspondence between the rectified stereo imagesusing POC [21]. Then, the system reconstructs dense 3Dpoint clouds from Ii and Ij and merges the reconstructed3D point clouds to the whole 3D point clouds. The pro-posed system can simultaneously capture images, updatedense 3D point clouds, and display the reconstructed 3Dmodel by using asynchronous multithread processing of theabove procedure.

3. Pose Estimation

This section describes details of pose estimation used inthe proposed system. The process of pose estimation con-sists of (i) feature-based correspondencematching, (ii) cam-era parameter estimation, and (iii) bundle adjustment. Notethat this process is executed when more than one image isinput, i.e., i > 1.(i) Feature-based correspondence matching

The corresponding point pairs between the images I iand Ii−1 are obtained using feature-based correspondencematching, since the stereo images include various geomet-ric transformation such as scaling, rotation, and nonlineartransformation due to a camera movement and a change offocal length. In the proposed system, we employ Scale-Invariant Feature Transform (SIFT) [16]. SIFT is robustagainst geometric deformation and illumination change be-tween images compared with other feature-based matchingmethods such as Speeded Up Robust Features (SURF) [6]and Binary Robust Invariant Scalable Keypoints (BRISK)[14]. We also empirically confirm that SIFT exhibits effi-cient performance of pose estimation in the proposed sys-tem compared with other feature-based methods. If thenumber of corresponding point pairs is not enough to es-timate pose estimation, the system requests another imageacquisition to the user.(ii) Camera parameter estimation

The camera parameter of Ii is estimated according to thecorresponding point pairs obtained in the previous step.

2

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

WACV#****

WACV#****


Yes

Yes

Imageacquisition

Stereo rectification

Area-basedmatching

3D reconstruction

Feature-basedmatching

Camera parameterestimation

Bundle adjustment

Stereo imagepair selection

No

Dense reconstruction

Poseestimation

# of pointsis sufficient?

is found?

(i)

(ii)

(iii)

Point cloudrefinement

No

Stereo image pair

Figure 2. Pipeline of the proposed sequential online 3D reconstruc-tion system: The i-th camera position is estimated from the imageIi taken by the user and the previous image Ii−1 using SfM. Thestereo image pair for the i-th camera consists of the image Ii andthe image Ij(i > j) which is close to Ii. We rectify the stereo im-ages and find dense correspondence between the rectified stereoimages using POC. We reconstruct 3D point clouds from the cor-responding point pairs and update the whole 3D point clouds.

First, we estimate the intrinsic parameter Ai of the cam-era taking the image Ii. The intrinsic parameter matrix Aof a camera is defined by

A =

⎛⎝

f wDw

0 w2

0 f hDh

h2

0 0 1

⎞⎠ , (1)

where f is the focal length of the camera, w and h are thewidth and height of the image, respectively, andDw andDh

are the width and height of the image sensor, respectively.The focal length f and the image resolution (w, h) are ob-tained from Exif (Exchangeable image file format) infor-mation of the image [12]. The image sensor size (Dw, Dh)depends on the camera and is given in the specification ofthe camera. According to Eq. (1), the intrinsic parametermatrix Ai is calculated from Ii.

Next, we estimate the extrinsic camera parameters of Ii.As for i = 2, we estimate the extrinsic camera parame-ters using the normalized five-point algorithm [18]. As fori > 2, we estimate the extrinsic camera parameters of Iiusing the method proposed by Kneip et al. [13] from thegeometric relation between the 3D points and the 2D coor-dinates of corresponding point pairs. We also employ RAN-dom SAmple Consensus (RANSAC) [7] for robust param-eter estimation.

The 3D points of corresponding point pairs between I iand Ii−1 are reconstructed according to the camera param-eters and triangulation. We refine reconstructed 3D pointsusing the following three techniques. If the baseline lengthof the 3D points which have been already reconstructed be-comes wide by using the image Ii, we reconstruct the 3Dpoints concerned using Ii again to improve the accuracy of3D points. We remove a 3D point having too large or toosmall apical angle between the stereo camera used to recon-struct the 3D point or having too large reprojection error asan outlier. We merge two 3D points whose distance is sig-nificant short into one 3D point having the mean coordinateof the original two 3D points to reduce the computation costof bundle adjustment.(iii) Bundle adjustment

We optimize the reconstructed 3D points and estimatedcamera parameters in nonlinear optimization by minimizingreprojection error using bundle adjustment [11, 24], sincethe accuracy of these parameters has an impact on the suc-ceeding steps. We employ global and local bundle adjust-ments depending on the target range in this paper.

Global bundle adjustment optimizes all the 3D pointsand camera parameters of all the images. Let P = {pi}(1 ≤ i ≤ K) be a set of estimated projection matrices andQ = {qj} (1 ≤ j ≤ L) be a set of coordinates of a recon-structed 3D point, where K is the number of images and Lis the number of 3D points Q. Global bundle adjustment

3

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

WACV#****

WACV#****


minimizes cost function Eg(P ,Q) defined by

Eg(P ,Q) =1

2

K∑i=1

L∑j=1

||mi,j −mrep(pi, qj)||2, (2)

where mi,j is the image coordinate of qj on the i-th image.The mrep(pi, qj) is the image coordinate reprojected fromqj with pi. Since the computational cost of global bundleadjustment significantly increases with increasing the num-ber of camera parameters and 3D points, we iteratively per-form global bundle adjustment at appropriate intervals.

Local bundle adjustment optimizes the camera parame-ters of the i-th image and 3D points observed in the positionof the i-th image. Let pi be an estimated projection matrixof the i-th image and Q′ = {q′

j} (1 ≤ j ≤ L′) be a setof coordinates of reconstructed 3D points observed in theposition of the i-th image, where L ′ is the number of 3Dpoints Q′. Local bundle adjustment minimizes cost func-tion El(pi,Q

′) defined by

El(pi,Q′) =

1

2

L′∑j=1

||mj −mrep(pi, q′j)||2. (3)

Since the computational cost of local bundle adjustment islow, we iteratively perform local bundle adjustment after es-timating the camera parameters. We employ the Levenberg-Marquardt (LM) method, which is one of the nonlinearleast-squares optimization methods, in these nonlinear opti-mization processes.

The optimized reconstructed 3D points and estimatedcamera position and poses are displayed when the pose es-timation process is finished.

4. Dense Reconstruction

This section describes dense 3D reconstruction from im-ages whose position is obtained in pose estimation. Theprocess of dense reconstruction consists of (i) stereo im-age pair selection, (ii) stereo rectification, (iii) dense stereomatching and 3D reconstruction, and (iv) point clouds re-finement.(i) Stereo image pair selection

Dense 3D reconstruction is executed between the stereoimage pair having small perspective distortion so as toprevent the quality of 3D reconstruction from decreasingby dense correspondence matching. We select one imageIj(j < i) whose position is closest to the image Ii andwhich is captured with the parallel optical axis, accordingto the following procedure. The adjacency of In against Iiis defined by

θn = (ri · rn)(di · dn), (4)

where dn and rn correspond to the unit vector along theoptical axis of the image and the unit vector in the direc-tion from the image to the centroid of the 3D points recon-structed using SfM, respectively. We select the image Ijsatisfying the following condition:

θlower < θj < θupper, (5)

where θlower and θupper indicate the lower bound and upperbound of the nearness, respectively. If there is no imagesatisfying Eq. (5), the dense reconstruction process is notexecuted for the image Ii.(ii) Stereo rectification

To obtain accurate and dense 3D points, we employ anarea-based correspondence matching method. However, itis hard for an area-based method to obtain correspondencebetween the stereo image pair having large perspective dis-tortion. Hence, we reduce the perspective distortion be-tween the stereo image pair Ii and Ij by stereo rectifica-tion. Stereo rectification is to transform an image pair asif the image pair is captured with a parallel stereo camera[24], that is, the scaling in vertical direction and rotationbetween the image pair are reduced and the geometric de-formation between the image pair is also limited to horizon-tal direction. Note that the correspondence search betweena stereo image pair is reduced to 1D search by stereo rectifi-cation. The rectified stereo image pair I ′

i and I ′j is obtainedby transforming Ii and Ij with the homography matrix cal-culated from the camera parameters.(iii) Dense stereo matching and 3D reconstruction

We obtain dense corresponding point pairs between rec-tified stereo images I ′i and I ′j using an area-based corre-spondence matching method. Unlike the feature-based cor-respondence matching method, the area-based correspon-dence matching method can obtain the point on I ′

j corre-sponding to the reference point placed on the arbitrary po-sition in I ′i . Hence, when many reference points are placedon I ′i , the dense corresponding points can be obtained so asto measure the fine 3D structure of the object.

Among area-based correspondence matching methods,the proposed system employs POC [21], since it can esti-mate translational displacement between images with sub-pixel accuracy and is robust against illumination changebetween images. This advantage is suitable for change oflighting condition and brightness change due to automaticgain selection of the camera. The use of POC makes itpossible to reconstruct the accurate 3D shape of the object,since coordinates of corresponding point pairs are with sub-pixel accuracy according to the analytical correlation peakmodel of POC function. POC also provides the matchingscore between local image blocks from the maximum cor-relation peak value of the POC function.

We reconstruct dense 3D points from correspondingpoint pairs between Ii and Ij according to triangulation.

4

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

WACV#****

WACV#****


(a)

(b)

18 images

28 images

Figure 3. Data sets and ground truth data used in the experiments:(a) Cat and (b) Dog

We remove 3D points having large variance of the distanceamong nearest K-3D points as an outlier.(iv) Point clouds refinement

The 3D point clouds Ci reconstructed in the previousstep are merged into the 3D point clouds C which havebeen already reconstructed. Both 3D point clouds C i andC are represented by octrees, where the smallest size of theoctree cell is the mode of the distance between the nearest-neighbor points. For each octree cell, only 3D points hav-ing the maximum matching score calculated in dense stereomatching using POC are held and others are removed. Wecalculate the coordinates of the whole 3D point clouds C ,if the camera parameters estimated before the image indexi are updated by global bundle adjustment in pose estima-tion. The updated whole 3D point cloud is displayed whenthe dense reconstruction process is finished.

5. Experiments and Discussion

This section describes a set of experiments to evaluateperformance of the proposed system using data sets takenby a moving camera. The performance of the proposedsystem is compared with the conventional system whichconsists of (i) camera parameter estimation using SIFT-based SfM and (ii) dense reconstruction using the Patch-based Multi-View Stereo (PMVS) algorithm proposed byFurukawa et al. [9].

5.1. Experimental Condition

The target objects are figurines of a catwith W30cm×D30cm×H10cm and a dog withW20cm×D15cm×H20cm as shown in Fig. 3. Weuse a consumer digital camera (Panasonic LUMIX DMC-GF6) with 1, 280× 960 color pixels. We use 18 images forthe cat and 28 for the dog. These data sets are taken in ad-vance, since the conventional system is based on the offlineprocess. The camera and the target object are about 1mapart. The initial values of the intrinsic camera parametersare estimated using Exif information of captured images. A3D mesh model for each target object is measured with thelaser scanner (Steinbichler COMET 5) as shown in Fig. 3to quantitatively evaluate the performance.

3D point clouds are reconstructed from the captured im-ages using the conventional and proposed systems. We as-sume that the image acquisition time per image is 2 seconds.As for the proposed system, the images are input with 2-second interval. As for the conventional system, the totaltime of image acquisition is added to the processing time,since the conventional system is not online. We can cre-ate a mask to separate the background regions from the im-age by simple subtraction, since the images are acquiredagainst a blue background. The accuracy of reconstruc-tion is evaluated by comparing the reconstructed 3D pointclouds and the ground-truth mesh model using the IterativeClosest Point (ICP). Note that not only rotation and transla-tions but also scale are estimated with ICP, since the scaleof the reconstructed 3D point clouds is indefinite.

We implement the conventional system using VisualSfM [28] to estimate the camera parameters and CMVS2[8] for dense reconstruction. The input images are all theimages, where the blue background is masked by black. Thelevel of image pyramid for CMVS2 is 0, the size of match-ing window is 7 × 7 pixels, the threshold of the correlationvalue is 0.6, and the size of the cells is 2 × 2. The otherparameters are the same as those in Furukawa et al. [9].

We implement the proposed system using C++. SIFT-based correspondence matching is implemented usingOpenCV [3]. The threshold of reprojection error forRANSAC is 0.5 pixels and the maximum number of iter-ations for RASAC is 100. The threshold of reprojectionerror and the minimum apical angle for outlier removal inSfM are 1.0 pixels and 3 degrees, respectively. Global andlocal bundle adjustment are implemented using Sparse Bun-dle Adjustment (SBA) [15]. Note that global bundle adjust-ment is executed when RMS of the reprojection errors ofall the camera is more than 0.5 pixels or RMS of the repro-jection error is below 0.5 pixels for the 5th time in a rowto frequently execute global bundle adjustment. The thresh-old for stereo image selection is 0.97 for θ lower and 0.99 forθupper. The interval of reference points placed on the im-age is 2 pixels. The size of window for POC-based imagematching is 32 × 32 pixels, and the threshold of the peakvalue is 0.6. We use Point Cloud Library (PCL) [4] to dis-play the reconstructed 3D point clouds and the estimatedcamera positions, remove outliers of 3D point clouds usinga statistical outlier removal filter, and merge 3D point cloudsusing octree. For outliner removal using the statistical fil-ter, the number of the nearest-neighbor points is 30 and thethreshold of the distance variance is 2. Both systems areimplemented on Windows 7 Professional, Intel R© Core

TMi7-

990X (3.47GHz), RAM 24 GB.

5.2. Experimental Results

Fig. 4 shows the reconstructed 3D point clouds and er-ror maps for the conventional and proposed systems. “Er-

5

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

WACV#****

WACV#****


Comventinal System Conventional SystemProposed System Proposed System

Figure 4. Reconstructed 3D point clouds (1–2 columns) and error maps (3–4 columns) whose range is from blue (0 mm) to red (3 mm).

Table 1. Summary of experimental results. The error in the bracket indicates the error calculated using 3D points that have error below 3mm. The number of points in the bracket indicates the rate of 3D points that have error below 3 mm.

Data setConventional system Proposed system

Error [mm] # of points Time [sec.] Error [mm] # of points Time [sec.]Cat 0.44 (0.43) 334,589 (99.97%) 175 0.47 (0.47) 345,433 (99.98%) 43Dog 19.89 (0.87) 203,355 (98.11%) 173 0.67 (0.64) 188,904 (99.69%) 52

Computation time [sec.]0

1

10

5

15

1

10

20

25

15

5

5040302010

Computation time [sec.]0 50 50 6040302010

Imag

e in

dex i

Imag

e in

dex i

Image loadingFeature detection

Feature matchingCamera parameter estimation

Dense matching and reconstructionPoint cloud refinement

Cat

Dog

Figure 5. Detailed computation time for each process in the proposed system.

ror” in Table 1 summarizes the Root Mean Square (RMS)of reconstruction errors. The reconstruction accuracy of theproposed system is better than that of the conventional sys-

tem. Hence, the quality of 3D point clouds reconstructedby the proposed system is sufficiently-high, since the 3Dreconstruction method used in the conventional system is

6

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

WACV#****

WACV#****


(a)

(b)

~ image 5 (11 sec.) ~ image 10 (25 sec.) ~ image 15 (34 sec.) ~ image 18 (43 sec.)

~ image 7 (16 sec.) ~ image 14 (32 sec.) ~ image 21 (46 sec.) ~ image 28 (52 sec.)Figure 6. Camera position and dense 3D point clouds displayed in the proposed system: (a) Cat and (b) Dog. (blue: unselected camera,cyan: selected camera, red: the final camera position).

known as one of the accurate ones. From the results of theproposed system as shown in Fig. 4, reconstruction error isincreased around the intersection of 3D points reconstructedin the first and last time. Multiple 3D point clouds may notbe completely overlapped, since accumulative error of cam-era pose estimation is increased with increasing the numberof images. To address the above problem, we can add loopclosing in pose estimation for robust parameter estimationand ICP in the post processing for accurate 3D reconstruc-tion of the whole shape of the object. “# of points” in Table1 shows the number of reconstructed 3D points. The pro-posed system can reconstruct a number of 3D points com-parable with the conventional system.

“Time” in Table 1 shows the computation time to recon-struct the whole 3D shape for each object. Fig. 5 showsdetailed computation time for each process in the proposedsystem. The computation time of the proposed system isabout one-third of that of the conventional system to recon-struct the whole 3D shape of the object. When a new imageis input to the system, the conventional system has to dodense reconstruction using PMVS again, in other words, ittakes additional hundreds seconds for the conventional sys-tem. On the other hand, the proposed system takes a fewseconds to do dense reconstruction as shown in Fig. 5, sincethe proposed system hides the processing time of compu-tationally expensive processes such as SIFT-based featurematching and dense 3D reconstruction using multithreadprocessing. Hence, the proposed system can provide an in-teractive interface for users through a process of 3D model-ing as shown in Fig. 6.

Fig. 7 shows other interesting examples of 3D pointclouds reconstructed from images taken by the consumerdigital camera. Settings are the same in the above exper-iments both for conventional and proposed systems. Theproposed system can reconstruct dense and accurate 3Dpoint clouds of all the objects.

As observed in the above experiments, the proposed sys-

tem exhibits efficient performance compared with the con-ventional system. Also, the use of the proposed systemmakes it possible to realize online 3D modeling for non-expert users.

6. Conclusion

This paper has proposed a sequential online 3D recon-struction system using dense stereo matching, which cansequentially reconstruct 3D point clouds when a new imageis taken. Through a set of experiments, we have demon-strated that the proposed system exhibits efficient perfor-mance in terms of reconstruction accuracy and computa-tion time compared with the conventional system. In futurework, we will develop an interactive interface so as to sup-port image acquisition by users. Also, we will reconstructlarge-scale 3D models of a variety of scenes using the pro-posed system.

References

[1] Agisoft PhotoScan — www.agisoft.ru. http://www.agisoft.ru/products/photoscan.

[2] Autodesk 123D Catch — 3D model from photos. http://www.123dapp.com/catch.

[3] Open Computer Vision Library. http://sourceforge.net/projects/opencvlibrary/.

[4] Point Cloud Library. http://pointclouds.org/.[5] S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and

R. Szeliski. Building Rome in a day. Proc. Int’l Conf. Com-puter Vision, pages 72–79, Oct. 2009.

[6] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-uprobust features (SURF). Computer Vision and Image Under-standing, 110(3):346–359, 2008.

[7] M. A. Fischler and R. C. Bolles. Random sample consensus:A paradigm for model fitting with applications to image anal-ysis and automated cartography. Comm. ACM, 24(6):381–395, 1981.

[8] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. To-wards internet-scale multi-view stereo. Proc. Int’l Conf.

7

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

WACV#****

WACV#****


(a)

(b)

(c)

Input images Reconstructed 3D point cloudsEstimated camera positions

26 images

27 images

28 images

Figure 7. Reconstruction results from images taken by a consumer digital camera: (a) Cat with 26 images, (b) Rabbit with 27 images and(d) metal component with 28 images (First column: input images, 2–3 column: reconstructed 3D point clouds using the proposed system).

Computer Vision and Pattern Recognition, pages 1434–1441, June 2010.

[9] Y. Furukawa and J. Ponce. Accurate, dense, and robust mul-tiview stereopsis. IEEE Trans. Pattern Analysis and MachineIntelligence, 32(8):1362–1376, Aug. 2010.

[10] G. Graber, T. Pock, and H. Bischof. Online 3d reconstructionusing convex optimization. In Computer Vision Workshops(ICCV Workshops), 2011 IEEE International Conference on,pages 708–711. IEEE, 2011.

[11] R. Hartley and A. Zisserman. Multiple View Geometry. Cam-bridge University Press, 2004.

[12] Japan Electronics and Information Technology IndustriesAssociation. Exchangeable image file format for digital stillcameras. http://www.jeita.or.jp/.

[13] L. Kneip, D. Scaramuzza, and R. Siegwart. A novelparametrization of the perspective-three-point problem for adirect computation of absolute camera position and orienta-tion. pages 2969–2976. IEEE, 2011.

[14] S. Leutenegger, M. Chli, and R. Y. Siegwart. Brisk: Binaryrobust invariant scalable keypoints. pages 2548–2555. IEEE,2011.

[15] M. I. A. Lourakis and A. A. Argyros. SBA: A software pack-age for generic sparse bundle adjustment. ACM Trans. Math.Software, 36(1):1–30, Mar. 2009.

[16] D. G. Lowe. Distinctive image features from scale-invariantkeypoints. Int’l J. Comput. Vision, 60(2):91–110, Nov. 2004.

[17] R. Newcombe. DTAM: Dense tracking and mapping in real-time. Proc. Int’l Conf. Computer Vision, 2011.

[18] D. Nister. An efficient solution to the five-point relative poseproblem. IEEE Trans. Pattern Analysis and Machine Intelli-gence, 26(6):756–770, 2004.

[19] M. Pollefeys, D. Nister, J. M. Frahm, A. Akbarzadeh, P. Mor-dohai, B. Clipp, C. Engels, D. Gallup, S. J. Kim, P. Mer-rell, C. others Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang,

H. Stewenius, R. Yang, G. Welch, and H. Towles. Detailedreal-time urban 3d reconstruction from video. InternationalJournal of Computer Vision, 78(2-3):143–167, 2008.

[20] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski. A comparison and evaluation of multi-viewsstereo reconstruction algorithms. Proc. Int’l Conf. ComputerVision and Pattern Recognition, pages 519–528, June 2006.

[21] T. Shibahara, T. Aoki, H. Nakajima, and K. Kobayashi.A sub-pixel stereo correspondence technique based on 1Dphase-only correlation. Proc. Int’l Conf. Image Processing,5:V–221–V–224, 2007.

[22] C. Strecha, W. von Hansen, L. V. Gool, P. Fua, and U. Thoen-nessen. On benchmarking camera calibration and multi-viewstereo for high resolution imagery. Proc. Int’l Conf. Com-puter Vision and Pattern Recognition, pages 1–8, June 2008.

[23] J. Stuhmer, S. Gumhold, and D. Cremers. Real-time densegeometry from a handheld camera. In Pattern Recognition,pages 11–20. Springer, 2010.

[24] R. Szeliski. Computer Vision: Algorithms and Applications.Springer-Verlag New York Inc., 2010.

[25] P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer,and M. Pollefeys. Live metric 3D reconstruction on mobilephones. Proc. Int’l Conf. Computer Vision, pages 65–72,2013.

[26] G. Vogiatzis and C. Hernandez. Video-based, real-timemulti-view stereo. Image and Vision Computing, 29(7):434–441, 2011.

[27] A. Wendel, M. Maurer, G. Graber, T. Pock, and H. Bischof.Dense reconstruction on-the-fly. pages 1450–1457, 2012.

[28] C. Wu. VisualSFM: A Visual Structure from Mo-tion System. http://homes.cs.washington.edu/

˜ccwu/vsfm/.

8

a sequential online 3d reconstruction system using dense

Documents