DENSE FEMUR RECONSTRUCTION FROM TWO X-RAY IMAGESUSING GENERIC 3D MODEL WITH TWIST CORRECTION

Ki-jung Kim ?, Seungkyu Lee ? and Yoon Hyuk Kim †

? Dept. of Computer Engineering, Kyung Hee University, Republic of Korea† Dept. of Mechanical Engineering, Kyung Hee University, Republic of Korea

ABSTRACT

Femur 3D reconstruction from x-ray images enables cost-effective and precise diagnosis of disease and surgery guid-ance. However, reconstructing accurate 3D model fromsparse x-ray images are not a trivial task. In this paper, wepropose a femur 3D reconstruction method from two x-rayimages. We incorporate prior knowledge on the femur de-formation such as twist distortion to improve reconstructionaccuracy. We deform our generic 3D model optimizing globalregistration and twist correction parameters simultaneously.And then, non-rigid deformation is performed constructingdense 3d femur model of a patient. We have tested our methodon real x-ray images and synthesized 3d femur models forquantitative and qualitative evaluations.

Index Terms— 3D reconstruction, x-ray, twist, generic3d model, femur, medical.

1. INTRODUCTION

Reconstructing three dimensional model of patient’s boneenables precise diagnosis of disease and surgery guidance.Computerized tomography (CT) obtains volumetric infor-mation of human body and gives 3d volume of any bone.However, it causes high radiation dose, cost and processingcomplexity. Therefore, researchers have tried to reconstruct3d model of human bone from multiple x-ray images. Akkoulet al. [1] present a 3D reconstruction framework of a proximalfemur using multiple pairs of 2D x-ray images. They recom-mend using 8 pairs of x-ray images (total 16 x-ray images).Zheng et al. [2] deform 3d model finding correspondencesbetween 2D features and their surface model. Zhu and Li [3]propose to use statistical shape model of distal femur usingtwo 2d fluoroscopic images. Baka et al. [4] reconstruct distalfemur based on canny edge map for landmark estimation.They register the silhouette image of statistical model to thelandmark edges from x-ray images. Karade and Ravi [5] re-construct 3D femur using only two x-ray images performingLaplacian surface deformation. Mostly, they reconstruct 3Dfemur model based on only two x-ray images, which can beimproved critically if we provide the general shape prior andanalytical deformation information. For example, most pre-

Fig. 1. Proposed Method for Dense Femur 3D Reconstructionusing Generic 3D Model with Twist Correction

vious work reconstructing femur from two x-ray images canonly deal with either proximal or distal femur, due to the lackof the degree of twist distortion amount between proximaland distal femurs.

In this paper, we propose a dense full femur 3D recon-struction method from two x-ray images using generic 3D fe-mur model and prior knowledge on the femur deformation.Based on the two (front and side) x-ray images, we applyglobal deformation (rotation, translation, resizing and bend-ing) and horizontal twist distortion between proximal and dis-tal femurs on our generic 3d model registering it to the in-put x-ray images. After that, our non-rigid registration stepdensely reconstructs 3D femur surface based on the deforma-tion index interpolation on each surface point.

Fig. 2. Global Registration of the Silhouette of the GenericModel to X-ray Images

2. PROPOSED METHOD

Figure 1 summarizes our proposed femur reconstructionmethod with respective example. Our proposed method con-sists of three steps: (1) femur region segmentation from frontand side x-ray images (2) global registration and twist cor-rection of the generic 3D model, and (3) non-rigid dense 3Dregistration. In order to segment femur region in an x-rayimage, we use the random forest regression based methodfor proximal femur segmentation [6]. Lindner et al. [6] pro-pose a proximal femur segmentation method using randomforest. It accurately segments a proximal femur based onthe significant points obtained from a statistical shape model.Segmented proximal femur is put as an initial state, we fur-ther perform an active shape model based segmentation [7]to obtain full accurate femur region in x-ray image. Basedon the segmented two (front and side) x-ray images, globalregistration and twist correction of our generic 3D modelare performed simultaneously followed by non-rigid denseregistration as explained in the following subsections.

2.1. Twist Correction and Global Registration

Figure 2 illustrates our global registration step. In the trans-formation, six parameters (rotation r, translation tx & ty ,resizing sx & sy and vertical bending b) are optimized mini-mizing the distance between the segmented femur from inputx-ray and silhouette of our generic model projected onto thecorresponding direction (front, side). Simple gradient descentmethod is used for the parameter optimization. Global regis-tration is done with front and side x-ray images sequentiallyapplying all calculated transformation on our generic model.

Before we perform the following non-rigid registrationstep, we further investigate input x-ray images to find the dis-

(a) 0o twisted (b) 22o twisted

Fig. 3. Global Registration Distance in different HorizontalTwist Bone Examples

tortion amount of horizontal twist between proximal and dis-tal femurs with respect to our generic model. Figure 3 showssample accuracy test results of two patients having twist an-gle θ = 0o and θ = 22o, respectively. X axis represents hori-zontal rotation angle of our non-twisted generic model and Yaxis represents the distance between x-ray image and globaltransformed model silhouette (non-overlapped area indicatedby gray color is counted at the ”Registration Result” in figure2). Figure 3 (a) shows that the patient has minimum distanceat almost HorizontalRotation = 0o for both proximal anddistal femurs. On the other hand, second patient in figure 3(b) shows different rotation angles for minimum distance (ap-proximately, HorizontalRotation = −12o and 10o for theproximal and distal femurs, respectively) indicating that fullfemur is twisted by corresponding amount of angles. We seethat a patient with twist distortion (figure 2 (b)) should takex-ray images at different rotation deviations for proximal anddistal femurs. In other words, front and side x-ray imagesare not really show the front and side views of the proximaland distal femurs, simultaneously. Without correction of thisdeviation error, reconstructed 3D femur model will includeerrors caused by the discrepancy in the shape of input x-rayand model silhouette.

In order to correct the twist distortion, we iterative per-form the global transformation of the silhouette image withrespect to the horizontal twist angle θ by minimizing the dis-tance between x-ray image and transformed model silhouette,obtaining twist amount θ of each patient and twist correctedgeneric 3D femur model.

2.2. Deformation Index Calculation

As can be seen in figure 3, global registration result after twistcorrection still contains shape discrepancy especially in thetop and bottom regions where, in general, large 3-dimensionalshape variations can be observed over patients. Therefore, weperform non-rigid registration to deform our generic modeldensely guided by the input x-ray images. In order to measure

Fig. 4. Measurement for Deformation Index ρ = dIdM

Calcu-lation

Fig. 5. Deformation Index Interpolation

the non-rigid discrepancy from our global registration result,we calculate deformation index ρ(φ) = dI(φ)

dM (φ) along a groupof fixed angles φ using the measurements shown in figure 4. cis fixed center point, dI(φ) and dM (φ) are the distances fromthe center to boundary along the angle direction of φ for in-put x-ray and model silhouette, respectively. Finally, we getρF (φi) and ρS(φj) for front and side views. Even thoughthe deformation index for proximal and distal femur parts arecalculated using this method, the deformation index for mid-dle body part is simply calculated only with horizontally asshown in the right image of figure 4.

2.3. Non-Rigid Dense 3D Registration

Based on the two deformation index values, we define new in-dex in 3D model ρ3d(φi,j) = ρF (φi)× ρS(φj). Now we ob-tain new 3d location of deformed surface point of the genericmodel ddeformed(φa,b) as follows.

ddeformed(φa,b) = dM (φa,b)× ρ3d(φa,b) (1)

where, ρ3d(φa,b) is obtained by bilinear interpolation offour neighbor indexes ρ3d(φi,j), ρ3d(φi+1,j), ρ3d(φi,j+1),and ρ3d(φi+1,j+1). This calculation deforms globally trans-formed generic model building a dense 3d fumer modelguided by the two x-ray images.

Fig. 6. 3D Reconstruction Results using Real X-ray ImagePairs and Virtual X-ray Image Obtained from SynthesizedFemur Model: Patient #1 has no twist distortion, Patient #2has 25o twist distortion, and the synthesized virtual input has−30o twist distortion.

3. EXPERIMENTAL RESULTS

Our proposed method is evaluated on real x-ray image pairsas well as synthesized x-ray images from 3D model for quan-titative and qualitative evaluations. Figure 6 shows originalgeneric 3D model compared with reconstructed patients’ fe-mur models. All 3D models are delaunay triangulated andPoisson surface reconstruction [8] is applied for the presen-tation. First patient has almost no twist distortion and thereconstructed patient’s model keeps the structure of proximaland distal femurs from the generic 3D model. On the otherhand, second patient shows around 25o twist distortion fromour generic model which is reflected on the reconstructed pa-tient’s model. (See the proximal femur of the side view, thatis rotated further toward the right side and shows differentstructure from the original generic model.)

Figure 7 compares 3d reconstruction results for the secondpatient with and without our horizontal twist correction step.

Fig. 7. Example of with/without Twist Correction of the Pa-tient 2: Without the twist correction step, serious distortion isobserved in the reconstructed 3D femur model.

(a) Error Rate (%) (b) Accuracy Gain

Fig. 8. Reconstruction Error Rate and Accuracy Gain afterTwist Correction: Reconstruction error rate is defined by thetotal # of voxels fall outside of the ground truth model dividedby the total # of voxels of the ground truth model.

Without our twist correction step, deformation index is cal-culated based on the silhouette images obtained from wrongdirection causing serious distortion in reconstructed shape.On the other hand, our twist correction allows transformedgeneric model stretches to correct front and side directions ofproximal and distal femurs in the non-rigid registration step.

For the quantitative evaluation, we synthesize 10 virtual3D femur models. We obtain virtual front and side segmentedx-ray images. And then, using our generic 3D model, we re-construct femur model based on the two virtual segmentedx-ray images. Reconstruction error rate (defined by the total# of voxels fall outside of the ground truth model divided bythe total # of voxels of the ground truth model) is calculatedbetween the reconstructed femur model and original groundtruth virtual model. Reconstruction error rate and accuracygain after twist correction results are summarized in figure 8.In all cases, our twist correction step gives more accurate re-construction result than the results without twist correction.Accuracy improvement gain increases as the twist angle be-comes bigger. We also observe that the error rate of twist cor-rection results does not increase so much as the twist angle isbigger.

4. CONCLUSION

In this paper, we propose a dense femur 3D reconstructionmethod based on two x-ray images (front and side) usinggeneric 3D femur model and prior knowledge on the femurtwist distortion. Our evaluation shows that the proposed twistcorrection significantly improves reconstruction accuracyquantitatively and qualitatively. Our work also suggest that ifwe are able to collect more abundant knowledge on the femurbone shape deformation, even 3D reconstruction based ononly two images can be very precise and useful. Finally, ifyou want to get more related information and materials, youcan find them at http://cvlab.khu.ac.kr/bone.html.

5. REFERENCES

[1] Sonia Akkoul, Adel Hafiane, Eric Lespessailles, andRachid Jennane, “3d femur reconstruction using x-raystereo pairs.,” in ICIAP, 2013, vol. 8157, pp. 91–100.

[2] Guoyan Zheng, Sebastian Gollmer, Steffen Schumann,Xiao Dong, Thomas Feilkas, and Miguel A. GonzlezBallester, “A 2d/3d correspondence building method forreconstruction of a patient-specific 3d bone surface modelusing point distribution models and calibrated x-ray im-ages,” Medical Image Analysis, vol. 13, pp. 883 – 899,2009.

[3] Zhonglin Zhu and Guoan Li, “Construction of 3d humandistal femoral surface models using a 3d statistical de-formable model,” Journal of Biomechanics, vol. 44, pp.2362 – 2368, 2011.

[4] N. Baka, B.L. Kaptein, M. de Bruijne, T. van Walsum,J.E. Giphart, W.J. Niessen, and B.P.F. Lelieveldt, “2d3dshape reconstruction of the distal femur from stereo x-rayimaging using statistical shape models,” Medical ImageAnalysis, vol. 15, pp. 840 – 850, 2011.

[5] V Karade and B Ravi, “3d femur model reconstructionfrom biplane x-ray images: a novel method based onlaplacian surface deformation,” International journal ofcomputer assisted radiology and surgery, pp. 1–13, 2014.

[6] Claudia Lindner, S. Thiagarajah, J. Mark Wilkinson,Gillian A. Wallis, and Timothy F. Cootes, “Fully auto-matic segmentation of the proximal femur using randomforest regression voting.,” IEEE Trans. on Med. Imaging,pp. 1462–1472, 2013.

[7] Gert Behiels, Dirk Vandermeulen, Frederik Maes, PaulSuetens, and P. Dewaele, “Active shape model-based seg-mentation of digital x-ray images.,” in MICCAI. Springer,1999, vol. 1679, pp. 128–137.

[8] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe,“Poisson surface reconstruction,” Eurographics Sympo-sium on Geometry Processing, pp. 61 – 70, 2006.