Toward Automated Tissue Retraction in Robot-Assisted Surgery

Sachin Patil and Ron Alterovitz

Abstract—Robotic surgical assistants are enhancing physicianperformance, enabling physicians to perform more delicate andprecise minimally invasive surgery. However, these devices arecurrently tele-operated and lack autonomy. In this paper, wepresent initial steps toward automating a commonly performedsurgical task, tissue retraction, which involves grasping and liftinga thin layer of tissue to expose an underlying area. Given a modelof tissues in the vicinity, our method computes a motion plan fora 6-DOF gripper that grasps a tissue flap at an optimal locationand retracts it such that an underlying target is fully visible.The planner considers three optimization objectives relevantto medical applications: minimizing the maximum deformationenergy, minimizing maximum stress, and minimizing the controleffort in lifting the tissue flap. The planner can be used to locallyimprove physician specified retraction trajectories based on theoptimization criteria or to compute a de novo plan. We use aphysically-based simulation to compute equilibrium configura-tions of the tissue flap subject to manipulation constraints. Theseconfigurations are used with a sampling-based planner to explorethe space of deformations and compute an optimal plan subjectto discretization and modeling error. Our experimental resultsillustrate the ability of the method to compute retractions forheterogeneous tissues while avoiding obstacles and minimizingtissue damage.

I. INTRODUCTION

Robotic surgical assistants (RSAs), such as Intuitive Surgi-cal’s da Vinci R© system [8], are enhancing physician perfor-mance, enabling them to perform more delicate and preciseminimally invasive surgery. Studies have shown that theserobots can improve procedure success rates, reduce bleed-ing, and decrease recovery time [9]. As a result, hospitalsare increasingly adopting RSAs, with over 1,000 installedworldwide [8]. In their current form, RSAs are tele-operated;the surgeon performs the medical procedure using an inputdevice outside the patient, and the robotic device duplicatesthe motions of the input device inside the patient (possibly ata different scale and with smoothed motions). Because of thedependence on tele-operation, clinically-used RSAs currentlylack autonomy and require direct control by physicians.

Integrating motion planning with RSAs has the potentialto enable RSAs to perform certain motions autonomously orto improve physician-specified trajectories to minimize tissuestress and forces. In this paper, we focus on a relatively simplebut commonly performed task: tissue retraction. The goal isto manipulate an outer layer of tissue to provide the physicianwith a line of sight to an area of interest underneath whileavoiding contact with obstacles and nearby sensitive structures(see Fig. 1). The manipulation must balance competing objec-tives: provide sufficient exposure and avoid excessively largeforces that damage the tissues.

S. Patil and R. Alterovitz are with the Department of Com-puter Science, University of North Carolina at Chapel Hill, USAsachin@cs.unc.edu, ron@cs.unc.edu

Area of interest

Gripper

Obstacle

Fig. 1. During tissue retraction, the tissue flap (initially flat) is manipulatedby the gripper such that the area of interest (marked by the red circle) iscompletely visible from a given camera viewpoint (left frame). An obstacle(protruding from left in left frame, shown at bottom from a differentperspective in right frame) must be avoided during retraction.

Enabling RSAs to autonomously perform sub-procedureslike tissue retraction could have a significant impact on patientcare. Adding some level of autonomy has the potential toreduce surgical errors by enabling the physician to focus on theimportant, challenging aspects of a procedure rather than beingdistracted by motion control of devices. This would also enableRSAs to utilize more than two manipulators simultaneously,enabling greater dexterity and faster procedure completiontimes. Also, the absence of force-feedback in most currenttele-operated surgical systems hampers physicians’ ability tocorrectly estimate crucial physical quantities such as exertedforces and stress buildups in the deformed tissue. Underesti-mating these quantities during a procedure may cause tissuedamage, resulting in side effects and longer recovery times.RSAs utilizing a motion planner with an embedded physically-based model could improve physician-specified trajectories tominimize unnecessary tissue damage.

We present an approach to compute plans for automatedtissue retraction using a generic 6-DOF gripper. Our methodcan either be used for fully automating the tissue retractiontask (global optimization) or for improving physician spec-ified retraction trajectories (local optimization). We considerthree optimization objectives relevant to medical applications:minimizing the maximum deformation energy, minimizingmaximum tissue stress, and minimizing the cumulative controleffort. The method relies on a physically-based simulationto compute equilibrium states of the tissue flap for sampledgripper configurations. These states are used to construct aglobal roadmap to explore the space of deformations andcompute an optimal plan that avoids obstacles. To computethe portion of a target area visible beneath the tissue flap, weuse a fast visibility test using occlusion queries on a graphicsprocessing unit (GPU). Our planner computes trajectories forthe gripper to accomplish the tissue retraction task whileavoiding obstacles, as shown in Fig. 1.

This planner is an initial step toward automating the taskof tissue retraction during robot-assisted surgery. Ultimately,we plan to improve planner performance and integrate it withpre-operative medical imaging and RSA hardware.

II. RELATED WORK

Robotic Surgical Assistants (RSAs): Tele-robotic or mas-ter/slave systems, such as the da Vinci R© system [8], operateunder direct control of the surgeon and offer improved pre-cision and dexterity for minimally invasive procedures com-pared to non-robotic approaches [8], [15]. Some autonomoussystems have been successfully used for performing surgerieson non-deformable tissue (such as bones) when detailed quan-titative pre-operative plans of the procedure are available [29].Our approach proposes to autonomously perform the task oftissue retraction involving deformable tissues.Modeling of Deformable Objects: Physically-based simu-lation of deformable objects is a well-studied area in solidmechanics [30] and computer graphics [22]. The choice of theunderlying model is application-specific and has a substantialinfluence on the physical accuracy of the estimated deforma-tions. Finite Element Methods (FEM) are based on the theoryof continuum mechanics [30] and simulate deformations accu-rately at the cost of increased computational complexity whenmodel parameters are known. Nonlinear FEM models arepreferred when the correctness of the simulation is importantand have been successfully used for simulating soft tissue forsurgical simulations [6], [24], [23].Motion Planning/Manipulation of Deformable Objects:Research in robot motion planning and manipulation has his-torically focused on robots composed of rigid links operatingin environments with rigid objects [20]. Recent work hasbegun to explore motion planning for deformable robots instatic environments. Lamiraux et al. [19] developed the f-PRM framework to plan paths for flexible robots of simplegeometric shapes such as surface patches or simple volumetricelements. Bayazit et al. [3] use a two-tier approach basedon probabilistic roadmaps (PRM) and free-form deformation(FFD) to plan paths for deformable robots. Gayle et al. [12]use a constraint-based motion planner for deformable objectsmodeled as mass-spring systems. Rodriguez at al. [25] userapidly exploring random trees (RRT) to plan for robots incompletely deformable environments. Frank et al. [11] use co-rotational linear FEM with a PRM-based planner to achievesignificant speedups for path planning in deformable environ-ments. Motion planning algorithms have also been developedfor clinical applications, including deformable robots travelingthrough body cavities [12], deformable linear objects [21], andflexible needle devices traveling through deformable tissue [2].

Prior work on robot grasping and manipulation have gen-erally assumed that the grasped object is rigid [4]. Howard etal. [17] model deformable parts using a mass-spring modeland use a neural network to control the gripper. Hirai etal. [16] propose a robust control law for local manipula-tions of deformable parts using tactile and video feedback.

Gopalakrishnan and Goldberg [13] extend the form closureframework to grasping of planar, deformable objects using atwo-point gripper. We address the problem of motion planningfor constrained deformable objects subject to contact andmanipulation constraints.

In concurrent work to ours, Jansen et al. [18] combine the D-Space framework proposed by Gopalakrishnan et al. [13] witha geometric approach to compute candidate grasp locationsand optimal trajectories for automated tissue retraction. Thiscomputationally fast approach considers a 2D cross-sectionalmodel of the tissue flap and computes trajectories for thegripper in a plane. In contrast, our approach works with a fully3D tissue model, handles generic contact and manipulationconstraints, and searches for optimal solutions in constrainedenvironments containing obstacles that must be avoided.

III. PROBLEM STATEMENT

We assume that the geometry of the tissue, including the flapto be retracted and the obstacles, is known from data obtainedfrom medical images or other sensors. We model the tissue asa 3D deformable body M (represented as a tetrahedral finiteelement mesh Mref for simulation purposes). We assume thatM has known material properties, which may be heterogenousthroughout the volume of the tissue. We model the robotgripper as a rigid 6-DOF gripper with a non-zero surface areaof contact. Let qi be a configuration of the gripper in SE(3).

We do not plan for the motion of the actuators or thelinkages but only consider their cumulative effect on thegripper pose. We assume that the tissue moves slowly as it ismanipulated by the gripper such that dynamics can be ignoredand we can approximate the process as quasi-static. We alsoassume that the motion of the gripper and the subsequent tissuedeformations are deterministic; we plan to consider the effectsof uncertainty due to factors such as material parameters,actuation, and slippage in future work.

Exerting excessive forces at the gripper or inducing exces-sively large deformations of soft tissue during manipulationcan cause tissue damage (such as tearing of tissue). Wepropose the following optimization objectives which quantifythese crucial physical quantities:

• Minimizing maximum deformation energy: The totaldeformation energy of a deformed body provides a quan-titative measure of the extent of its overall deformation.In a clinical application, paths that involve high-energyintermediate states may introduce plastic deformationsand cause other irreversible tissue damage. Paths withlower deformation energies will be less susceptible tothese effects.

• Minimizing maximum stress: The maximum stress (ten-sile or compressive) encountered in the deformed bodyrepresents an upper limit to the forces that can be appliedto deformable objects without causing phenomena suchas tearing and fracture [30]. Minimizing the maximumstress helps discard states that are close or beyond thepoint of tearing or fracture, resulting in safe plans.

• Minimizing total control effort: We define the totalcontrol effort expended in performing the task as acombination of the total deformation energy and thegripper displacement, which measures mechanical effort.Minimizing the total control effort balances two com-peting objectives: minimizing the deformation energy ofthe intermediate states along the path and minimizing thetotal mechanical effort involved in executing the plan.

The choice of optimization objective depends on severalfactors such as maneuverability of the gripper in the workspaceand the type of tissue involved in the retraction task and shouldbe determined by the physician during the procedure.

The problem can now be broken down into two subproblemsand formally stated as follows:Problem 1: To compute a sequence of gripper configurations(position and orientation) that optimizes the chosen objectivesubject to the constraint that the area of interest beneath thetissue flap becomes completely visible from the physician’sfixed viewpoint.

Input: Initial configuration of the gripper q0, tissue mesh inits reference stateMref, area of interest beneath the tissue flapA, fixed camera viewpoint V , description of the environment(set of obstacles O) and tissue material parameters (e.g.Young’s modulus E, Poisson’s ratio ν, and density ρ).

Output: An optimal trajectory (discrete sequence of config-urations) P = (q0,q1, . . . ,qn) of the gripper to manipulatethe tissue flap and perform the retraction procedure.Problem 2: The first problem assumes that the initial grasplocation of the gripper is known. This can be generalized toalso compute the optimal grasp location for the gripper q*

0,which optimizes the chosen objective over all possible grasplocations for a given scenario.

IV. MANIPULATION AND PATH PLANNING

The two key components of our approach are a sampling-based global planner and a physically-based simulator asshown in Fig. 2. For each randomly sampled configuration ofthe gripper qi ∈ SE(3) generated by the planner, the simulatorcomputes the corresponding deformed equilibrium state of themeshMi subject to contact and manipulation constraints (seeSec. V). These configurations are used to explore the space ofdeformations and compute an optimal plan. It should be notedthat even though the planner is essentially independent of thesimulator, the accuracy, correctness, and speed of computationof the results is implicitly dependent on the chosen simulator.

A. Generating Valid Samples

Manipulation Constraints: The tissue flap in the meshM isassumed to be anchored at a subset of nodes (usually alongone or more edges of the flap) and can be manipulated byspecifying a non-zero displacement of the grasped nodes alongthe unconstrained edges of the flap boundary. To account fora realistic grip, we constrain multiple nodes on the surfaceof the flap at the gripper location. Based on the initial grasp

vGenerate valid

samples

Search for

optimal plan

Check plan

validity

Update

roadmap

Post-processing

Optimal plan

PlannerSimulator

Nonlinear

FEM

simulator

Inputs: [ , , , , , , , ]0q ref E

0 1( , , , )n

q q q

Fig. 2. System Overview: The sample-based global planner (Planner) relieson the deformable body simulator (Simulator) to compute static, equilibriumstates of the tissue flap corresponding to randomly sampled gripper configura-tions and generate an initial roadmap. The roadmap is queried for an optimalplan based on the chosen optimization objective, which is then checked forvalidity using the Simulator. Notation for input/output explained in Sec. III.

location in the reference stateMref, we define all nodes withinthe bounding box of the gripper jaws as “grasped” nodes.

We generate manipulation constraints of the tissue by ran-domly sampling configurations qi ∈ SE(3) of the 6-DOFgripper. For a given sample qi, we displace and fix all graspednodes by the transformation given by qi. As described inSec. V, we can then compute an equilibrium mesh state Mi

corresponding to the gripper configuration qi.Checking Mesh Validity: We consider a mesh state Mi

(and corresponding configuration qi) to be valid if the finaltopology of the mesh is valid (i.e. no inverted elements orself-penetration). We use Tetgen [28] to discretize the meshand reliably detect self-penetration.Sample Optimization: Since the gripper configuration qi issampled randomly, some states may have undesirably highstresses in the elements constrained by the gripper. For everyvalid sample generated, we perform local gradient-descentoptimization on the orientation of the gripper (while keepingits position in R3 constant) to alleviate the stress on theconstrained elements (as shown in Fig. 3). Our experiments in-dicate that fewer states are eventually required in the roadmapto compute a feasible plan if this local improvement is applied.Local Improvement of Physician-Specified Trajectories: Inthe absence of force-feedback mechanisms and/or restrictedvisibility, our method can also be used for local improvementof physician specified retraction trajectories in challengingscenarios. The physician indicates an approximate retractionpath for the gripper (using the tele-operated surgical system).In our implementation, the physician indicates a final gripperposition (in R3) and the trajectory is implicitly defined asa straight line joining the initial and user-specified gripperposition. A multi-linear trajectory (based on multiple way-

Fig. 3. For each sampled gripper configuration, we optimize orientationto minimize the stresses in the elements constrained by the gripper. For arandomly generated configuration, these stresses can be high (left). Localoptimization re-orients the gripper to minimize these stresses (right).

points) can also be considered. The gripper configurationsqi are then uniformly randomly sampled in a user-definedneighborhood of this trajectory. It should be noted that thequality of the final solution obtained will depend on thephysician-specified trajectory and the choice of neighborhoodsample distribution.

B. Computing Visibility

The objective of the retraction task is to expose an area ofinterest A from the given camera viewpoint V . A mesh stateMi is designed as a goal state if A is completely visible fromthe given viewpoint.

There are two primary types of methods to compute visi-bility. Object-space methods intersect the view frustum of thecamera with primitives in the scene and compute the exactarea visible from the given viewpoint. These methods areexact but computationally expensive. Image-space methodsrender a scene as an image from the camera viewpoint andcompute the area visible in terms of pixels. These methodsare approximate but fast. The pixel accuracy suffices sincewe are only interested in the exposure of a pre-defined areaof interest. For highly-parallel, fast performance, we imple-mented the visibility computation using occlusion queries ongraphics processing units (GPUs) [7]. Based on results ofthe visibility tests, we identify the set of goal mesh statesMg : {M(1)

g , . . . ,M(m)g } and corresponding gripper configu-

rations Qg : {q(1)g , . . . ,q(m)

g }.

C. Constructing the Roadmap

A PRM-based planner can be used to compute a plan fromthe start configuration qs and a goal configuration from set Qg .We store the mesh state Mi along with each sample qi forthe purposes of computing the distance metric defined below.Distance Measure: We use a heuristic distance measure [19]for selecting the nearest neighbors of a sample. If Mi andMj are the mesh equilibrium states corresponding to config-urations qi and qj , then the distance measure is defined asd(qi,qj) = dD(Mi,Mj) + dC(qi,qj), where dD(Mi,Mj)is the DISP distance metric defined as the maximum lengthover all displacement vectors for each node on the surface ofMi toMj and dC(qi,qj) is the distance metric between twoconfigurations defined for the SE(3) group [20]. Attempts to

Fig. 4. Heuristic sampling strategy for local improvement of physicianspecified retraction trajectories. In our test cases, the user specifies a trajectoryby an initial gripper location and a final gripper position in R3 (left). Thegripper configurations qi are sampled within a user-defined neighborhood ofthe trajectory (right).

weight dD and dC have not yielded better results but we foundthat using only dD works reasonably well.Local Planning: We define a path between two samplesrepresented by gripper configurations qi and qj by linearlyinterpolating between them in SE(3) [20] and executingthe simulator to obtain intermediate mesh states. The edgeconnecting any two samples in the roadmap is considered toto be valid if each of the intermediate mesh states is valid(see Sec. IV-A) and if both the mesh and the gripper do notcollide with any obstacles in the workspace. We use PQP [14]for collision detection with obstacles.

D. Assigning Roadmap Edge Weights

We use the optimization objectives outlined in Sec. III toassign weights to edges in the roadmap. These weights arecomputed as follows:Minimizing maximum deformation energy: The cost of anedge is set to be the maximum deformation energy encoun-tered as the gripper moves from configuration qi to qj . Thetotal energy of a single tetrahedral element τ is given by thesum of the internal elastic energy stored in the tetrahedronand the work done by the applied external forces fext on eachof the nodes η ∈ τ . The total deformation energy Πi of themesh stateMi is: Πi =

∑τ∈Mi

v(τ) ·W (E)+∑η∈τ fηext ·uη ,

where v(τ) is volume of the tetrahedron τ , uη is displacementof node η, E is the Green strain tensor, and W (E) is the energydensity function for the chosen material model [30], [23].Minimizing maximum stress: The cost of a edge is chosen tobe the maximum stress (tensile or compressive) encounteredas the gripper moves from qi and qj . For a tetrahedron τ ∈Mi, let σi(Sτ ), i ∈ {1, 2, 3} be the ith eigenvalue of thesecond Piola stress tensor Sτ . Positive eigenvalues correspondto tensile stresses and negative ones to compressive stresses.Since Sτ is real and symmetric, it will have three real, notnecessarily unique, eigenvalues. The maximum stress in τ isthen given by max(abs(σi)), i ∈ {1, 2, 3} [30].Minimizing control effort in lifting tissue flap: We expressthe total control effort C using the metric defined in Mollet al. [21] along a specified path (q1, . . . ,qn) as: C =∑n−1i=1

12d(qi,qi+1)(Πi + Πi+1). A better approximation of

this integral can be obtained by considering a larger numberof subdivisions along the path.

E. Computing Optimal Plans

The local planner requires running the simulation along theinterpolated path, which is computationally expensive. We usea lazy local planning scheme [5] to reduce computation times.We construct an initial roadmap by assigning edge weightsusing configurations at endpoints only. For optimization ob-jectives such as minimizing maximum deformation energy ortissue stress, we set the edge weight to be the maximum of theobjective function value at the two endpoints (which serves asa lower bound for the actual edge weight). For minimizing thetotal control effort, the actual edge weight is readily computed.

We search for a minimum-cost path from qs to Qg inthis roadmap. We use Dijkstra’s shortest path algorithm forsearching for an optimal path from q0 to Qg for additiveoptimization objectives (such as minimizing the total controleffort). The problem of minimizing the maximum edge weightin an acyclic graph (the edge weight is the deformation energyor tissue stress in our case) is referred to as the BottleneckShortest Paths problem [1] and can be solved by a minormodification to Dijkstra’s shortest path algorithm.

The local planner is used to check if all the edges alongthe path are valid. Any invalid edges are discarded and validedges are marked as processed and their edge weights updated.This process is repeated until a valid (optimal) plan is obtainedor no solution is found. As the number of samples generatedincrease, the solution converges to the true optimal.

Since the paths generated by PRM-like methods are notnecessarily short or smooth, we use an iterative shorteningand smoothing scheme to improve the final solution. Givena solution path comprising of discrete gripper configurations,P = (q1, . . . ,qn), the final continuous sequence of controlsis obtained by interpolating between the configurations usinga cubic spline [10] to ensure C1-continuity.

V. SIMULATION OF TISSUE FLAP RETRACTION

Simulating soft tissue is challenging since simulation mustprovide realistic response to manipulations such as graspingand pulling. We chose to model tissue deformations usinga finite element method (FEM) and nonlinear continuummechanics to account for large tissue deformations. The tissueitself is assumed to be isotropic and nearly incompressible andexhibit hyper-elasticity according to the St. Venant Kirchhoffmaterial model (commonly chosen to represent bio-mechanicaldeformable objects [24], [23]). We refer the reader to [30] foran introduction to nonlinear FEM.

The equilibrium configuration of the tissue mesh is deter-mined by balancing all global external forces fext with allinternal elastic forces Φ(u), where u represents the globaldisplacement of all the nodes N in the mesh and Φ is a(possibly nonlinear) function describing the internal elasticforces as a function of the nodal displacements u.Assembling External Forces: The tissue flap is subject toexternal forces exerted by gravity as well as other contact andmanipulation constraints. To compute gravitational forces, weuse a lumped mass formulation [24]. We resolve collisions

of the flap with the underlying tissue (modeled as a planein our experiments) using a penalty-based scheme wherereaction forces are proportional to the penetration depth ofthe intersecting nodes. More sophisticated collision resolutionschemes could also be used [11]. We do not directly considerthe interaction forces between the gripper and the tissue flapand instead fix tissue nodes inside the gripper to the gripperusing displacement constraints.Assembling Elastic Forces: The elastic force acting on anode is given by the negative gradient of the elastic energyin the element with respect to the nodal displacement. Thetotal elastic force acting on a node is obtained by summing upthe elastic forces exerted by all the individual tetrahedra thatare incident to the node. We follow the approach suggested in[23] for computing elastic forces. We also add penalty forces toeach node of the tetrahedron proportional to the variation in thevolume of the tetrahedron as suggested in [24]. This allows usto model the nearly incompressible nature of soft tissue and toprevent inversion of tetrahedra under manipulation constraints.Solving for Static Equilibrium: The tissue flap is assumed tobe anchored at a subset of nodes (usually along one or moreedges along the boundary of the flap) and is also manipulatedby the gripper. Let Nfix denote the set of anchored nodes(which have zero displacement) and grasped nodes (for whichthe displacement is determined by the gripper configurationqi). The equilibrium state of the mesh Mi correspondingto the gripper configuration qi is now given by solving the(reduced) system of equations, Φ(u) = fext, where u nowrepresents the global displacement of all the nodes in the setN \ Nfix and fext represents the external forces acting on thecorresponding nodes. We solve this system of equations usingan iterative nonlinear conjugate gradient solver [27].

VI. RESULTS

We implemented the planner in C++ using OpenMP forparallelization and tested it on a 3.33 GHz 4-core Intel R© i7TM

PC. We use a rectangular tissue flap model of dimensions 5 cm× 5 cm × 0.25 cm for our experiments. The tetrahedralizedtissue flap mesh (constructed using Tetgen [28]) contains 4000elements. We set tissue density ρ = 1000 kg/m3, Poisson’sratio ν = 0.45, and Young’s Modulus E = 1 MPa (unlessotherwise specified) and gravity at 9.8 m/s2 acting downwards.

We applied our method in two contexts. The first is fullautonomy, where the planner globally sampled the entirefeasible space to compute a de novo plan. The second is localimprovement of a physician-specified trajectory, as shown inFig. 3. We evaluated our method using two tissue retractionscenarios described below.Scenario 1: Fig. 5 shows an incision in the tissue from an over-head viewpoint. The objective here is to expose the arbitrarilyshaped area of interest by parting the two tissue flaps whileminimizing the maximum deformation energy and avoidingobstacles. Since the two tissue flaps are symmetric in theinitial boundary conditions (each tissue flap has one straightnon-fixed edge), we compute equilibrium mesh states in the

pre-processing stage for a single tissue flap. The optimal initialgrasp configuration q*

0 is obtained by performing an exhaustivesearch over a set of 5 possible initial grasp configurationsalong the free edge of each tissue flap. The trajectories of thegrippers are then computed by our framework independentlyfor the two tissue flaps. Fig. 5 shows the optimal initial grasppositions and the retraction trajectories of the grippers.Scenario 2: Fig. 6 shows a scenario in which the tissue flapis free along two straight edges and the gripper is allowed tograsp anywhere along these edges. An obstacle protrudes fromthe side and extends over the tissue flap. The tissue flap is mod-eled as a heterogeneous structure with vessels that are stiffer incomparison to the surrounding tissue. In our implementation,we use a threshold image of the tissue texture to segment thevessels and assign appropriate tissue parameters to the meshelements in the flap. In this experiment, we set the Young’sModulus for the vessels to be 1 MPa and for the surroundingtissue to 250 KPa. The optimal initial grasp configurationq*

0 is obtained by performing an exhaustive search over aset of 9 possible initial grasp configurations. Figs. 6 and 7show the optimal grasp locations and retraction trajectoriesfor the gripper when the objective is to minimize the controleffort and the tissue stress respectively. In both cases, thegripper successfully lifts the tissue while avoiding obstacleintersection. Fig. 3 shows the optimized retraction trajectoryobtained after local improvement of physician-specified input.Performance: Table I provides detailed computation times.Table II compares the variation in the chosen optimizationobjective for the global optimization method to compute a denovo plan versus a local improvement method.

Lazy planningScen- Num Sample Collision Mesh FEM Totalario samples generation detection validation sim

(hrs) (secs) (secs) (secs) (secs)

1 1000 1.21 (± 0.15) 0.56 24.94 7.11 32.71 (± 5.98)

2 1000 3.03 (± 0.23) 1.80 32.85 48.15 83.32 (± 38.98)

TABLE IMETHOD PERFORMANCE. AVERAGE TIMES AND STANDARD DEVIATIONS

COMPUTED OVER MULTIPLE INITIAL GRASP LOCATIONS q(i)0 .

Global optimization Local improvement % RMS errorScen- Num Sample Num Sample Min. Min.ario samples generation samples generation maximum maximum

(hrs) (hrs) energy stress

1 1000 1.21 (± 0.15) 200 0.16 (± 0.01) 88.9% 73.6%

2 1000 3.03 (± 0.23) 200 0.29 (± 0.02) 65.6% 90.9%

TABLE IICOMPARISON OF THE GLOBAL OPTIMIZATION AND LOCAL IMPROVEMENTMETHODS. % RMS ERROR IN COMPUTED FOR LOCAL IMPROVEMENT ASCOMPARED TO GLOBAL OPTIMIZATION OVER 10 SUCCESSFUL RUNS FOR

THE OPTIMAL GRIPPER CONFIGURATION q*0 .

The sample generation process involves expensive simula-tions and is the computational bottleneck of our method. Weplan to build on recent advancements in finite element simula-tion [6] to enable interactive execution of our method. Never-theless, our method is successfully able to compute retractionsfor heterogeneous tissues in constrained environments, whileavoiding obstacles and minimizing tissue damage.

VII. CONCLUSION

We have described a framework that takes steps toward au-tomating the task of tissue retraction in robot-assisted surgicalprocedures. The method is directly applicable to arbitrarilyshaped tissue flaps. Given a model of the tissue flap, aphysically-based simulator generates equilibrium mesh statescorresponding to randomly sampled gripper configurations.These can be used with a sampling-based motion planner tocompute an optimal sequence of controls for the gripper. Wedemonstrated the method for several optimization objectivesparticularly relevant to medical applications: minimizing themaximum deformation energy, minimizing maximum stress,and minimizing the control effort in lifting the tissue flap.

A key challenge in future work is to improve the speed ofthe method to enable interactive execution. Another importantissue that needs to be addressed is the discrepancy betweensimulation and observed tissue behavior due to uncertainty inmaterial parameters, gripper actuation, and tool-tissue interac-tion. As in prior work on Stochastic Motion Roadmaps [2], wewill investigate approaches to encode uncertainty informationinto the global roadmap used for tissue retraction planning.

VIII. ACKNOWLEDGEMENTS

This research was supported in part by the National ScienceFoundation (NSF) under grant #IIS-0905344. The authorsthank M. Cenk Cavusoglu, Ken Goldberg, Wyatt Newman,and Pieter Abbeel for their valuable input on this problem.

REFERENCES

[1] R. K. Ahuja, T. L. Magnanti, and J. B. Orlin, Network Flows: Theory,Algorithms and Applications. Prentice Hall, 1993.

[2] R. Alterovitz, T. Simeon, and K. Goldberg, “The stochastic motionroadmap: A sampling framework for planning with Markov motionuncertainty,” in Proc. Robotics: Science and Systems (RSS), 2007.

[3] O. B. Bayazit, J.-M. Lien, and N. M. Amato, “Probabilistic roadmapmotion planning for deformable objects,” in Proc. IEEE Int. Conf.Robotics and Automation (ICRA), 2002, pp. 2126–2133.

[4] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,”in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2000, pp.348–353.

[5] R. Bohlin and L. E. Kavraki, “Path planning using lazy prm,” in Proc.IEEE Int. Conf. Robotics and Automation (ICRA), 2000, pp. 521–528.

[6] N. Chentanez, et al., “Interactive simulation of surgical needle insertionand steering,” ACM Transactions on Graphics (Proc. SIGGRAPH),vol. 28, no. 3, pp. 88:1–88:10, Aug. 2009.

[7] M. Craighead and D. Ginsburg, “ARB occlusion query,” 2001.[Online]. Available: http://oss.sgi.com/projects/ogl-sample/registry/ARB/occlusion query.txt

[8] da Vinci R© Surgical System, “http://www.intuitivesurgical.com/,” 2005.[9] S. A. Darzi and Y. Munz, “The impact of minimally invasive surgical

techniques,” in Annu Rev Med., vol. 55, 2004, pp. 223–237.[10] M. DeLoura, Game Programming Gems 2. Charles River Media, 2002.[11] B. Frank, M. Becker, C. Stachniss, W. Burgard, and M. Teschner, “Effi-

cient path planning for mobile robots in environments with deformableobjects,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA),2008, pp. 3737–3742.

[12] R. Gayle, P. Segars, M. C. Lin, and D. Manocha, “Path planning fordeformable robots in complex environments,” in Proc. Robotics: Scienceand Systems (RSS), 2005, pp. 225–232.

[13] K. G. Gopalakrishnan and K. Goldberg, “D-space and deform closuregrasps of deformable parts,” in International Journal of Robotics Re-search, vol. 24, no. 11, 2005, pp. 899–910.

[14] S. Gottschalk, M. C. Lin, and D. Manocha, “OBB-tree: A hierarchicalstructure for rapid interference detection,” 1996. [Online]. Available:http://www.cs.unc.edu/∼geom/SSV

(a) (b) (c) (d)Fig. 5. Scenario 1: The objective is to part the tissue flaps on either side of the incision and completely expose the underlying area of interest (red). Optimalinitial grasp locations as determined by our method (a). Figures (b)–(d) illustrate the trajectory of the two grippers as the tissue is retracted while minimizingthe maximum deformation energy and avoiding the obstacle in the workspace.

(a) (b) (c) (d)Fig. 6. Scenario 2: Grasp location and gripper trajectory that minimizes control effort. The gripper starts on the other side of the obstacle. (a) The gripperperforms a twist to minimize displacement, thereby inducing large deformations. (b) The gripper rectifies itself and performs the retraction successfully (c–d).

(a) (b) (c) (d)Fig. 7. Scenario 2: Grasp location and gripper trajectory that minimizes the maximum tissue stress. The gripper starts on the side of the obstacle (a). Thegripper maneuvers around the obstacle and raises the tissue flap straight up in order to minimize torsional stress in the constrained nodes (b–d).

[15] J. W. Hill, P. S. Green, J. F. Jensen, Y. Gorfu, and A. S. Shah,“Telepresence surgery demonstration system,” in Proc. IEEE Int. Conf.Robotics and Automation (ICRA), 1994, pp. 2302–2307.

[16] S. Hirai, T. Tsuboi, and T. Wada, “Robust grasping and manipulation ofdeformable objects,” in Proc. IEEE International Symposium Assemblyand Task Planning, 2001.

[17] A. M. Howard and G. A. Bekey, “Intelligent learning for deformableobject manipulation,” in Autonomous Robots, vol. 9, 2000, pp. 51–58.

[18] R. Jansen, K. Hauser, N. Chentanez, F. can der Stappen, and K. Gold-berg, “Surgical retraction of non-uniform deformable layers of tissue:2D robot grasping and path planning,” in Proc. IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, 2009, pp. 4092–4097.

[19] F. Lamiraux and L. E. Kavraki, “Planning paths for elastic objects undermanipulation constraints,” in International Journal of Robotics Research,vol. 20, no. 3, 2001, pp. 188–208.

[20] S. M. LaValle, Planning Algorithms. Cambridge, U.K.: CambridgeUniversity Press, 2006, Available at http://planning.cs.uiuc.edu.

[21] M. Moll and L. E. Kavraki, “Path planning for deformable linearobjects,” in IEEE Transactions on Robotics, vol. 22, no. 4, 2006, pp.625–636.

[22] A. Nealen, M. Muller, R. Keiser, E. Boxerman, and M. Carlson, “Phys-

ically based deformable models in computer graphics,” in ComputerGraphics Forum, vol. 25, no. 4, 2006, pp. 809–836.

[23] H.-W. Nienhuys, “Cutting in deformable models,” Ph.D. dissertation,Utrecht University, 2003.

[24] G. Picinbono, H. Delingette, and N. Ayache, “Nonlinear and anisotropicelastic soft tissue models for medical simulation,” in Proc. IEEE Int.Conf. Robotics and Automation (ICRA), 2001, pp. 1370–1375.

[25] S. Rodriguez, J.-M. Lien, and N. M. Amato, “Planning motion incompletely deformable environments,” in Proc. IEEE Int. Conf. Roboticsand Automation (ICRA), 2006, pp. 2466–2471.

[26] J. R. Shewchuk, “An introduction to the conjugate gradient methodwithout the agonizing pain,” Carnegie Mellon University, Tech. Rep.,1994. [Online]. Available: http://portal.acm.org/citation.cfm?id=865018

[27] H. Si, “Tetgen,” 2007. [Online]. Available: http://tetgen.berlios.de/[28] R. H. Taylor, et al., “An image-guided robotics system for precise

orthopaedic surgery,” in IEEE Transactions on Robotics and Automation,vol. 10, no. 3, 1994, pp. 261–275.

[29] O. C. Zienkiewicz, R. L. Taylor, and J. Z. Zhu, The Finite ElementMethod: Its Basis and Fundamentals. New York: Butterworth-Heinemann, 2005.