slip classification for dynamic tactile array...
TRANSCRIPT
Slip Classification for Dynamic Tactile Array Sensors
Barrett Heyneman1 Mark R. Cutkosky1
December 2, 2014
Abstract
The manipulation of objects held in a robotic hand orgripper is accompanied by events such as making andbreaking contact and slippage, between the fingertipsand the grasped object and between the grasped ob-ject and external surfaces. Humans can distinguishamong such events, in part, because they excite thevarious mechanoreceptors in the hands differently. Aspart of an effort to provide robots with a similarcapability, we propose two features that can be ex-tracted from dynamic tactile array data and used todiscriminate between hand/object and object/worldslips. Both features rely on examining how slippageaffects an array of dynamic tactile sensors comparedto the way it affects individual elements of the array.In comparison to approaches that require extensivetraining with particular combinations of objects andskin, the features work for a wide range of frequenciesand grasp conditions. The performance and general-izability of the features are verified with testing onthree different kinds of sensors and for a range of ob-ject textures, grasp forces and slip conditions. Bothfeatures demonstrate greater than 85% accuracy inidentifying the location of slip.
1 Introduction
Human skill in grasping and manipulation relieson remarkable tactile sensing capabilities. We canquickly identify textures and fine features using ourfingertips, and while holding objects we subcon-sciously control grip forces and prevent slippage.When using tools we act as if they are extensions of
1Stanford University, Stanford, CA.
our limbs, which requires not only dexterous manip-ulation but sophisticated and implicit understandingof how the sensations at our fingertips relate to theinteractions between the tool and the environment.
In comparison, robotic tactile sensing is impover-ished. A sophisticated hand may have dozens or evena hundred tactile sensors, but not tens of thousands;and the sensors are typically of only one or two types.Practical challenges include wiring and the need toembed sensors in compliant skin materials which ex-perience frequent impacts and deformation. In con-sequence, the use of tactile information in roboticshas proceeded slowly in comparison to vision.
Notwithstanding these challenges, the need for tac-tile sensing in robotics is steadily increasing. Robotsare being pushed to perform increasingly complextasks, often using human tools in unstructured en-vironments. New developments in tactile sensingare exploiting advances in flexible electronics withshrinking sensors and microprocessors (often drivenby the smartphone industry) for local signal process-ing and communication. With these advances we areseeing robust tactile sensor arrays capable of deliv-ering dynamic data into the kilohertz range (Ascariet al, 2009; Aukes et al, 2014; Jentoft et al, 2013; Linet al, 2009; Mittendorfer and Cheng, 2011; Schmitzet al, 2011), with recent reviews of the state of the artin (Cutkosky and Ulmen, 2014; Dahiya et al, 2010;Lucarotti et al, 2013; Tiwana et al, 2012; Yousef et al,2011).
The work presented here is part of an effort to usethese improved tactile sensing capabilities to iden-tify and distinguish between different kinds of contactevents, particularly those that occur between a handand a grasped tool or between a grasped tool and the
environment. In routine tasks, autonomous roboticsystems will encounter both types of events. In iden-tifying these events, we seek algorithms that do notrequire extensive training, which work robustly for avariety of sensor types, and which are not sensitiveto variations in grasp force and object texture.The following sections start with a brief review
of relevant prior work, including studies of humanmechanoreception and manipulation event detection.We then present two features that are based on howevents such as hand/object and object/world slippageaffect arrays of tactile sensors versus individual ele-ments within an array. The corresponding algorithmsare suited for real-time implementation on hardwarewith limited computational power, allowing for dis-tributed computation to identify and classify theseevents. The features are compared for three differenttypes of sensors and fingers, and a variety of graspforces and object textures. The results show bet-ter than 85% identification of slip type across thesewidely varying conditions. We conclude with a dis-cussion of extensions that could further improve theevent detection and classification approach and withareas for future work.
2 Related Prior Work
The work presented here draws inspiration fromhuman mechanoreception and event detection andbuilds upon prior work in robotic slip detection andtexture identification.
2.1 Human Mechanoreception in Ma-
nipulation
An extensive body of research on human tactile sens-ing has elucidated the roles of fast-acting and slow-acting mechanoreceptors for detecting and distin-guishing among various types of events during ma-nipulation (Johansson and Flanagan, 2009). Fast-acting mechanoreceptors are of particular importancefor detecting the formation and breaking of contactsand the onset of slip. Meissner corpuscles are super-ficial fast-acting mechanoreceptors mechanically cou-pled to the dermal papillae (Fig. 1). They respond
strongly to events in the 10-60 Hz range that deformthe papillae, including deformations that occur asfingers press upon an object and “plucking” of thefingerprint ridges as an object starts to slide. Sens-ing these events is important to human grasp forcemodulation (Johansson and Flanagan, 2009; Johans-son and Westling, 1984; Witney et al, 2004), becausethey indicate incipient slip which occurs before grossslip between the hand and a held object.
In contrast, when a held object vibrates, the manyindividual Meissner corpuscles are not significantlyexcited. As shown in (Hunt, 1961), the Pacinian cor-puscles instead register significant neuronal activity.More specific research by (Brisben et al, 1999) showedthat human perception of held object vibrations fol-lows the response curves from individual Paciniancorpuscles, indicating that these receptors are re-sponsible for our perception of transmitted vibration.Considering that the receptive fields of the Paciniancorpuscle are large (Fig. 1) it is not surprising thatthey are primarily responsible for sensing unlocalizedvibrations in the 50-300 Hz range.
The coding of tactile signals involves more thanthe response spike-rate of certain mechanoreceptors.Various studies have indicated that relative timing,phasing, and even coherence of the responses of dif-ferent mechanoreceptors may play a pivotal role inconveying tactile information. In humans, Johanssonand Birznieks (2004) demonstrated that the relativetimings of the first intra- and inter-mechanoreceptorneural spikes at the onset of stimulation are suffi-cient to predict tangential force direction as a finger-tip comes into contact with an object. Other stud-ies of mechanoreceptors in cats’ paws (Greensteinet al, 1987) and the halteres of flying insects (Fox andDaniel, 2008; Fox et al, 2010) indicate that relativephasing within populations of neurons carries impor-tant information. Collectively, these results suggestavenues for robotic sensing and perception which donot rely simply on the amplitude of a given input sig-nal but rather on more complicated, time-dependentmeasures.
Figure 1: The depth of human mechanoreceptors andtheir mechanical coupling to surrounding tissue givesthe fast-acting type I and II receptors very differentreceptive fields. Receptive fields for Meissner corpus-cles and Pacinian corpuscles are shown here, with thearea scaled to reflect actual receptive field measure-ments from (Knibestol, 1973). Image adapted from
(Cutkosky and Ulmen, 2014).
2.2 Robotic Slip Detection
Slip detection is one of the earliest tactile sensingproblems discussed in robotics. Many of the demon-strated solutions have focused on detecting the onsetof slippage, or even incipient slippage. Some of theearliest work (Howe and Cutkosky, 1989) utilized ac-celerometers embedded in the skin to measure thesmall vibrations that accompany the onset of slid-ing between an object and a finger. Embedded dy-namic sensors, such as polyvinylidene-flouride stressrate sensors, have continued to be a popular optionwith a variety of working implementations (Fujimotoet al, 2003; Howe and Cutkosky, 1993; Yahud et al,2010; ?). These approaches directly measure vibra-tions in the skin or finger medium that are producedby the stick-slip phenomena occurring at the surface.Other investigators have used less direct measure-
ments of these micro-vibrations, for example examin-ing the variation of the center of pressure with respectto the resultant forces to determine the onset of slid-ing (Gunji et al, 2008; Ho and Hirai, 2014; Melchiorri,2000). Still others have applied computer vision tech-niques to the output of tactile sensing arrays, treat-ing them as grayscale images, in order to track tactilefeatures moving across the array (Ascari et al, 2009;?).In addition, a wide variety of processing techniques
have been used to characterize and identify the dy-namic force variations associated with slip, includinghigh-pass filtering, fast fourier transforms, extractionof predefined features, and artificial neural networks(Holweg et al, 1996; Jamali and Sammut, 2012; Ro-mano et al, 2011; Teshigawara et al, 2011; Zhang andLiu, 2012; ?). In most cases these require trainingagainst a specific set of objects and grasp conditions,or assumptions about the frequencies of vibrationsthat occur as specific features of the robotic skinslide on an object. However, there has been littleinvestigation of strategies to distinguish among slipsat different interfaces (e.g. finger/object versus ob-ject/world) for a variety of grasping conditions.
2.3 Robotic Texture Recognition
Like slip detection, texture identification has been acommon application of tactile sensors in robotics. Forthe present case, the interest is primarily to considerhow the skin and tactile sensors are affected by atextured object so that slip detection can be maderelatively insensitive to object texture.
With a few exceptions (e.g. (Johnson and Adel-son, 2009; Kim et al, 2005a; Maheshwari and Saraf,2006)) most texture sensing uses dynamic signals gen-erated as a skin with sensors passes over the surfaceof an object. Indeed, the same sensors used for de-tecting slip can generally be used for texture identifi-cation. Approaches to texture discrimination varywidely, from modeling of the sensor/skin response(Howe and Cutkosky, 1993; Mazid and Russell, 2006),to utilizing statistical or machine learning techniqueson features extracted from the time- or frequency-space representations of the sensor signals (Decherchiet al, 2011; Hosoda et al, 2006; Jamali and Sammut,2011; Kim et al, 2005b; Lepora et al, 2010; Sinapovet al, 2011). In a few cases, researchers have identifiedsignal features that are relatively insensitive to otheraspects of the surface interaction (e.g. sliding speedor pressure). For example, Oddo et al (2011) usedrelative phasing extracted from signal cross-spectraas a speed-invariant feature for texture classification.
3 Classification Features
As noted in Section 1, there are increasing exam-ples of tactile array sensors that, like fast-actingmechanoreceptors, can respond to dynamic changesin surface stresses, with bandwidths of 50Hz orgreater. In this section we present two methods touse signals from such arrays to distinguish betweenslips at two distinct interfaces: between the sensorand a held object or between the held object andthe environment. The features are intended to berelatively insensitive to variations in grasp force andobject characteristics such as texture.We start with some definitions to frame the prob-
lem and limit its scope. The tactile array sensorsare modeled as N elements, or taxels, which are em-bedded in the grasping surfaces of a robotic hand orgripper. Often, the array elements are sampled bymicroprocessors located within or near the array andcommunicating over a digital line. The nth samplefrom the discretized array signal is given as
pn = [p1(nT + δ1), . . . , pN (nT + δN )] (1)
where pi sample from the ith sensor. The array oftaxels is sampled at period T , while individual taxelsmay be sampled with an individual constant delay δi.Depending on the architecture, they may be sampledsequentially, where each element is read one-by-one,or simultaneously, where all elements are read at es-sentially the same time:
δi =
�T (i−1)
N sequential sampling
0 ∀i, simultaneous sampling(2)
Of the N taxels of the array, there is an identifiablesubset, Scontact, which are in contact with the heldobject. For this work we assume that the sensor’ssurface texture does not vary significantly over thiscontact patch and that slip, if it occurs, results in sim-ilar relative velocity across the entire patch. In ourexperiments this is true of the entire contact patchand we therefore utilize all the taxels physically be-neath it. In more complex situations this might notbe the case; however this initial work does not ex-plore how to incorporate knowledge concerning areaswith significantly varying contact conditions.
Figure 2: A cross section view of a contact patch,including M individual contact points, and a set ofnearby sensors. The skin and finger medium forma transduction path to the sensors, represented byF (t). The sensors will pick up vibrations due to bothobject vibrations and slip between it and the finger.
The taxels are assumed to be embedded in a lin-early viscoelastic material forming the gripper skin orbody. It can be modeled as a linear, time-invariantsystem transforming surface tractions (normal stress,σzz(t), and the two shear stresses, τzx(t) and τzy(t))to sensor outputs, similar to the model used in thesoft finger analysis by (Fearing and Hollerbach, 1985).Most tactile sensors designed for dynamic sensing
applications have a textured surface. Textures havebeen shown to produce greater high-frequency signalsand more consistent friction, and they have been in-vestigated by a variety of researchers including (Fear-ing and Hollerbach, 1985; Howe and Cutkosky, 1993;Oddo et al, 2011). We assume the grasping surfacesof the hand make contact with a held object in a finitenumber of locations due to the surface features andthat surface traction inputs occur only at these M
locations. Therefore, the finger/sensor system mapsthe traction input vector σ(t) ∈ R
3M to the sensorsignal via the N × 3M linear system F (t):
p(t) = (F ∗ σ) (t) (3)
A schematic of an example sensor in contact witha rigid object is shown in Figure 2 to illustrate theseassumptions.Both features are extracted from the signals in fre-
quency space by computing spectral quantities overa given frequency band. While there are many tech-niques for spectral estimation from discretely sam-pled signals, in this work we will rely on the Dis-crete Fourier Transform, implemented via the FastFourier Transform algorithm, and Welch’s algorithmto improve estimation accuracy at the expense of fre-quency resolution.The frequency space representation of a signal will
be denoted with non-script text; e.g. the time sig-nal x(t) becomes the spectral coefficients x(k); thetime or frequency component arguments t and k aredropped where unambiguous. These spectral coeffi-cients are complex quantities and ∗ is used to rep-resent the complex-conjugate, as in x∗. The powerspectral density, generally a matrix quantity, is de-noted Px,y, where the second subscript is omittedwhen they are both the same.
3.1 Signal Power Ratio
As noted in Section 2.1, humans use different fast-acting mechanoreceptors with small and large recep-tive fields to characterize and distinguish among dif-ferent contact events. In a similar way, the signalsfrom individual taxels of an array can be used tomeasure local vibrations while the sum of the taxelsignals measures vibrations affecting the entire arrayas an ensemble (Figure 3).We hypothesize that classification can be made
based on the ratio of ensemble to local signal power.When slip occurs between the object and the envi-ronment we expect to see comparatively more signalpower in the ensemble signal than the local signal,just as Pacinian corpuscles are more excited by suchevents in humans. In contrast, when slip occurs be-tween the hand and the object we expect to see com-paratively more signal power in the local signal thanthe ensemble signal, analogous to the increased ac-tivity seen in Meissner corpuscles.
3.1.1 Definition
We begin by developing a measure of the local effectof vibrations on the taxel array. The spectral coef-ficients, pi(k), are estimated with a resolution of f0
(a) Receptive fields for each individual sensor
(b) Receptive field for the sensors as an ensemble
Figure 3: The data from tactile array sensors canbe processed to provide local and distributed recep-tive fields as found in human biology. In (a) eachtaxel is sampled and processed individually, produc-ing smaller localized responses to stimuli at the sur-face. When the taxels are sampled together, as ifthey were one larger sensor, the response to surfacestimuli spreads out, as shown in (b).
for each discrete signal. The power contained in afrequency band centered at f with width 2w is cal-culated for each taxel, and then summed across alltaxels in contact with the held object
L(f, w) ��
i∈Scontact
(f+w)/f0�
k=(f−w)/f0
|pi(k)|2
(4)
By summing power rather than the time-domainsignals pi(t) (or spectral coefficients) directly thereis no cancellation due to differences in phasing. Thesignal at each taxel contributes to L(f, w) in the sameway, independent of the signals seen at any othertaxel. For this reason the measure more directly re-flects how strongly the vibrations in the given fre-quency range are affecting each of the sensor elementslocally.In order to compute a similar indicator for the ef-
fect of vibrations on the array as an ensemble, the
signals from all the taxels in contact with the objectare combined in order to mimic a single virtual taxel
that covers the entire grasping surface and thereforehas a large receptive field. The power in this vir-tual sensor signal is computed for the same frequencyrange as in Equation (4)
E(f, w) �(f+w)/f0�
k=(f−w)/f0
������
i∈Scontact
pi(k)
�����
2
(5)
Here the formation of the single virtual taxel is per-formed by summing the spectral coefficients, whichis equivalent to summing the individual signals inthe time domain. For some sensors, a more specificmethod may be appropriate. For example, an ap-proach for the BioTac R�sensor presented in (Wettelsand Loeb, 2011) uses an artificial neural network toextract the point-of-application and vector of appliedforce from the array of signals. With any method itwill be possible for the signals from different taxels tocancel one another at some frequencies while addingconstructively at others.Finally, this biologically inspired feature is defined
as the ratio of the ensemble power measure to thelocal power measure:
Γ(f, w) � 1
Nc
E(f, w)
L(f, w)(6)
where Nc is the number of taxels in contact and nor-malizes Γ(f, w) such that it always lies in the range[0, 1].
3.1.2 Sampling Delays
In order compute the ensemble power E(f, w) someadditional steps must be taken. Because the inten-tion is to mimic a virtual sensor covering the grasp-ing surface, the relative timing of the taxel samples,given by δi, must be taken into account. If the arrayis simultaneously sampled the individual signals cansimply be averaged as described above. For example,if all sensors are experiencing the same input signal,the relative phasing of the sequentially sampled sig-nals will vary, causing cancellation in Equation (5).However, if the same input were to be applied and
the sensors simultaneously sampled, there would beno such cancellation.If the samples pi of the ith taxel have been delayed
by δi relative to the ideal simultaneous sample time,a resampled signal �pi must be formed which estimatesthe discrete sequence that would have been generatedby sampling the taxel with no delay. Because a simplesummation is used in Equation (5) to combine theindividual signals, the pure delay can be correctedby applying a time advance to the original samplesignal directly in the frequency domain. With thisstep added, Equation (5) becomes
E(f, w) =
(f+w)/f0�
k=(f−w)/f0
������
i∈Scontact
pi(k)e2πjkδif0
�����
2
(7)
where f0 is the frequency resolution of the discretespectral estimate. We have used the fact that a puretime delay manifests in frequency space as a purephase delay; the complex exponential term above.This formula can be applied generally. In the simul-taneous sampling case when δi = 0 the exponentialterm from the pure delay becomes unity and may bedropped.This method can also be used to provide the resam-
pled time series in cases where a more complicatedprocedure is necessary to combine individual taxelsinto an ensemble, e.g. the artificial neural networkprocess mentioned above. The original signals canbe transformed to the frequency domain, the timeadvance applied, and then transformed back to thetime domain.
3.2 Coherence
The second candidate classification feature is moti-vated by considering the source of vibrations andhow those vibrations are transmitted to the individ-ual taxels. Consider the case in which the finger isin contact with a rigid object, as shown in Figure 2.The surface tractions, σ(t), are now due to the com-bination of the motion of the held object and anyslip that occurs between the object and the finger.We assume that over the sample duration, the ra-tios of the components of linear and angular velocity
of the object with respect to the contact patch onthe finger remain constant. Therefore, the velocitiescan be described with a single time sequence, v(t). Inaddition, the stick-slip phenomena at each contact lo-cation are assumed to generate surface tractions withthe same constraint: the three components are not in-dependent and can be characterized by a single timesequence, si(t). Under these assumptions the surfacetractions can be defined as
σ(t) = (B ∗ (bv)) (t) + Ts(t) (8)
where b ∈ R6 describes how the single degree of free-
dom of the body’s vibration is described relative tothe finger, B(t) ∈ R
3M×6 represents how that mo-tion creates surface tractions at each contact loca-tion, and T ∈ R
3M×M maps the single degree offreedom of each stick-slip phenomenon to the pro-duced surface tractions. Note that T is block di-agonal, reflecting the fact that the stick slip at eachcontact location creates surface tractions at that loca-tion. An expanded system block diagram with theserelationships is shown in Figure 4. Combining thismodel with Equation (3) and transforming to fre-quency space we have
p = F (bv + Ts) (9)
When the object vibrates, there is a commonsource of vibrations being transmitted through thefinger medium to each sensor. Assuming those trans-missions paths can be modeled as linear, the signalsat each sensor element should be coherent with oneanother.However, when the object slips against the finger
the vibrations at each contact region are due to thelocal stick-slip behavior of the surface features. Ifthese stick-slip events are independent they will nothave consistent phase relationships. As indicated inSection 3, we assume that the contact conditions donot vary significantly across the contact patch, andfor these reasons we can assume each stick-slip eventhas a similar power spectrum. Therefore the follow-ing holds for the power spectrum of the set of allsignals:
Ps(k) = ckI (10)
Figure 4: A block diagram representation of the ob-ject/gripper/sensor model in Figure 2. Object vibra-tion is transmitted to the system at each of the M
contact points where the individual signals due to slipbetween the object and the gripper are also injected.
for some constants ck ∈ C, and the identity matrixI.For purposes of this initial development we will ad-
ditionally assume that the sensor is designed to min-imize crosstalk between the taxels or that the surfacetractions at each contact location are sensed by a sin-gle taxel. This implies that for an appropriate order-ing of contact locations, F also has a block diagonalstructure and many of the fi,j blocks in Figure 4 areidentically zero.
3.2.1 Applying Signal Coherence: Two Sen-sors
The complex coherence between two signals, denotedCx,y, is defined as the normalized cross-spectral den-sity of those two signals with respect to the spec-tral density of the individual signals, as in Equation(11). The mean square or magnitude squared coher-ence, C2
x,y is also often used. As the name implies, itis the magnitude of the complex coherence squared,and due to the relationships between cross-spectraland auto-spectral density it lies in the range [0, 1]
Cx,y � Px,y�PxPy
(11)
There are two physical interpretations of complexand mean square coherence that are relevant to thiswork. Firstly, C2
x,y measures the consistency of therelative phase between the two signals at each fre-quency; when the relative phase is constant C2
x,y = 1,and as it varies C2
x,y → 0. Secondly, the complexcoherence values are related to the linear filter coeffi-cients which minimize the power density of the esti-mated signal when used to estimate one signal fromthe other. Additionally, the mean square coherencemeasures the fraction of the original signal power thatis present in the estimated signal using this filter; ifit is unity the signals can be perfectly reconstructedfrom one another. A more thorough treatment of co-herence and its interpretations can be found in mostspectral analysis texts, e.g. Marple (1987).If we consider the filtering interpretation of coher-
ence, we can demonstrate how to apply it to the slipinterface classification problem using a simple twosensor example (N = 2). In the case where the objectslips against the environment and not against the fin-ger the surface tractions are due only to the vibrationof the object. Each taxel signal is a linear filtered ver-sion of this underlying signal, and therefore of eachother, and we expect them to be completely coherent.Equation (9) reduces to
p = Fbv (12)
and the power spectrum of p computed as
Pp = (Fb)∗ Pv (Fb)T (13)
Because Pv ∈ C we know that rank (Pp) = 1.When it is normalized per Equation (11) the meansquare coherence matrix is
C2p =
�1 11 1
�(14)
This indicates that the two signals are perfectly co-herent when the surface tractions are due only to vi-bration of the held object.In the second case we assume the object only slips
against the finger, and the surface tractions are dueonly to the interaction between the finger and object.In this case Equation (9) reduces to
p = FTs (15)
with the power spectrum of p given by
Pp = F∗T∗PsTTFT
= cF∗T∗TTFT
using the fact that the signals are identical and in-
coherent (Equation (10)). The no-crosstalk sensorassumption, and the corresponding block-diagonalstructure of F and T, can be used to show that theoff-diagonal terms of Pp are identically zero. Themean square coherence in this case becomes
C2p =
�1 00 1
�. (16)
When we compare the results for the two cases wesee that the sensor signals in the object/world slipcase are completely coherent while the signals in theobject/hand slip case are completely incoherent. Ina real physical system this contrast is unlikely to beas stark for a variety of reasons. First, it is possi-ble that there will be aspects of both types of slipat any given time, even in cases where only one isdominant. It is also likely that the surface tractionsat the M locations due to object/hand slip will notbe completely incoherent, or will have slightly differ-ent power spectra due to local differences in texture,curvature, contact pressure, etc.
In practice, few tactile array sensors achieve zerocrosstalk, largely due to physical coupling in the fin-ger medium. However, any practical array sensor ar-ray must limit the crosstalk or coupling between tax-els in order to generate useful information across thearray. As we relax the no-crosstalk assumption, therewill be an increase in the coherence of signals from thetaxels during object/hand slip, due to the fact thateach is partially sensing the vibrations occurring inthe other’s active area.
Nonetheless, despite anticipated violations of ourassumptions in practice, we expect to see significantlyhigher coherence values between two sensors whenthey experience object/world slip as compared to ob-ject/hand slip. As seen in Section 4, this expectationis borne out in the empirical results.
3.2.2 Extending Coherence
The example above used only two taxels and the co-herence between them. In practice an array of N
taxels will generate an N×N complex coherence ma-trix. In order to simplify the feature space used inclassification we extend the concept of mean squarecoherence to cover groups of N signals. Motivatedby the problem at hand, this will be accomplishedby extending the optimal filtering interpretation todefine the group square coherence, or GSC. The re-sulting definition is similar to that found in Ramırezet al (2008). However in this case there is a specificphysical interpretation of the derivation which pro-vides insight into how and why it is appropriate forthis application.The GSC, G2
x(ω), measures the maximum normal-ized fraction of signal power across the group that canbe estimated using a single underlying source signaland a set of optimal linear filters. In this section weshow that the GSC thus defined can be efficientlycalculated via an eigenvalue analysis of the complexcoherence matrix.We begin the derivation by calculating the normal-
ized total power in the error between the measuredsignals x(t) and the signals produced by filtering asingle input signal u(t) through a filter bank h(t).The error vector e(t) is given by
e(t) = x(t)− (h ∗ u)(t) (17)
The power in each error signal is normalized bythe power of the original signal in order to preventsignals with higher power from dominating the opti-mization. If the errors are not normalized the optimalfilters and source signal, hopt and uopt respectively,will skew towards recreating the signals with morepower more accurately, even if the majority of thesignals are coherent but much lower power
Pe =N�
i=1
Pei
Pxi
(18)
= Tr�D−1/2
x PeD−1/2x
�(19)
where Dx is a matrix consisting of only the diagonalentries of Px. We can then expand Pe by using the
definition in Equation (17) to get
Pe = Px + h∗PuhT − h∗Pux − Pxuh
T. (20)
Finally, Pe is minimized with respect to the spec-tral coefficients of the optimal filter bank, h, by tak-ing partial derivatives and setting equal to zero:
∂
∂hPe =
∂
∂hTr
�D−1/2
x PeD−1/2x
�
=∂
∂hTr
�D−1
x Pe
�
=∂
∂hD−1
x Pe
= D−1x
∂
∂hPe
= 0
where we have used the properties that Tr(AB) =Tr(BA) and ∂
∂x Tr(A) = ∂∂xA. Substituting ∂
∂hPe
computed from Equation (20) and then solving for hgives the optimal set of filters to minimize the sumof normalized error powers.
hopt =PTux
Pu=
P∗xu
Pu(21)
minh
Pe = N − PuxPxu
Pu(22)
In order to minimize Pe with respect to u we firstnote that the space of all square-integrable functions,L2, can be represented by a Hilbert space, HP with
the cross spectral density between two signals as theinner product.We can define the space X ⊂ HP as the subspace
spanned by the individual signals, i.e. the rows ofx(t). Any source signal u(t) can be decomposed intoa linear combination of the signals which span X,aTx(t), and an additional signal which is orthogonalto the subspace, u⊥(t), such that
u(t) = aTx(t) + u⊥(t). (23)
Using this decomposition into orthogonal sub-spaces one can compute the power spectral densitymatrices in Equation (22):
Pu = aTPxa+ Pu⊥ (24)
Pu,x = aTPx (25)
Px,u = Pxa (26)
Combining these results with Equation (22) resultsin
minh
Pe = N − aTPxD−1x Pxa
aTPxa+ Pu⊥
(27)
from which we can see that Pe is minimized if andonly if Pu⊥ = 0Finally, using Equation (11) yields a simplified ex-
pression for Pe:
minh
Pe = N −�D1/2
x a�T
CxCx�D1/2
x a�
(28)
where Dx is a matrix with only the diagonal entriesof Px.Equation (23) indicates that minimizing over the
source signal u(t) is equivalent to minimizing overthe coefficients a. This is an eigenvalue decomposi-
tion problem which is minimized when D1/2x a = kv1,
where v1 is the first eigenvector of Cx associated withthe largest eigenvalue, λ1. The minimized value canbe expressed as
minh,u
Pe = N − λ21k
2. (29)
Because u(t) and h(t) are convolved in Equation(17) the optimal solution is only unique up to a scal-ing factor between the two, i.e. if uopt(t) and hopt(t)minimize Pe, so do 2uopt(t) and 1/2hopt(t). To finda unique solution by fixing k, we restrict Pu = 1 atall frequencies:
Pu = aTPxa
=�D1/2
x a�T
Cx�D1/2
x a�
= λ1k2
= 1
Applying this final scaling factor yields the results
of the optimization.
minh,u
Pe = N − λ1 (30)
hopt = Pxaopt (31)
CxD1/2x aopt = λ1 (Cx)D1/2
x aopt (32)
Equipped with the optimal source signal and fil-ters we can define G2
x and then simplify it. Let thegroup square coherence be defined as the average ofthe normalized power in each estimated signal.
G2x(ω) �
��Ni=1
Pxi(ω)
Pxi (ω)
�− 1
N − 1
2
(33)
We normalize to N − 1, including the −1 term inthe numerator, instead of N to reflect the fact thatthe underlying signal u(t) can, at worst, perfectlyestimate one of our measured signals xi(t).Following similar simplifications steps as for Pe
above and substituting the optimal filters hopt andsignal uopt, this definition can be reduced to
G2x(ω) =
�hToptD−1
x h∗opt − 1
N − 1
�2
=
�λ1 − 1
N − 1
�2
(34)
In Appendix A we demonstrate the following use-ful properties that stem from this definition: thatG2xω) = C2
x(ω) for N = 2 and that the range andextrema of G2
x(ω) occur when all pairwise C2xi,xj
(ω)values are at the same extremes (0 or 1).
4 Slip Experiments
Three sets of experiments, each utilizing a differentsetup and tactile array sensor technology, were per-formed in order to generate examples of slip occurringat both interfaces. While elements of the interactionwere held constant across these experiments for com-parison purposes, a breadth of conditions were testedto evaluate the ability to work with a range of tex-tures from smooth to rough and at low and high grasp
forces and high and low sliding speeds typically usedin manipulating objects.In all experiments, the held object consisted of a
specific texture sample taken from the set describedin Table 1 and Figure 5. The set includes smoothsurfaces, various grits of sandpaper, regular patternsof holes or slats, and even a silicone mold of the skinon the outer surface of the capacitive sensor (Sec-tion 4.2). This last texture was chosen specifically topresent a difficult classification problem, especiallyfor the capacitive sensor, in that the same surface isinvolved in both object-world and object-hand slip.In this section we describe the sensors and the ex-
periments performed. Any sensor specific considera-tions taken when extracting the features described inthe previous section are discussed with the results inSection 5.2
Figure 5: The 8 textures plates used in experiments(described in Table 1).
4.1 BioTacsTM
and MC Hand
The first set of experiments utilize the Syntouch Bio-Tac 1 sensor mounted on a manually controlled Mo-tionControl MC 2 prosthetic hand.The BioTac sensor is based on the the work of Wet-
tels and Loeb (Wettels, 2012) and consists of a rigidcentral core surrounded by an elastic, fluid-filled skinwith fingerprint ridges like those of the human fin-
1www.syntouchllc.com2http://www.utaharm.com
Table 1: Textures used in controlled manipulationexperiments (labels match samples in Figure 5)
descriptionA smooth high-density fiberboard (HDF)B HDF with 1.59mm holes on 6.35mm gridC HDF with 3.18mm holes on 6.35mm gridD sensor skin (textured silicone rubber)E HDF with 6.35mm wide rectangular slatsF 60 grit sandpaperG 150 grit sandpaperH 400 grit sandpaper
Figure 6: Crossection of a BioTac R�sensor demon-strates its construction and the location of varioussensing elements. The pressure sensor and impedanceelectrodes are useful in slip discrimination, mimick-ing the FA-II and FA-I human mechanoreceptors, re-spectively. Reproduced with permission of SynTouch
LLC
gertip (Figure 6). The sensor includes thermal, vi-bration and skin deformation sensing capabilities, ofwhich the latter two are useful for slip and textureclassification.
Skin deformation is sensed via an array of 19 pas-sive sensing electrodes distributed across the surfaceof the inner core of the sensor. These electrodes mea-sure the impedance through the fluid to 4 active ex-citation electrodes at 100Hz. In addition, as the fin-ger rubs across surfaces, vibrations are detected bya hydro-acoustic pressure transducer with a filteredbandwidth of 1040Hz. The signals from this dynamicsensor, roughly akin to the human FA-II mechanore-ceptors, have been used to characterize surface tex-tures (Fishel and Loeb, 2012).
Figure 7: (Experiments with BioTac R�sensors used apassively loaded prosthetic MC Hand R�equipped withtwo opposing sensors.
A single embedded microcontroller performs se-quential analog-to-digital conversions of the signalsdescribed above, in addition to temperature informa-tion and low frequency pressure measurements. Thedifferent measurements are interleaved, such that ev-ery high frequency pressure reading is followed bya different electrode, low-frequency pressure, or tem-perature measurement. This results in a 4400Hz sam-ple rate for the pressure sensor and a 100Hz sam-ple rate for each electrode. Up to three BioTacs canbe synchronized such that each pressure or electrodemeasurement is simultaneous interfinger, while theelectrodes are sampled sequentially intrafinger.
A MotionControl MC prosthetic hand was instru-mented with two BioTacs in the thumb and forefin-ger positions as in Figure 7. The hand, staticallymounted on a stand, was closed upon objects pas-sively with a grip force of ∼2.5N using a pre-loadspring. Motion during the experiments was producedmanually by the experimenter.
Four separate motions were performed in order toproduce slip at the two interfaces (Table 2). In thefirst, the MC hand was closed upon one of the textureplates which was pulled through the grasp via exter-nal weights, inducing slip between the fingers and thetexture plate. The other three motions produced slip
Table 2: Types of motions used to generate differentslip conditions on the BioTac R�experimental system
No. Description Slip1 Held object drops with ad-
ditional weights.object/hand
2 Stylus drags across surfaceof held object.
object/world
3 Soft secondary grippergrasps object and pulls.
object/world
4 Hard secondary grippergrasps object and pulls.
object/world
between the held texture plate and an external ob-ject. The second scenario consisted of bracing theheld texture plate against a vertical rod to preventlarge oscillations while a blunt aluminum stylus, ap-proximately 15 cm long with a tip radius of 3mm,was dragged across it with a normal force of approxi-mately 2N. The third and fourth motions were iden-tical except for the contact surfaces involved. Bothutilized an additional passive gripper mounted on avertical shaft to pinch the texture plate. This addi-tional gripper was manually drawn along the shaft,slipping against the texture plate. In one case thegripper’s surface was hard plastic while in the otherthe gripping surface was covered with the same sili-cone skin used for texture D.In these experiments slip was produced at either a
low speed, of approximately 2 cm/s or a high speedof approximately 5 cm/s. Each motion, texture, andspeed combination was performed 10 times for a totalof 640 trials. Data were recorded using custom soft-ware provided by Syntouch from both BioTacs for aperiod of 3 seconds, including the slip events of inter-est. The data were manually cropped after collectionby visually observing the pressure signal to determinewhen the trial occurred in that 3 s window.
4.2 Capacitive Sensors and Adaptive-
Gripper
For the second set of experiments, capacitive sensorswere affixed to a Robotiq Adaptive Gripper R�. Thesuite, shown fully installed on the hand in Figure 8,
consists of 132 taxels arrayed across the grasping sur-faces. The taxels have an average area of ∼1cm2 andare covered with a silicone rubber skin textured with1.6mm tall posts with 2mm diameter hemispheri-cal ends with a 77% packing ratio (same as textureD in Figure 5). This configuration yields a sensorand skin with stiffness of 300N/m and low force sen-sitivity of 6mN. In addition to the capacitive tax-els, each distal phalanx is further equipped with athree-axis accelerometer for measuring vibrations ofthe sensor package on that phalanx. The sensor de-sign is covered in (Aukes et al, 2014; Heyneman andCutkosky, 2012). The sensors have a physical band-width of > 200Hz and are sampled by an Analog De-vices AD7147 capacitive to digital converter (CDC).Each AD7147 sequentially samples up to 12 taxels ata rate of 300Hz.
Figure 8: The Robotiq AdaptiveGripper R�with thecapacitive sensing suite installed (a). The full suiteincludes 132 capacitive taxels and 3, 3-axis ac-celerometers mounted on the back of each distal pha-lanx. Insets show closeups of the outer skin textureand the configuration of the capacitive pads on theinner surface (b) and back (c) of the distal phalanx,as well as the location of the 3-axis accelerometer.
The Robotiq AdaptiveGripper equipped with thecapacitive sensor suite was mounted on a position-controlled AdeptOne 5-axis SCARA robot arm, (Fig-ure 9) to move the gripper into position and pro-duce manipulation primitives to generate both types
of slip conditions. The gripper was used at its loweststrength setting producing grip forces of ∼15N duringall trials.
Figure 9: Experiments using the capacitive sensorsuite for the AdaptiveGripper R�were performed withthe gripper mounted on an AdeptOne robot arm.Manipulation primitives were scripted via simple co-ordinated motion of the arm and gripper, while datawere recorded from the tactile sensor suite.
Five motions were executed, as outlined in Table3. Motion 4 was repeated twice; once with a roundedhemispherical stylus and once with a sharp pointedstylus. Motions 1 and 2 both produce slip betweenthe finger surface and the texture plate, while mo-tions 3-5 produce slip between the held texture plateand the environment. Motion 5 was specifically cho-sen to be difficult. In this case the surfaces involvedat both interfaces are the same: texture plate andsilicone molded skin. The speed was held constant inall trials and each motion/texture combination wasrepeated 10 times for a total of 480 trials. Markerswere automatically inserted into the captured dataafter the Adept had finished accelerating into its mo-tion and after the motion had completed to allow forautomatic extraction of relevant slip data.
Table 3: Manipulation primitives used in experimentswith the AdeptOne
No. Description Slip1 Pinch stationary object and
pull awayobject/hand
2 Rub single finger across astationary object
object/hand
3 Remove object wedged (butnot clamped) in a vise open-ing
object/world
4 Drag stylus across station-ary object
object/world
5 Drag object across patch ofsilicone with skin texture
object/world
4.3 PVDF Sensors and Benchtop
Platform
The final set of experiments utilized another customsensor design. These sensors incorporate embeddedstrips of polyvinylidene-flouride, or PVDF, a piezo-electric polymer often used for dynamic tactile sens-ing (Cutkosky and Ulmen, 2014). The embeddedstrips are thin and flexible, having little effect on theelastomers in which they are embedded.
The fingertips are designed to be interchangeablewith those of the Robotiq Adaptive Gripper and con-tain up to 8 strips of PVDF, along with an AnalogDe-vices AD7608 digital acquisition chip. The chip pro-vides variable sampling rates, matched analog anti-aliasing prefilters, and simultaneous sampling on upto 8 channels. Due to communication throughputconsiderations in the rest of the acquisition system,the sensors were sampled at 3.125kHz, with an anti-alias filter with a cutoff at 1.5kHz.
Although they can be mounted to the Adaptive-Gripper, the experiments with the PVDF sensorswere performed on a dedicated benchtop test plat-form designed for controlled slip experiments. As il-lustrated in Figure 11, the platform consists of threeengagement mechanisms arranged around a centrallymounted texture sample. Each fingertip is broughtinto contact with the sample via a spring loaded 4-barmechanism, and can then be moved vertically using
Figure 10: An exploded view of a PVDF sensor fin-gertip shows the basic construction. A two step mold-ing process with both an inner core and outer skin al-lows the position of the PVDF strip(s) to be tightlycontrolled. Rivets connect the strip(s) to the circuitboard, which is bonded to a 3D printed finger back-ing. The inset shows a completed sensor.
a linear slider. The texture sample is mounted on acommercial 6-axis JR3 force/torque sensor with 1kHzbandwidth. This sensor provides an initial contactsignal for calibration as well as total shear force dur-ing experiments and an independent measure of thevibrations experienced by the sample. An infraredposition sensor is mounted inside each engagementmechanism to measure motion of the samples before,during, and after slip.
As in the BioTac experiments, slip was producedmanually by the experimenter. While the system al-lowed for much more sensing of the entire process,the breadth of motions was much more limited. Twofingertips with sensors, and a third dummy finger-tip with no sensors, were brought into contact withthe sample. Then either the dummy or the two fin-gertip sensors were manually slid along the vertical
Figure 11: (Left) A top view of the test platformshows the arrangement of engagement mechanismaround the JR3 force/torque sensor where the tex-ture samples are mounted. (Right) A side view of theengagement mechanism highlights the parallel four-bar mechanism which is used to bring sensors or inertobjects into contact with the texture samples and thevertical slider track which allows them to slip againstit.
tracks. Sliding the passive dummy fingertip producesobject/world vibrations that are transmitted throughthe object to the sensor fingertips; sliding the sensorfingertips produces object/finger slippage.For these experiments the texture samples were
limited to the three sandpapers used in the othertests. Contact forces of ∼5.5N and ∼10N were pro-duced by varying the preload springs in the engage-ment mechanisms. To reduce abrasive wear, the ob-ject samples were covered with a thin sprayed-on rub-ber film. Each experiment was repeated 10 times,resulting in a total of 120 trials.
5 Classification Results
Once relevant features are extracted from raw sen-sor data there are many algorithms that can be ap-plied in order to make classifications. Since the focusof this research is on the design of physically moti-vated and extendable features we evaluate their per-
formance using a simple naive-Bayes maximum like-lihood estimator (MLE). The features are extractedfrom each trial and grouped based on their slip type:object/hand vs. object/world. Using a 10-fold val-idation scheme each group is split in to 10 sets andeach used as the validation set in turn. The remain-ing 9 sets not used for validation are instead used tofit the features from each class to a normal distribu-tion. The features from the validation trials are thenassigned slip classes to maximize the likelihood of themeasured features, given the probability distributionsinferred from the test sets.The performance of a feature is simply the num-
ber of trials that were correctly identified using thisprocedure. While this is certainly not the most so-phisticated method for using these features, it doesprovide an equal footing for comparison.
5.1 General Feature Performance
Either of the features can be computed over any fre-quency range, including the entire signal spectrum.While the results from the naive-Bayes MLE serve asa quantitative comparison, performance can be qual-itatively demonstrated by graphing the classifiers fora range of frequencies. In Figure 12a, the feature G2
(eq. 34) is computed for the BioTac sensor experi-ments in frequency bands of 10Hz. The 10 trials fora given experiment are averaged to produce each datapoint in the figure. In this plot, as well as Figures 15,17, and 18, unfilled data points indicate object/handslip while filled indicate object/world slip trials. Tosimulate the action of the naive-Bayes MLE, Figure12b shows the regions bounded by one standard de-viation for the sets of all trials for both classes ofslip. In this example one can see very good separa-tion of these two groups, suggesting that the MLEwill adequately classify examples.In order to investigate the general performance of
these features the naive-Bayes MLE is used in twoways. First, the features are computed using theentire spectrum. Second, the frequency band whichgives the maximum accuracy is determined by com-paring the performance of the features when com-puted across all possible bands. The latter approachcorresponds to the case where a sufficient body of
0 200 400 600 800 1000
Freq [Hz]
−0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
G2
Trial Averages
Manip. (1)
Manip. (4)
Manip. (3)
Manip. (2)
0 200 400 600 800 1000
Freq [Hz]
−0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
G2
Statistical Regions
Object/Hand slip
Object/World slip
BioTac Multi-Finger G2 (No Preprocessing)
Figure 12: A visual comparison of feature perfor-mance for the average G2 value across the 10 trialsfor each BioTac experiment at a variety of frequen-cies (a). The Object/Hand slip trials (Manip. 1 fromTable 2) are shown as unfilled circles; Manip. 2,3,4denote Object/World slips. The regions describingthe examples of each type of slip can then be de-fined by the average (solid line) and standard devi-ation (dotted line) at each frequency (b). The G2
value is generated using the two pressure sensors inthe BioTac fingers, an example of interfinger featureanalysis.
data can be collected such that the features aretrained based on that body.Tables 4-6 show the quantitative performance us-
ing the naive-Bayes MLE as described above. Thefeature G2 has been calculated from a single finger, aswell as for multiple fingers. In the case of the BioTac,the single-finger G2 utilizes the array of impedanceelectrodes while the multi-finger G2 uses the high-frequency pressure signals from the two fingers.While some combinations of sensors and features
work across the entire spectrum, others do not; forexample computing the single-finger G2 with the Bio-
Table 4: Power ratio Γ discrimination performance
Sensor Full Best (f, w) [Hz]spectrum band
BioTac 50.39% 76.33% (23.5, 0.5)Capacitive 83.37% 89.55% (0.5, 0.5)PVDF 86.55% 87.39% (122.6, 2.5)
Table 5: Single Finger Group Coherence G2 Perfor-mance
Sensor Full Best (f, w) [Hz]spectrum band
BioTac 49.38% 79.84% (48.0, 2.0)Capacitive 66.31% 67.8% (135.0, 2.0)PVDF 92.86% 98.74% (1540, 7.5)
Tac electrode data is only marginally better thanrandomly guessing. For combinations with poor per-formance it is important to consider how knowledgeabout the sensor may help in preprocessing the dataor choosing a frequency band to use, without resort-ing to the generation of a large learning set.
5.2 Augmenting with Sensor Specific
Knowledge
Although the two features are meant to be generaliz-able across sensors, they are not required to be sen-
sor agnostic. In this section we examine propertiesof each of the sensors that can be used to improvedperformance.There are two general categories of sensor-specific
preprocessing that were discussed in Section 3;sequential-sampling correction and contact estima-
tion. For the BioTac and capacitive sensors, whichsample arrays sequentially, the taxel data are trans-formed into the frequency domain and a pure time-delay applied to each signal individually with the ap-propriate sampling delay. This yields a new set ofsignals which estimate a simultaneously sampled sys-tem. Note that this correction only affects the cal-culation of Γ, since signal coherence, including G2, isinsensitive to constant phase shifts due to time de-lays. Contact estimation is covered in the sections
Table 6: Multi Finger Group Coherence G2 Perfor-mance
Sensor Full Best (f, w) [Hz]spectrum band
BioTac 75.78% 86.72% (5.0, 5.0)Capacitive 59.49% 75.9% (146.5, 2.5)PVDF 99.16% 100% (22.55, 2.5)
below for each type of sensor.
5.2.1 BioTacs
The fluid-filled skin construction of the BioTacs in-troduces physical coupling between measurementsacross the electrode array. Because the fluid is in-compressible, as the skin compresses in one area itbulges away from the core in others (Figure 13), lead-ing to increased impedance at the contact locationand decreased impedance away from it. A simplemethod for contact estimation is to look for electrodeswhich have an impedance above the nominal by somethreshold amount. In this case, that threshold is de-termined by the noise on the sensor estimated fromthe standard deviation of the signals seen from thesensor at rest with no contact.
Figure 13: The fluid-filled elastomeric skin results instrong coupling between impedance measurements.A contact force with a significant shear componentcauses the skin to compress on one side and bulgeon the other (b), while a purely normal contact forcecauses both sides to bulge as the contact area com-presses (b). Modified from (Wettels and Loeb, 2011)
After accounting for both sequential sampling andcontact estimation, the new performance values forthe features on the BioTac data are shown in Fig-
ure 14. In all cases the corrected data either showsimprovement in classification performance or is indis-tinguishable from the uncorrected versions.
Γ Single-Finger G2
Multi-Finger G2
0
20
40
60
80
100
Accuracy[%
]
50.4 49.4
75.8
87.7
54.8
75.6
Full Spectrum
no correction
with correction
Γ Single-Finger G2
Multi-Finger G2
0
20
40
60
80
100
76.379.8
86.788.7
79.6
86.9
Best Band
Biotac Performance with and without Correction
Figure 14: The classification performance of the var-ious features are shown for the BioTac sensors usingboth original, uncorrected data as well as data thathas been corrected for sequential sampling and usesthe estimated contact location. These sensor specificprocessing steps result in significant improvementsfor some features and negligible changes for others,while none are negatively affected.
5.2.2 Capacitive Sensors
The capacitive sensors have little coupling betweentaxels. However, they are affected by being mountedon a hard, flat surface in contact with flat objects.Upon examining the qualitative results of the fea-ture Γ (Figure 15) it is clear that the examples ofobject/hand slip have generally higher values thanobject/world slip, in contrast to what one might ex-pect based on human mechanoreception and what isseen in the other two sensors.This result can be explained by considering the
flat-on-flat nature of the AdaptiveGripper/objectcontacts. Figure 16 demonstrates the different mech-anisms at work in this scenario. When an object slipsagainst an external object (Figure 16a) it predomi-nantly rotates in the fingertip grasp of the Adaptive-
0 20 40 60 80 100 120 140
Freq [Hz]
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Γ
Trial Averages
Manip. (2)
Manip. (1)
Manip. (3)
Manip. (5)
Manip. (4b)
Manip. (4a)
0 20 40 60 80 100 120 140
Freq [Hz]
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Γ
Statistical Regions
Object/Hand slip
Object/World slip
Capacitive Power Ratio (Contact & Sampling)
Figure 15: The visual comparison for the capaci-tive sensors demonstrates that Γ for object/world slip(Manip. 1, 2 from Table 3) is generally smaller thanfor object/hand (Manip. 3-5), unlike what is pre-dicted from human mechanorecption. This inversionis likely due to the fact that the capacitive sensor ex-periments consisted primarily of flat-on-flat contactconditions.
Gripper. This rotation causes relatively large, out-of-phase signals on the taxels near the tip of the fin-ger relative to those at the base. These signals sig-nificantly cancel when calculating P�(ω) resulting insmall values of Γ. In contrast, when the object slipsin the grasp this cancellation is not nearly as domi-nant, meaning the magnitudes of P�(ω) and L(ω) areof a similar scale, and Γ is closer to 1.
5.2.3 PVDF Sensors
For the PVDF sensor, the vibration frequencies ofthe surface texture elements are of importance whenobjects slide over the surface of the finger. Using a fi-nite element model and sliding tests, it was confirmedthat the fundamental vibration frequency of individ-
(a)
Finger
Taxel A Taxel BDielectric
Pillars
Object
(b)
Finger
Taxel A Taxel B
Dielectric
Pillars
Object
(a) Effect of slip between anobject and the environmentwhen held by the capacitivesensor fingertips
Finger
Taxel A Taxel BDielectric
Pillars
Object
(a)
(b)
Finger
Taxel A Taxel BDielectric
Pillars
Object
(b) Effect of slip betweenan object and the capacitivesensor fingertip
Figure 16: Because the surfaces of both the capacitivesensor fingertips and the texture plates used in the ex-periments are globally flat, they exhibit pathologicalresponses to the vibrations caused by object/world(left) and object/hand (right) slip. This is com-pounded by the fact that the taxels are relativelylarge compared to the surface texture features.
ual texture elements was at approximately 1360Hz.The classification features also show the largest sepa-ration for frequencies in this approximate range (Fig-ure 17).
Slip between the object and hand will excite vi-brations at this frequency as individual surface tex-ture elements experience stick-slip and resonate atthis fundamental frequency, generating significant in-coherent signal content. In contrast, slip between theobject and environment will not generally exhibit sig-nificant content at this frequency. By focusing on fre-quencies that are indicated by properties of the sen-sor itself, like the fundamental frequency of vibrationof surface features, the signal processing and classifi-cation will operate on data which best captures theinherent differences between slip at the two differentinterfaces.
0 200 400 600 800 1000 1200 1400
Freq [Hz]
0.0
0.2
0.4
0.6
0.8
1.0
1.2
G2
Trial Averages
Manip. (1)
Manip. (2)
Manip. (3)
Manip. (4)
0 200 400 600 800 1000 1200 1400
Freq [Hz]
0.0
0.2
0.4
0.6
0.8
1.0
1.2
G2
Statistical Regions
Object/Hand slip
Object/World slip
PVDF Multi-Finger G2 (Contact Estimation)
Figure 17: The visual comparison G2 between thetwo PVDF sensors highlights the utility of charac-terizing important frequencies in a sensor design.The largest separation between object/world and ob-ject/hand slip examples occurs above 1300Hz. Thiscorresponds very closely to the first mode frequenciesof the texture elements on the sensor surface, whichare primarily excited during object/hand slip.
5.3 Biomimetic Sensors vs. Bioin-
spired Processing
The power-ratio classification feature, Γ, presentedin Section 3.1 can be computed for any tactile arraysensor. While inspired by the function of the two rel-evant human mechanoreceptors it does not require,and may not produce comparable results with, a het-erogeneous bio-inspired sensing suite.
Sensing suites described as biomimetic often usetwo or more types of sensors, each measuring a differ-ent quantity through different transduction methodsand/or placement, in order to more directly mimicthe FA-I and FA-II mechanoreceptors in the humanhand. In the classification feature Γ presented herewe have relied upon the ratio taken in Equation (6)
to cancel common-mode amplitude effects on boththe individual and ensemble measures due to differ-ent slip conditions. While this is likely to be truewhen both measures are calculated from a single ar-ray, it may not be true when the signals are generatedby diverse sensors. Similarly, for the coherence fea-ture using two different sets of sensors invalidates themodeling used in its derivation and makes a relativecomparison of group coherence less meaningful. Forexample, one sensor may generally have higher co-herence than the other due to the mechanics of thetransduction, regardless of the type of slip involved.As an example, if we compute the Γ feature for
a combination of electrode and pressure sensors inthe BioTac sensor, Figure 18, it is readily apparentthat there is no region in which the two types of slipdemonstrate reasonable separation. Indeed, when theclassifier is qualitatively evaluated, the classificationaccuracy drops by 8.9%.
0 10 20 30 40 50
Freq [Hz]
−40
−20
0
20
40
60
80
100
120
140
Γ
Trial Averages
Manip. (1)
Manip. (4)
Manip. (3)
Manip. (2)
0 10 20 30 40 50
Freq [Hz]
−40
−20
0
20
40
60
80
100
120
140
Γ
Statistical Regions
Object/Hand slip
Object/World slip
BioTac Biomimetic Power Ratio (Contact & Sampling)
Figure 18: When using two separate sensors to moredirectly mimic FA-I and FA-II mechanoreceptors, theeffectiveness of the classification feature Γ decreases,as indicated by the significant overlap between trialsof different types of slip.
6 Conclusions and Future
Work
This work focused on a new tactile classification prob-lem relevant to advanced grasping and manipulationin real world environment: determining whether slip-page has occurred between the hand and the held ob-ject or between the held object and the environment.By considering the function of human mechanorecep-tors in the same scenarios and the mechanics of tactilesensors we proposed two simple, frequency-based fea-tures that can be extracted from tactile array signals.
These features have been evaluated using three dif-ferent tactile array sensors in three different experi-mental setups. With appropriate sensor-specific pro-cessing steps, such as contact estimation and sequen-tial sampling correction, a simple naive-Bayes maxi-mum likelihood estimator using 10-fold cross valida-tion was able to achieve> 85% classification accuracyfor the majority of the feature and sensor combina-tions.
Some of the experiments included manually gener-ated motions, producing only moderately controlledand repeatable trajectories, as typically experiencedin manipulating objects. We believe the variationsin slip speed, contact force, or other contact condi-tions caused by this manual process are beneficial indemonstrating the insensitivity of these classificationfeatures to those factors. As an example, when sen-sors with regular surface textures slip they generatefrequency content based on the texture’s spatial fre-quency and the slip speed. If slip speed were con-trolled tightly from trial to trial the result could bea misapprehension that a certain frequency range isparticularly useful.
We discussed the utility of sensor characterizationfor improving the generalizability of these features foruse with other sensors. For example, understandinghow to perform contact estimation, and which fre-quencies will be preferentially excited will help gen-erate the most discriminating features. Our resultsdemonstrate that these features can be used to dis-tinguish between object/world and object/hand slipeven when using broad-spectrum data. However, asdiscussed for the PVDF sensors, we expect they will
be most effective when the sensor mechanics, sensorelectronics, and signal processing methods are work-ing together: the surface texture should naturallygenerate signals at the frequency of peak sensitivityfor the electronics, and the methods described hereshould use the signal content around that frequency.The data presented here were post-processed. In
implementing a real-time system the processing re-quired to generate these features is not prohibitive.For example, the process of computing the the co-herence feature for an array of 20 sensors using 1024sample points and Welch’s algorithm (256 samplewindows, 128 sample overlap) takes 23.9ms, domi-nated by the time required to compute the largesteigenvalue. This benchmark is taken using unopti-mized python code running on a 2.5GHz i7 proces-sor. Instead, the main trade-off will be between thetime required for an accurate spectral estimate andthe acceptable latency in the system. Consideringa goal similar to human reaction times on the orderof 200ms, sensors with a lower sample-rate like theBioTac impedance array at 100Hz will have troublemaking this trade-off, as there will not be enough newsamples (20 in this example) from which to make arobust spectral estimate.The classification features presented here are not
intended to be used in isolation. To reduce the inci-dence of false positive and false negative errors in slipdetection it will be important to take advantage of ad-ditional information from joint-torque sensors, con-tact sensors, low-frequency tactile arrays, and evenvision systems. These sensors can be used to estimatethe contact state of the hand/arm/object system inorder to inform the use of any event classificationalgorithms. For example, when manipulating an ob-ject, knowledge about the proximity of other itemsin the environment may generate a prior probabilitydistribution for use with classification methods likethe naive-Bayes MLE used here; if there are no ex-ternal objects in the vicinity of the hand it is quiteunlikely that sensed vibrations in the tactile arraysare due to object/world slip.Although not explicitly tested in this work, there
are two situations in which a constant slip classifi-cation will generate false-positives when neither slipis occurring and noise or other vibrations in the sys-
tem or environment are sensed by the tactile arrays.Because noise sources are typically common to allchannels (e.g. power supply noise, EMF coupling,etc.) and therefore coherent, we would expect falseobject/world slip detection. In order to alleviate thisissue, characterizing the noise floor of the array is nec-essary so that classification is not performed when thesensor signals lie below it. The second situation couldoccur in a variety of scenarios such as a task beingperformed on a mobile base (vibrations of the movingbase are picked up by all the sensors) or if another ob-ject slips along a nearby, but unsensorized surface ofthe hand. In these cases we expect the classificationto identify object/world slips more often. Correctlyidentifying periods of no slip in these conditions willlikely require more general system knowledge, as dis-cussed above.Finally, this work has explored the use of data from
multiple fingers in order to classify slip by computingthe G2 feature across fingers. In real manipulationsslip may occur at any of the many interfaces betweenthe object and the multiple fingers and the environ-ment. A further extension to coherence, following thesame principles as the G2, may be helpful in deter-mining the coherence between populations of sensorsrelative to the coherence within those groups in orderto identify all interfaces at which slip is occurring atany given moment.
7 Acknowledgments
This work has been supported in part by the DARPAARM-H project with additional support from ARLMAST.
References
Ascari L, Bertocchi U, Corradi P, Laschi C, Dario P(2009) Bio-inspired grasp control in a robotic handwith massive sensorial input. Biological Cybernet-ics 100(2):109–128
Aukes DM, Cutkosky MR, Kim S, Ulmen J, Gar-cia P, Stuart H, Edsinger A (2014) Design andTesting of a Selectively Compliant Underactuated
Hand. International Journal of Robotics Research33(5):721–735
Brisben AJ, Hsiao SS, Johnson KO (1999) Detec-tion of Vibration Transmitted Through an ObjectGrasped in the Hand. Journal of Neurophysiology81:1548–1558
Cutkosky MR, Ulmen J (2014) Dynamic Tactile Sens-ing. In: Balasubramanian R, Santos VJ (eds) TheHuman Hand as an Inspiration for Robot Hand De-velopment SE - 18, Springer Tracts in AdvancedRobotics, vol 95, Springer International Publish-ing, pp 389–403
Dahiya RS, Metta G, Valle M, Sandini G (2010) Tac-tile Sensing From Humans to Humanoids. IEEETransactions on Robotics 26(1):1–20
Decherchi S, Gastaldo P, Dahiya RS, Valle M, ZuninoR (2011) Tactile-Data Classification of ContactMaterials Using Computational Intelligence. IEEETransactions on Robotics 27(3):635–639
Fearing RS, Hollerbach JM (1985) Basic Solid Me-chanics for Tactile Sensing. The International Jour-nal of Robotics Research 4(3):40–54
Fishel JA, Loeb GE (2012) Bayesian Exploration forIntelligent Identification of Textures. Frontiers inNeurorobotics 6(June):1–20
Fox J, Daniel T (2008) A neural basis for gyroscopicforce measurement in the halteres of Holorusia.Journal of Comparative Physiology A 194(10):887–97
Fox JL, Fairhall AL, Daniel TL (2010) Encodingproperties of haltere neurons enable motion featuredetection in a biological gyroscope. Proceedings ofthe National Academy of Sciences of the UnitedStates of America 107(8):3840–5
Fujimoto I, Yamada Y, Morizono T, Umetani Y,Maeno T (2003) Development of artificial fingerskin to detect incipient slip for realization of staticfriction sensation. Proceedings of IEEE Interna-tional Conference on Multisensor Fusion and Inte-gration for Intelligent Systems, MFI2003 pp 15–20
Greenstein J, Kavanagh P, Rowe MJ (1987) Phasecoherence in vibration-induced responses of tactilefibres associated with pacinian corpuscle receptorsin the cat. The Journal of Physiology 386:263–275
Gunji D, Mizoguchi Y, Teshigawara S (2008) Grasp-ing Force Control of Multi-fingered Robot Handbased on Slip. In: IEEE International Conferenceon Robotics and Automation, pp 2605–2610
Heyneman B, Cutkosky MR (2012) Biologically in-spired tactile classification of object-hand andobject-world interactions. In: IEEE InternationalConference on Robotics and Biomimetics
Ho AV, Hirai S (2014) Mechanics of Localized Slip-page in Tactile Sensing, Springer Tracts in Ad-vanced Robotics, vol 99. Springer InternationalPublishing, Cham
Holweg E, Hoeve H, Jongkind W, Marconi L, Mel-chiorri C, Bonivento C (1996) Slip detection by tac-tile sensors: algorithms and experimental results.In: IEEE International Conference on Roboticsand Automation, April, pp 3234–3239
Hosoda K, Tada Y, Asada M (2006) Anthropo-morphic robotic soft fingertip with randomly dis-tributed receptors. Robotics and Autonomous Sys-tems
Howe RD, Cutkosky MR (1989) Sensing skin accel-eration for slip and texture perception. In: IEEEInternational Conference on Robotics and Automa-tion, IEEE Comput. Soc. Press, pp 145–150
Howe RD, Cutkosky MR (1993) Dynamic tactilesensing: perception of fine surface features withstress rate sensing. IEEE Transactions on Roboticsand Automation 9(2):140–151
Hunt C (1961) On the nature of vibration receptorsin the hind limb of the cat. Journal of Physiologypp 175–186
Jamali N, Sammut C (2011) Majority Voting :Material Classification by Tactile Sensing UsingSurface Texture. IEEE Transactions on Robotics27(3):508–521
Jamali N, Sammut C (2012) Slip prediction usingHidden Markov models: Multidimensional sensordata to symbolic temporal pattern learning. In:2012 IEEE International Conference on Roboticsand Automation, IEEE, pp 215–222
Jentoft LP, Tenzer Y, Vogt D, Wood RJ, HoweRD (2013) Flexible, stretchable tactile arrays fromMEMS barometers. In: 2013 16th InternationalConference on Advanced Robotics (ICAR), IEEE,pp 1–6
Johansson RS, Birznieks I (2004) First spikes inensembles of human tactile afferents code com-plex spatial fingertip events. Nature Neuroscience7(2):170–177
Johansson RS, Flanagan JR (2009) Coding and useof tactile signals from the fingertips in object ma-nipulation tasks. Nature Reviews: Neuroscience10(5):345–359
Johansson RS, Westling G (1984) Roles of glabrousskin receptors and sensorimotor memory in auto-matic control of precision grip when lifting rougheror more slippery objects. Experimental Brain Re-search pp 550–564
Johnson M, Adelson E (2009) Retrographic sensingfor the measurement of surface texture and shape.In: 2009 IEEE Conference on Computer Vision andPattern Recognition, Ieee, pp 1070–1077
Kim SH, Engel J, Liu C, Jones DL (2005a) Textureclassification using a polymer-based MEMS tactilesensor. Journal of Micromechanics and Microengi-neering 15(5):912–920
Kim SH, Engel J, Liu C, Jones DL (2005b) Textureclassification using a polymer-based MEMS tactilesensor. Journal of Micromechanics and Microengi-neering 15(5):912–920
Knibestol M (1973) Stimulusresponse functions ofrapidly adapting mechanoreceptors in the humanglabrous skin area. The Journal of physiology232:427–452
Lepora NF, Evans M, Fox CW, Diamond ME, Gur-ney K, Prescott TJ (2010) Naive Bayes textureclassification applied to whisker data from a mov-ing robot. In: The 2010 International Joint Confer-ence on Neural Networks (IJCNN), IEEE, pp 1–8
Lin CH, Erickson TW, Fishel JA, Wettels N, LoebGE (2009) Signal Processing and Fabrication ofa Biomimetic Tactile Sensor Array with Thermal,Force and Microvibration Modalities. In: IEEE In-ternational Conference on Robotics and Biomimet-ics
Lucarotti C, Oddo CM, Vitiello N, Carrozza MC(2013) Synthetic and bio-artificial tactile sensing:a review. Sensors (Basel, Switzerland) 13(2):1435–66
Maheshwari V, Saraf RF (2006) High-resolution thin-film device to sense texture by touch. Science (NewYork, NY) 312(5779):1501–1504
Marple SLJ (1987) Digital Spectral Analysis: withapplications. Prentice-Hall, Englewood Cliffs, NJ
Mazid a, Russell R (2006) A Robotic Opto-tactileSensor for Assessing Object Surface Texture. 2006IEEE Conference on Robotics, Automation andMechatronics pp 1–5
Melchiorri C (2000) Slip Detection and Control UsingTactile and Force Sensors. IEEE/ASME Transac-tions on Mechatronics 5(3):235–243
Mittendorfer P, Cheng G (2011) Humanoid Mul-timodal Tactile-Sensing Modules. IEEE Transac-tions on Robotics 27(3):401–410
Oddo CM, Controzzi M, Beccai L, Cipriani C, Car-rozza MC (2011) Roughness Encoding for Dis-crimination of Surfaces in Artificial Active-Touch.IEEE Transactions on Robotics 27(3):522–533
Ramırez D, Vıa J, Santamarıa I (2008) A generaliza-tion of the magnitude squared coherence spectrumfor more than two signals: definition, propertiesand estimation. In: IEEE International Confer-ence on Acoustics, Speech and Signal Processing,pp 3769–3772
Romano JM, Hsiao K, Niemayer G, Chitta S,Kuchenbecker KJ (2011) Human-Inspired RoboticGrasp Control with Tactile Sensing. IEEE Trans-actions on Robotics (99):1–13
Schmitz A, Maiolino P, Maggiali M, Natale L, Can-nata G, Metta G (2011) Methods and Technolo-gies for the Implementation of Large-Scale RobotTactile Sensors. IEEE Transactions on Robotics27(3):389–400
Sinapov J, Sukhoy V, Sahai R, Stoytchev A (2011)Vibrotactile Recognition and Categorization ofSurfaces by a Humanoid Robot. IEEE Transactionson Robotics 27(3):488–497
Teshigawara S, Tsutsumi T, Shimizu S, Suzuki Y,Ming A, Ishikawa M, Shimojo M (2011) Highly sen-sitive sensor for detection of initial slip and its ap-plication in a multi-fingered robot hand. In: 2011IEEE International Conference on Robotics andAutomation, IEEE, pp 1097–1102
Tiwana MI, Redmond SJ, Lovell NH (2012) A reviewof tactile sensing technologies with applications inbiomedical engineering. Sensors and Actuators A:Physical 179:17–31
Wettels N (2012) Biomimetic tactile sensor for ob-ject identification and grasp control. Ph.d. thesis,University of Southern California
Wettels N, Loeb GE (2011) Haptic Feature Extrac-tion from a Biomimetic Tactile Sensor: Force, Con-tact Location and Curvature. In: IEEE Interna-tional Conference on Robotics and Biomimetics
Witney AG, Wing A, Thonnard JL, Smith AM(2004) The cutaneous contribution to adaptive pre-cision grip. Trends in Neurosciences 27(10):637–643
Yahud S, Dokos S, Morley JW, Lovell NH (2010) Ex-perimental validation of a polyvinylidene fluoridesensing element in a tactile sensor. In: IEEE Inter-national Conference of the Engineering in Medicineand Biology Society, vol 2010, pp 5760–5763
Yousef H, Boukallel M, Althoefer K (2011) Tac-tile sensing for dexterous in-hand manipulation inrobotics: a review. Sensors and Actuators A: Phys-ical 167(2):171–187
Zhang X, Liu R (2012) Slip detection by array-typepressure sensor for a grasp task. In: 2012 IEEEInternational Conference on Mechatronics and Au-tomation, IEEE, pp 2198–2202
A Properties of G2
This definition of G2x can be checked for four basic
properties which indicate it is a useful extension ofclassical coherence:• it has the same range as C2; G2
x ∈ [0, 1]• it is equivalent to C2 when N = 2• it takes on the extreme G2
x = 0 iff C2xi,xj
= 0 fori �= j
• it takes on the extreme G2x = 1 iff C2
xi,xj= 0 for
i �= j
The first property follows from the fact that Cx isa normalized hermitian matrix. As a hermitian ma-trix, all its eigenvalues are non-negative real values.Because it is normalized, its diagonal entries are all1’s and therefore its eigenvalues sum to N. These to-gether bound λ1 ∈ [1, N ], which leads to G2
x ∈ [0, 1].The second property can be verified by examining
Cx for N = 2:
Cx =
�1 Cx1,x2
C∗x1,x2
1
�
and the eigenvalues of this matrix are λ = 1±|Cx1,x2 |.Plugging into Equation (34) yields |Cx1,x2 |2, as ex-pected.To prove that G2
x only attains the extreme value of0 when all individual signals are pair-wise incoherentwe simply note that the only normalized hermitianmatrix with eigenvalue λ1 = 1 is the identity matrix.If Cx = I, then all the off-diagonal terms, the pair-wise coherence between signals, are 0 as expected.The proof for the conditions under which G2
x = 1is slightly more involved. First we note that λ1 = N
from Equation (34). This implies that λi = 0 fori �= 1 and therefore that rank (Cx) = 1. From thisrank constraint we know that we can decompose thecomplex coherence matrix as
Cx = v∗vT
=
v∗1v
T1 v
∗1v
T2 · · · v
∗1v
TN
v∗2v
T1 v
∗2v
T2 · · · v
∗2v
TN
......
. . ....
v∗Nv
T1 v
∗Nv
T2 · · · v
∗Nv
TN
for some vector v. From the constraint that all di-agonal elements of Cx must be 1, we have |vi| = 1.
Therefore all elements of Cx have magnitude 1, andwe have
C2x =
1 1 · · · 11 1 · · · 1...
.... . .
...1 1 · · · 1
as desired.To prove the reverse direction, first note that for a
set of completely coherent signals, x(t), each signalcan be defined as a filtered version of the first signal:
xi(t) = (hi ∗ x1) (t), or
x(t) = (h ∗ x1) (t), with
h(t) = [1, h2, . . . , hN ]T
This set of signals has the complex coherence ma-trix
Cx = h∗hT
This shows that rank (Cx) = 1, and therefore λ1 =N,λi = 0 for i �= 1. Finally, from Equation (34) thisyields G2
x = 1, as desired.