FINE LANDMARK-BASED SYNCHRONIZATION OF AD-HOC MICROPHONE ARRAYS

Tsz-Kin Hon, Lin Wang, Joshua D. Reiss and Andrea Cavallaro

Centre for Intelligent Sensing, Queen Mary University of London, UK{tsz.kin.hon, lin.wang, joshua.reiss, a.cavallaro}@qmul.ac.uk

ABSTRACTWe use audio fingerprinting to solve the synchronizationproblem between multiple recordings from an ad-hoc arrayconsisting of randomly placed wireless microphones or hand-held smartphones. Synchronization is crucial when employ-ing conventional microphone array techniques such as beam-forming and source localization. We propose a fine audiolandmark fingerprinting method that detects the time differ-ence of arrivals (TDOAs) of multiple sources in the acousticenvironment. By estimating the maximum and minimumTDOAs, the proposed method can accurately calculate theunknown time offset between a pair of microphone record-ings. Experimental results demonstrate that the proposedmethod significantly improves the synchronization accuracyof conventional audio fingerprinting methods and achievescomparable performance to the generalized cross-correlationmethod.

Index Terms— Synchronization, audio fingerprinting,microphone array

1. INTRODUCTION

Ad-hoc acoustic sensor networks composed of randomly dis-tributed wireless microphones or hand-held smartphones havebeen attracting increased interest due to their flexibility in sen-sor placement [1]. A challenge in such ad-hoc arrays is thatthe locations of the microphones are generally unknown andthere is no precise temporal synchronization between the mi-crophones. Traditional microphone-array techniques, such asbeamforming and sound source localization, which rely onthe knowledge of microphone positions and assume sample-synchronized audio channels, cannot be applied directly [2,3].The synchronization problem between multiple audio chan-nels has been addressed using generalized cross-correlation[2, 4, 5] and audio fingerprinting [5–8].

The generalized cross-correlation (GCC) method [9],which calculates the delay that maximizes the correlationcoefficient between two audio channels, is well known for itsaccurate time delay estimation and synchronization. How-ever, due to high computational cost, GCC is more suit-

This work was supported by the U.K. Engineering and Physical SciencesResearch Council (EPSRC) under Grant EP/K007491/1.

able for audio channels that are already coarsely synchro-nized [4, 5]. Another synchronization approach is based onaudio fingerprinting, which has been originally applied tomusic information retrieval [6], and clustering and synchro-nizing multi-camera videos [7, 8]. By matching the audiofingerprints extracted from the sound track, the audio chan-nels can be synchronized. Well-known audio fingerprintingalgorithms include onset, audio landmark, and Philips Ro-bust Hash (PRH). Onset is based on detecting the increaseof the signal energy and consists of audio features extractedfor example from 24 individual frequency bands [10, 11].PRH extracts more detailed features including all the energychanges in consecutive bands and frames, and is more robustthan onset to ambient noise and audio compression distor-tion [12]. Audio landmark operates in the time-frequencydomain, and the extracted features consist of a set of time-frequency pairs [6–8]. With low computational cost and ro-bustness against noise, audio fingerprinting techniques havebeen widely used in movie and video synchronization [13].However, the synchronization accuracy achieved by conven-tional audio fingerprinting methods is limited by the time-frequency analysis hop size, with typical values between afew and tens of milliseconds [10], which is enough for videoapplications but far below the sample-wise synchronizationrequirement in microphone array signal processing.

In this paper, we investigate audio-landmark-based syn-chronization, aiming at sample-wise alignment within anad-hoc microphone array. We show that by reducing thetime-frequency analysis hop size, the classical landmarkaudio fingerprinting algorithm is able to detect the time dif-ference of arrival (TDOA) of nearby sound sources that arecaptured by different microphones. Existing audio finger-printing methods usually use a large hop size, e.g. tens tohundreds milliseconds, in the application of music informa-tion retrieval or video synchronization. The TDOA informa-tion of multiple sources becomes ambiguous with such a hopsize. We further exploit this property to detect the maximumand minimum TDOAs around the microphone array, whichcan be used to further improve the synchronization accuracy.A sample-accuracy synchronization can be achieved with theproposed algorithm if a sufficient number of sound sources islocated around the array.

23rd European Signal Processing Conference (EUSIPCO)

978-0-9928626-3-3/15/$31.00 ©2015 IEEE 1341

2. PRELIMINARIES

Consider an anechoic environment with an ad-hoc micro-phone array consisting of M independent microphones andunknown number of N sources randomly distributed aroundthe array. The locations of the microphones and sources areunknown and are denoted as M = [m1, · · · ,mi, · · · ,mM ]and R = [r1, · · · , rb, · · · , rN ], respectively. The recordingai(t) at each microphone is asynchronous with unknown starttime Ti, i = 1, · · · ,M . Let the pairwise time offset betweentwo microphones i and j be denoted as

Tij = Ti − Tj . (1)

The signal recorded at each microphone is denoted asai(t) =

∑Nb=1 gibsb(t − tib), i = 1, · · · ,M , where sb(·),

tib and gib are the b-th source signal, the propagation timeand the attenuation from the b-th source to the i-th micro-phone, respectively. The propagation time of arrival (TOA)tib from the b-th sound source to the i-th microphone istib = ||rb−mi||

c , where c is the speed of sound and || · ||denotes the Euclidean distance. The TDOA of the b-th sourcebetween the i-th and j-th microphones, τijb, can be expressedas

τijb =||rb −mi|| − ||rb −mj ||

c+ Tij . (2)

The goal is to blindly estimate the time offset Tij betweeneach microphone pair from the microphone recordings. With-out loss of generality, we consider only two microphones iand j.

In [2], a GCC-based framework is proposed to estimatethe extreme (maximum and minimum) TDOAs of the sourcesaround the microphone array, which are further utilized to es-timate the time offset between two asynchronous recordings.It is assumed in [2] that, among the N sources, there are al-ways two sources locating at the end-fire positions with re-spect to the microphone pair. The end-fire positions are de-fined as the points on a line that connects the two microphonesexcluding the ones between the two microphones (Fig. 1).Based on the reverse triangle inequality (||rb−mi|| − ||rb−mj || ≤ ||mi −mj ||), the maximum or minimum TDOAsamong the N sources can be identified once the sources arelocated at the end-fire locations, and can be expressed as

τmax =||mi −mj ||

c+ Tij ; τmin = −||mi −mj ||

c+ Tij .

(3)Naturally, the time offset Tij is obtained as

Tij =τmax + τmin

2. (4)

In this way, the goal of time offset estimation becomesthe task of extreme TDOA estimation. In this paper, we usethe same assumption in [2], and show how the maximumand minimum TDOAs can be estimated with the proposedlandmark-based audio fingerprinting algorithm.

Fig. 1. Illustration of the end-fire source locations.

3. AUDIO LANDMARK AND SINGLE-SOURCETDOA

The audio landmarks are usually used for coarsely synchro-nizing two audio recordings [5–7]. However, the extractedlandmark features contain some valuable information aboutthe TDOA information of the sound sources.

The classical audio landmark fingerprinting method op-erates in the time-frequency domain and converts a time-domain signal ai(t) into a sparse, high-dimensional, anddiscrete-time landmark feature set Fi(n) [6]. At first, thetime-domain signal ai(t) is transformed into the short-timeFourier transform (STFT) Ai(n, k), where n is the frameindex and k is the frequency index, which downsamples thetime axis via the STFT hop size R. The onsets of local fre-quency peaks are detected from the STFT and are representedas sparse time-frequency points, f(np, kq), where p and qdenotes the frame and frequency indices of the detected localpeak, respectively. Landmarks are formed by pairing up eachtwo nearby local peaks, represented as yi(n1, k1;n2, k2). Fi-nally, the obtained landmarks associated with the frame n arehashed into a time-indexed feature set represented as Fi(n).

Assume that only the b-th source is active and the timeoffset between two microphones is zero. The STFT of ai(t)and aj(t) can be expressed as{

Ai(n, k) = gibSb(n− nib, k)Aj(n, k) = gjbSb(n− njb, k)

, (5)

where Sb(n, k) is the STFT of sb(t), nib = btib/Rc, andnjb = btjb/Rc, where the operator b·c denotes the integerpart.

By matching landmarks between the two channels [6], thelandmarks corresponding to the same time-frequency peakpairs can be extracted. This is expressed as

yi(n1−nib, k1;n2−nib, k2) = yj(n1−njb, k1;n2−njb, k2),(6)

and consequently

Fi(n) = Fj(n− bτijbRc), (7)

23rd European Signal Processing Conference (EUSIPCO)

1342

Fig. 2. Block diagram of the proposed coarse-to-fine synchronization method.

where Fi(·) and Fj(·) denote the extracted audio fingerprintsof ai(·) and aj(·), respectively, and yi(·) and yj(·) denotetwo matched local peak pairs in the two channels.

LetIij(τ) = I(Fi(n),Fj(n− τ)) (8)

be the correlation function denoting the matching score be-tween Fi(n) and a time-shifted version Fj(n−τ). The TDOAof the b-th source is obtained by matching the audio land-marks, i.e.

τijb = argmaxτ{Iij(τ)} ·R. (9)

The correlation function (matching score) Iij(·) is calculatedfrom the number of matched landmarks between two chan-nels, with more details provided in [6]. The TDOA is detectedas the peak of the correlation function. This is similar to theGCC method, but here the correlation function is calculatedby matching the audio landmarks.

4. TIME OFFSET ESTIMATION

4.1. Extreme TDOA estimation

To estimate the maximum and minimum TDOAs in a multi-source environment, we employ a fine landmark strategy. Asshown in (7), the resolution of the landmark-based TDOA es-timation is confined by the STFT hop size R. An improvedresolution can be achieved by reducing the hop size. A hopsize as small as R = 1 can be used so that a sample-wise res-olution can be achieved. In this way, the TDOAs of differentsources can be distinguished from each other in the correla-tion function Iij(τ), appearing as different peaks.

We apply a simple peak detector to Iij(τ), obtaining a setof P peaks expressed as T = {τ1, · · · , τP }. The maximumand minimum TDOAs can be estimated from T as

τmin = min(T) and τmax = max(T). (10)

4.2. Coarse-to-fine synchronization

The fine landmark strategy requires a small time-frequencyanalysis hop size. The computation of landmark feature ex-traction and matching would become extremely intensivewhen a large time offset exists between two channels and a

large searching area of τ is required to calculate Iij(τ). Toimprove the computation efficiency, we propose a coarse-to-fine synchronization scheme (Fig. 2).

In the coarse synchronization stage, the audio landmarksare extracted from the STFT spectra of ai(t) and aj(t), witha large hop size R=R1. After landmark matching, the coarsetime offset is estimated from the correlation function I1(τ) as

T1 = argmaxτ{I1(τ)} ·R1. (11)

The channel aj(t) is time shifted with T1 to coarsely align itwith ai(t). This is expressed as

aj(t) = aj(t− T1). (12)

In the fine synchronization stage, the audio landmarks areextracted from the STFT spectra of ai(t) and aj(t), with asmall hop sizeR=R2=1. After landmark matching, the max-imum and minimum TDOAs can be estimated from I2(τ),using (10). The precise time offset is estimated as

T2 =τmax + τmin

2. (13)

Finally, aj(t) is time shifted with T2, obtaining aj(t).The coarse synchronization stage is suitable for record-

ings with large time offsets, while the fine synchronizationstage can improve the synchronization accuracy. Combiningthe two stages, the time offset between two recordings is

Tij = T1 + T2. (14)

5. EXPERIMENTAL RESULTS

Two experiments are conducted to evaluate the performanceof the proposed method. In the first experiment, the audiofeatures extracted by three popular audio fingerprinting al-gorithms (audio landmark (AL) 1 [6], onset [11], PRH [12])are compared to show the unique TDOA detection ability ofthe AL algorithm. In the second experiment, we comparewith real recorded data the synchronization performance ofthe mentioned three audio fingerprinting algorithms, and also

1The AL algorithm is implemented using the code from [14].

23rd European Signal Processing Conference (EUSIPCO)

1343

(a) (b)

(c) (d)

(e) (f)

Fig. 3. Feature analysis of three audio fingerprints on TDOAestimation. The left column is the illustration of the featuresand the right column is the corresponding correlation resultsfor (a)-(b) audio landmark, (c)-(d) onset, (e)-(f) PRH.

compare the synchronization performance of the proposedmethod and the GCC-based method [2].

The GCC-based synchronization method [2] also calcu-lates the extreme TDOAs for the estimation of the time off-set. When implementing this algorithm, we use the AL al-gorithm as a preprocessing step to coarsely synchronize thedata. This preprocessing step is the same as the coarse syn-chronization stage of the proposed method. The absolute er-ror ε = |Tij− Tij | is used as the objective measure for perfor-mance comparison, where Tij is the ground-truth time offset.The time offset between two sound tracks is uniformly ran-domly selected in the range [−1, 1]s.

In the first experiment, we used simulated sound tracksrecoded by a microphone pair placed 5m apart from eachother. Three speech sources are talking concurrently, withtwo placed at two end-fire locations and one placed in themiddle between two microphones. We apply AL, onset, andPRH to perform fingerprint extraction and matching on thetwo channels. The same hop size of 0.001s is used for thethree algorithms. Fig. 3 shows the audio feature extractionand matching results by the three algorithms, with left col-umn depicting the extracted features, the right column de-

(a) (b)

(c)

(d)

Fig. 4. Performance comparison with real data. (a) Publicsquare where the data is collected. (b) Source and micro-phone locations. (c) Performance of the 3 audio fingerprint-ing methods. (d) Performance of the proposed method andthe AL+GCC method. The results are represented by medianvalues of 10 independent realizations. The error bars are thefirst and third quartiles.

picting the matching results, and each row representing onealgorithm. As can be seen from the 2nd-3rd rows, both on-set and PRH extract a large amount of audio features, andobtain one peak in the correlation functions. In contrast, asshown in the first row, the features extracted by AL consist ofsparse peak pairs. Since the matched peak pair comes fromthe same source, multiple peaks can be observed in the corre-lation function, with each peak denoting one source. In Fig.3(b), three peaks are clearly observed, which is equal to thenumber of sources. With this experiment, the unique TDOAestimation ability of audio landmarks is demonstrated.

In the second experiment, we recorded sound tracks us-ing four Samsung Galaxy III smartphones at 8kHz samplingin a 12m × 12m public square. The square and locations ofthe smartphones are shown in Fig. 4 (a) and (b), respectively.Twelve source positions were set according to the end-fire lo-cations of each microphone pair in Fig. 4(b) and two sources(1 male and 1 female speech) about 20s were played simul-taneously by a loudspeaker in each recording. We selectedfour microphone pairs for the comparison with microphone

23rd European Signal Processing Conference (EUSIPCO)

1344

distances d01 = 5.31m, d02 = 5.60m, d03 = 3.55m andd12 = 1.47m.

As seen in Fig. 4(c), the three algorithms generallyachieve bigger estimation errors for larger microphone dis-tances. This is because they only detect the highest peak inthe correlation function to estimate the time offset. Whenthe distance between two microphones increases, the highestpeak in the correlation might deviate from the true value. Forexample, the deviation can be as much as 0.017s when themicrophone distance is 6m. Among the three algorithms,PRH performs best among the three audio fingerprint algo-rithms, while onset is the least robust one. PRH combinesfeatures in neighbouring frames [12], and thus is more robustto TDOA deviation. Onset only uses the features extracted inindividual frames, and thus is least robust. AL uses peak pairsin neighbouring frames, and its performance is in the middleof the other two. Overall, the estimation errors of the threealgorithms range between several and tens milliseconds.

As seen in Fig. 4(d), both the proposed method and theAL+GCC method can achieve estimation errors less than0.001s and the accuracy is ten times higher than the conven-tional audio fingerprinting methods. This is because they bothexploit the maximum and minimum TDOAs to estimate thetime offset instead of using the highest peak of the correlationfunction. Both algorithms achieve higher errors when themicrophone distance is large. This is because the extractedfeatures become weak when energies of the sound receivedby the microphones decrease with the microphone distance,leading to deviated estimations of the extreme TDOAs. Theproposed method performs slightly better than the AL+GCCmethod.

6. CONCLUSIONS

In this paper, landmark-based fingerprinting has been refinedto detect the TDOAs of the sources in a multi-source environ-ment. Comparison among three fingerprint features demon-strates that only the landmark features can be fined-tuned forTDOA estimation. By estimating the maximum and mini-mum TDOAs, the proposed method can estimate the time off-set between two microphone recordings efficiently. A com-plete coarse-to-fine synchronization framework is further pro-posed to deal with recordings with large time offsets. Whenend-fire sources exist, the proposed method can achieve accu-racy comparable to the recent fine synchronization based onGCC using sound tracks recorded in a real environment, andcan significantly improve the accuracy of the conventionallandmark method.

One assumption of the proposed method is the require-ment of a sufficient number of sound sources around the ar-ray (i.e. end-fire sources). Further work is required to finelysynchronize recordings without the need of end-fire sources.

REFERENCES

[1] A. Bertrand, “Applications and trends in wirelessacoustic sensor networks: a signal processing perspec-tive,” in Proc. SCVT, 2011, pp. 1 – 6.

[2] P. Pertila, M.S. Hamalainen, and M. Mieskolainen,“Passive temporal offset estimation of multichannelrecordings of an ad-hoc microphone array,” IEEETrans. Audio, Speech, Language Process., vol. 21, no.11, pp. 2393–2402, Nov. 2013.

[3] N. Ono, H. Kohno, N. Ito, and S. Sagayama, “Blindalignment of asynchronously recorded signals for dis-tributed microphone array,” in Proc. WASPAA, 2009,pp. 161–164.

[4] A.B. Nielsen and L.K. Hansen, “Synchronization andcomparison of lifelog audio recordings,” in Proc.MLSP, 2008, pp. 474–479.

[5] N.Q.K. Duong, C. Howson, and Y. Legallais, “Fast sec-ond screen TV synchronization combining audio finger-print technique and generalized cross correlation,” inProc. ICCE, 2012, pp. 241 – 244.

[6] A.L. Wang, “An industrial-strength audio search algo-rithm,” in Proc. ISMIR, 2003, pp. 7–13.

[7] N.J. Bryan, P. Smaragdis, and G.J. Mysore, “Cluster-ing and synchronizing multi-camera video via landmarkcross-correlation,” in Proc. ICASSP, 2012, pp. 2389–2392.

[8] N.Q.K. Duong and F. Thudor, “Movie synchronizationby audio landmark matching,” in Proc. ICASSP, 2013,pp. 3632–3636.

[9] C.H. Knapp and G.C. Carter, “Generalized correlationmethod estimation of time delay,” IEEE Trans. Acoust.,Speech, Signal Process., vol. 24, no. 4, pp. 320327,Aug. 1976.

[10] P. Shrestha, M. Barbieri, H. Weda, and D. Sekulovski,“Synchronization of multiple camera videos usingaudio-visual features,” IEEE Trans. Multimedia, vol.12, no. 1, pp. 79–92, Jan. 2010.

[11] J. E. Schrader, “Detecting and interpreting musical noteonsets in polyphonic music,” M.S. thesis, EindhovenUniversity of Technology, Netherlands, 2003.

[12] J. Haitsma and T. Kalker, “A highly robust audio finger-printing system,” in Proc. ISMIR, 2002, pp. 107–115.

[13] J.S. Seo, “An asymmetric matching method for a ro-bust binary audio fingerprinting,” IEEE Signal Process.Lett., vol. 21, no. 7, pp. 844–847, Jul. 2014.

[14] D.P.W. Ellis, Robust Landmark-Based AudioFingerprinting, web resource, 2009, available:http://labrosa.ee.columbia.edu/matlab/fingerprint/.

23rd European Signal Processing Conference (EUSIPCO)

1345