visualization of hummingbird flight - stanford university · visualization of hummingbird flight...

4
Visualization of Hummingbird Flight Rivers Ingersoll School of Mechanical Engineering Stanford University Stanford, California 94305 Email: [email protected] Abstract—Hummingbirds are one of nature’s most fascinating creatures. They have been shown to flap their wings over 40 times a second [1] and quickly perform aerial maneuvers while drinking nectar from flowers. Their swiftness is far too fast for the human eye to capture in real time, but with the use of high-speed video cameras, we can slow down and view their movement on a different scale. I set out to take high-speed color videos of hummingbirds in their natural habitat and use image processing techniques to create a visually appealing time lapse of their movement. I. I NTRODUCTION To track the birds from one frame to the next, it is important to first separate the birds from the background. There are multiple methods to separate backgrounds from foregrounds of images. Each method has its own benefits and drawbacks. The most basic method is using a threshold value to separate the bright and dark areas of an image. This is usually performed using Otsu’s method [2] which searches for a threshold value that minimizes the within class variance. Another method commonly used especially in video tracking is subtracting the median image of all frames from each individual frame. This results in only seeing the objects that have moved in the video. When attempting to track the birds using the median background subtraction method, it was found that the slight camera movement would cause the frame to become unaligned and the method failed. Even with the addition of aligning each frame, the non-uniform movement of other objects in the image such as flowers and trees make it hard to detect the bird as the only moving object. With the advice from the ECE368 teaching staff, it was suggested that I implement the GrabCut [3] algorithm for high quality segmentation of foreground and background which is based on the GraphCut algorithm [4], [5], [6], [7]. A multi- label optimization library (gco-v3.0) that utilized GraphCut was found to interface nicely with the high-speed hummingbird images. II. I MPLEMENTATION Filming of the hummingbirds was performed at the Stan- ford Arizona Cactus Garden on both an overcast as well as a sunny day. Video was recorded at 2000 FPS using a Phantom Miro LC310 color video camera. The raw .cine videos files were imported to Phantom Camera Control Application (PCC) where they were clipped and exported as .avi files. A. Graphical User Interface To mask the backend processing and allow a user unfamil- iar with the algorithm to process their high-speed video files, a graphical user interface (Figure 1) was created in MATLAB [8]. This GUI allows the user to select the high-speed video file of their choice, and set desired input parameters that will affect how the processed video will look. The computational progress during processing is displayed in the main MATLAB command window. The user process is as follows: Place all videos in the ‘videos’ folder in the working directory. Enter ‘main GUI’ in the MATLAB command window to run program. Highlight a video from the listbox. Select load video. Move the slider through the range to find the start frame. Select ‘Set Current Frame as Start Frame’. Move the slider through the range to find the end frame. Select ‘Set Current Frame as End Frame’. Use the first drop down menu to specify how many intermediate frames to process. Use the second drop down menu to specify the number of ghost images to show. Use the third drop down menu to specify how often the ghost images appear. Use the fourth drop down menu to specify how dark to make the ghost images. Check the first box to have an outline drawn around each tracked bird in each frame. Check the second box to have the eccentricity plotted below for each frame. Check the third box to have the centroid plotted on each tracked bird in each frame. Press the ‘Select with Rectangle’ button to select the bounding box for the bird in each frame. Press the ‘Select with Rectangle’ button again to track more birds if desired. Press the ‘Process’ button to run the algorithm over the defined frames. Press the ‘Save Video’ button to export the processed clip as an .avi file.

Upload: others

Post on 21-Mar-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Visualization of Hummingbird Flight - Stanford University · Visualization of Hummingbird Flight Rivers Ingersoll School of Mechanical Engineering Stanford University Stanford, California

Visualization of Hummingbird Flight

Rivers IngersollSchool of Mechanical Engineering

Stanford UniversityStanford, California 94305Email: [email protected]

Abstract—Hummingbirds are one of nature’s most fascinatingcreatures. They have been shown to flap their wings over 40times a second [1] and quickly perform aerial maneuvers whiledrinking nectar from flowers. Their swiftness is far too fast forthe human eye to capture in real time, but with the use ofhigh-speed video cameras, we can slow down and view theirmovement on a different scale. I set out to take high-speed colorvideos of hummingbirds in their natural habitat and use imageprocessing techniques to create a visually appealing time lapse oftheir movement.

I. INTRODUCTION

To track the birds from one frame to the next, it is importantto first separate the birds from the background. There aremultiple methods to separate backgrounds from foregrounds ofimages. Each method has its own benefits and drawbacks. Themost basic method is using a threshold value to separate thebright and dark areas of an image. This is usually performedusing Otsu’s method [2] which searches for a threshold valuethat minimizes the within class variance. Another methodcommonly used especially in video tracking is subtractingthe median image of all frames from each individual frame.This results in only seeing the objects that have moved in thevideo. When attempting to track the birds using the medianbackground subtraction method, it was found that the slightcamera movement would cause the frame to become unalignedand the method failed. Even with the addition of aligning eachframe, the non-uniform movement of other objects in the imagesuch as flowers and trees make it hard to detect the bird as theonly moving object.

With the advice from the ECE368 teaching staff, it wassuggested that I implement the GrabCut [3] algorithm for highquality segmentation of foreground and background which isbased on the GraphCut algorithm [4], [5], [6], [7]. A multi-label optimization library (gco-v3.0) that utilized GraphCutwas found to interface nicely with the high-speed hummingbirdimages.

II. IMPLEMENTATION

Filming of the hummingbirds was performed at the Stan-ford Arizona Cactus Garden on both an overcast as well as asunny day. Video was recorded at 2000 FPS using a PhantomMiro LC310 color video camera. The raw .cine videos fileswere imported to Phantom Camera Control Application (PCC)where they were clipped and exported as .avi files.

A. Graphical User Interface

To mask the backend processing and allow a user unfamil-iar with the algorithm to process their high-speed video files,

a graphical user interface (Figure 1) was created in MATLAB[8]. This GUI allows the user to select the high-speed videofile of their choice, and set desired input parameters that willaffect how the processed video will look. The computationalprogress during processing is displayed in the main MATLABcommand window. The user process is as follows:

• Place all videos in the ‘videos’ folder in the workingdirectory.

• Enter ‘main GUI’ in the MATLAB command windowto run program.

• Highlight a video from the listbox.

• Select load video.

• Move the slider through the range to find the startframe.

• Select ‘Set Current Frame as Start Frame’.

• Move the slider through the range to find the endframe.

• Select ‘Set Current Frame as End Frame’.

• Use the first drop down menu to specify how manyintermediate frames to process.

• Use the second drop down menu to specify the numberof ghost images to show.

• Use the third drop down menu to specify how oftenthe ghost images appear.

• Use the fourth drop down menu to specify how darkto make the ghost images.

• Check the first box to have an outline drawn aroundeach tracked bird in each frame.

• Check the second box to have the eccentricity plottedbelow for each frame.

• Check the third box to have the centroid plotted oneach tracked bird in each frame.

• Press the ‘Select with Rectangle’ button to select thebounding box for the bird in each frame.

• Press the ‘Select with Rectangle’ button again to trackmore birds if desired.

• Press the ‘Process’ button to run the algorithm overthe defined frames.

• Press the ‘Save Video’ button to export the processedclip as an .avi file.

Page 2: Visualization of Hummingbird Flight - Stanford University · Visualization of Hummingbird Flight Rivers Ingersoll School of Mechanical Engineering Stanford University Stanford, California

Fig. 1: MATLAB Graphical User Interface

B. Algorithm

1) Segmenting each object: Every frame of the video isread into a matrix containing the RGB data for each frame inthe double format. The first frame is shown in Figure 2 below.

Fig. 2: User selected area in first frame

The defined searching window for each bird is used to runthe GraphCut algorithm over a smaller window only containingeach bird as shown in Figure 3.

Fig. 3: Search windows for second frame

The GraphCut algorithm is specified to only separate theimage into two regions. The GraphCut algorithm outputs abinarized matrix of the smaller window with the foregroundand background segmented as shown in Figure 4.

Fig. 4: GraphCut output

If a majority of the perimeter points in the matrix havea non-zero value, then the matrix is inverted signifying thatthe background was segmented as the object. The regionpropsalgorithm is then run over this binary image to separate thebirds from the other features detected such as trees or flowers.If more than one object is detected then the centroids of eachobject is compared to the centroid of the bird in the previousframe as shown in Figure 5.

Fig. 5: Measuring distance to centroids

Page 3: Visualization of Hummingbird Flight - Stanford University · Visualization of Hummingbird Flight Rivers Ingersoll School of Mechanical Engineering Stanford University Stanford, California

The closest centroid is assumed to be that of the bird. Thenext step is to define the new hummingbird searching windowfor the next frame. This is done by tripling the size of theclosest bounding box in the current frame. The binary matrixof the bird silhouette is then set into the full frame as shownin Figure 6 below.

Fig. 6: Binary mask of bird silhouette

2) Creating ghost effect: To create the desired ghostingeffect, the following parameters are given by the user:

• Maximum number of ghosts.

• Transparency of darkest ghost image.

• Number of frames between each ghost image.

For every frame the binary bird mask matrix is multiplied bya transparency value and subtracted from each channel of thecolor image. This is done for every ghost image desired ineach frame. The transparency of the ghost image can fade asthe video progresses.

3) Drawing outline: If the draw outline checkbox is se-lected, then for each frame, a Laplace of Gaussian filter isapplied to the image silhouette. This resulting outline is thendrawn around the bird in each frame as shown in Figure 7.

Fig. 7: Binary mask of bird outline

4) Centroid plotting: If the plot centroid checkbox isselected, the centroids of each frame as well as the previousframes are shown in the main video window after processing.This is helpful to see the path of the flight as well as howsmooth the tracking performed as shown in Figure 8.

Fig. 8: Centroids and outlines shown

5) Eccentricity plotting: If the plot eccentricity checkboxis selected then for each regionprops call, the eccentricity ofthe bird is calculated. Ravnan [9] showed how the eccentricityplotted over time can give insight into the phase of the flightwhich could be helpful for future exploration on the subject.The eccentricity is plotted in Figure 9 for the two birds inFigure 8 showing patterns aligning with wingstroke phase.

Fig. 9: Eccentricity

III. RESULTS

The algorithm was run over twenty different videos ofhummingbirds. In videos where the tracked hummingbirdoverlapped with other objects (Figure 10) or had non uniformbackgrounds (Figure 11), the tracking did not work becausethe GraphCut algorithm was set to only find two regions.

Fig. 10: Object overlap

Fig. 11: Cluttered background

Page 4: Visualization of Hummingbird Flight - Stanford University · Visualization of Hummingbird Flight Rivers Ingersoll School of Mechanical Engineering Stanford University Stanford, California

The best results were seen from the videos taken on anovercast day because the sky was very gray while the birdwas very dark. Screenshot from the clearest four processedvideos are shown in Figure 12.

IV. FURTHER WORK

It was shown that without background clutter the birds canbe successfully segmented from the background. To increasethe robustness of the algorithm it would be beneficial to takeinto account the area of the assumed bird object. If this areastarts to grow rapidly, it is probable that some backgroundclutter is being detected and a different method can be usedselect only the bird. Another possible method to increase therobustness of the algorithm is increased the number of regionsthe GraphCut algorithm detects.

ACKNOWLEDGMENT

Thank you to Professor Bernd Girod, David Chen and MattYu for their support during the course. I would also like tothank Dr. Roland Angst for his guidance for this project andthe LentinkLab for access to the high-speed video camera.Partial inspiration for this project is from Dennis Hlynsky onVimeo [10] [11] who filmed birds flying in the air at standardframe rates and used Adobe After Effects to display the pathsof the birds over time.

REFERENCES

[1] D. L. Altshuler, E. M. Quicazan-Rubio, P. S. Segre, and K. M.Middleton, “Wingbeat kinematics and motor control of yaw turns inanna’s hummingbirds (calypte anna),” The Journal of experimentalbiology, vol. 215, no. 23, pp. 4070–4084, 2012.

[2] N. Otsu, “A threshold selection method from gray-level histograms,”Automatica, vol. 11, no. 285-296, pp. 23–27, 1975.

[3] C. Rother, V. Kolmogorov, A. Blake, and M. Brown, “Grabcut,” March2014. [Online]. Available: http://goo.gl/eDh41h

[4] V. Kolmogorov and R. Zabih, “What energy functions can be minimizedvia graph cuts?” IEEE transactions on Pattern Analysis and MachineIntelligence, vol. 26, no. 2, pp. 147–159, February 2004.

[5] Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.” IEEEtransactions on Pattern Analysis and Machine Intelligence, vol. 26,no. 9, pp. 1124–1137, September 2004.

[6] Y. Boykov, O. Veksler, and R. Zabih, “Efficient approximate energyminimization via graph cuts,” IEEE transactions on Pattern Analysisand Machine Intelligence, vol. 20, no. 12, pp. 1222–1239, November2001.

[7] S. Bagon, “Matlab wrapper for graph cut,” December 2006. [Online].Available: http://www.wisdom.weizmann.ac.il/∼bagon

[8] M. U. Guide, “The mathworks,” Inc., Natick, MA, vol. 5, 1998.[9] E. Ravnan, “Automatic phase detection of a bird wing in flight,”

March 2014. [Online]. Available: http://goo.gl/W9yjC8[10] D. Hlynsky, “Starlings at sunset,” March 2014. [Online]. Available:

http://vimeo.com/78737089[11] D. Hylnsky, “Black vultures,” March 2014. [Online]. Available:

http://vimeo.com/78188069

Fig. 12: Video Screenshots