angry birds project -- vision component · 2014. 2. 18. · the angry birds project an agent that...
TRANSCRIPT
![Page 1: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/1.jpg)
Angry Birds Project -- vision component
By: Andrew WangSupervisor: Dr. Stephen Gould
COMP4560 Advanced Computing Project
![Page 2: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/2.jpg)
Outline
● Background● Existing system● Proposed method
– Connected-component extraction● Pixel labelling● Edge detection
– Shape deduction
● Results● Conclusion & future work
![Page 3: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/3.jpg)
Background
Fig 1. Screen-shot from Chrome version of the game
![Page 4: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/4.jpg)
The Angry Birds project
● An agent that automatically plays the game– Input: screen-shot of the game
– Output: sequence of shots which maximise the score
● Component breakdown– Computer vision
– Knowledge representation & reasoning
– Planning
– Machine learning
![Page 5: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/5.jpg)
Existing System
● Compress each pixel to 9 bits● Extract connected-component by matching each pixel against
possible colours for each class● Bounding box determined by minimum/maximum x, y values● Time: < 0.2 s● Issues:
– Actual shape and orientation not detected
– Stacked objects
– Inaccurate bounding box
![Page 6: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/6.jpg)
Existing System
Fig 2. Example where shape and orientation of objects are not detected
![Page 7: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/7.jpg)
Existing System
Fig 3. Stacked ice blocks treated as a single object
![Page 8: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/8.jpg)
Existing System
Fig 4. Inaccurate bounding boxes
![Page 9: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/9.jpg)
Existing System
Fig 4. Inaccurate bounding boxes
![Page 10: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/10.jpg)
Existing System
Fig 5. Failure to describe the objects accurately making it almost impossible to reason
![Page 11: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/11.jpg)
Proposed Method
● Assign a class label to every pixel in the image (e.g. red bird, wood, ice etc.)
● Determine object boundaries using an edge detector
● Extract objects as connected-components of the same class
● Deduct shape of the component from its boundary points
![Page 12: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/12.jpg)
Pixel Labelling● Manually labelled training data
– Set of points in RGB space and their corresponding class
– e.g. {160, 0, 0, RED_BIRD}
● Classify using Nearest-Neighbour rule– Each point is assigned the same label as its closest neighbour in training data
● Issue: NN is slow● Solution:
– Compress each pixel to 15-bits and calculate the class label for all colours during initialisation phase
– Label each pixel via table lookup
![Page 13: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/13.jpg)
Pixel Labelling
Fig 6. left: original image; right: labelled image, 1 colour for each class label
![Page 14: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/14.jpg)
Edge Detection
● Existing approach: Canny edge detector
![Page 15: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/15.jpg)
Edge Detection
● Issue with intensity based● Colour space: {hue, saturation, intensity}● Define similarity measure as weighted Euclidean
distance– D (a⃗ , b⃗) = √wh(ahue - bhue)2 + w s (asat - bsat)
2
![Page 16: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/16.jpg)
Edge Detection
● Consider 3x3 neighbourhood of a point
● Edge strength in each direction i {1, 2, 3, 4}∈–
● Non-maximum suppression and double threshold applied for each direction
● Combine edges in all directions
2 1 8
3 0 7
4 5 6
G i = D( v⃗0, v⃗ i) + D( v⃗0, v⃗ i+4)
![Page 17: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/17.jpg)
Edge Detection
Fig 7. left: original image; right: edge image
![Page 18: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/18.jpg)
Connected Components
● BFS until edge / different class label encountered
Fig 8. left: original image; right: all connected components
![Page 19: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/19.jpg)
Shape deduction
● Required shapes: rectangles, circles and polygons
● Draw a bounding box around the component for angles from 0 to 90 degrees
– Rotate boundary points by θ
– Size of bounding box determined by min/max x, y coordinates
● Reduce computation complexity
– Only use points where edge orientation changes (corners)
(x, y)
x' = x cos θ - y sin θ
(x', y')
y' = x sin θ + y cos θ
![Page 20: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/20.jpg)
Shape deduction
● Record the maximum and minimum area● Circles: minArea * t1 > maxArea● Polygon: minArea > component size * t2● Otherwise shape is given by the minimum area
rectangle
![Page 21: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/21.jpg)
Results
Fig 9. Vision output on level 19
![Page 22: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/22.jpg)
Results
Fig 10. Vision output on level 25
![Page 23: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/23.jpg)
Conclusion & Future work
● Improvement over the existing system– Time: < 0.2 seconds → < 0.4 seconds
● Generally reliable for known objects● Improve result by reasoning● Extend to detect unknown objects
![Page 24: Angry Birds Project -- vision component · 2014. 2. 18. · The Angry Birds project An agent that automatically plays the game – Input: screen-shot of the game – Output: sequence](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd9a2a40d0d467cb95c5eec/html5/thumbnails/24.jpg)
Questions