hacking for salone: drone races - di saverio; lippolis - codemotion milan 2016
TRANSCRIPT
HACKING FOR SALONE: DRONE RACESEMANUELE / SIMONE
EMANUELE DI SAVERIOPRINCIPAL DESIGN TECHNOLOGIST
Emanuele started writing software for mobile in the pre-iPhone era. Didn’t stop since then.
WHO’S ON THE STAGE
SIMONE LIPPOLISSENIOR DESIGN TECHNOLOGIST
Simone started writing software when COBOL was a hype word. Now his interests span from web-bots to data-visualization to embedded applications and IOT.
EVERYTHING STARTED ON A BLACK FRIDAY
WE NEEDED
MORE ENTERTAINING
TO MAKE IT
A CLASSIC GAME CAME IN OUR HELP
7
TODO LIST
TALK TO EACH OTHER
SENSE EACH OTHER
HARDER, BETTER, FASTER, STRONGER
TALK TO EACH OTHER
COMPONENTSARCHITECTURE
• Drones on wheel (JumpingSumo by Parrot)
• An open-security 5 Ghz Wi-Fi AP
• Android Smartphones
• A WebSocket Server
ControllerMobile App
JumpingSumolinux+busybox
Game ServerNodeJS
ControllerMobile App
JumpingSumolinux+busybox
ControllerMobile App
JumpingSumolinux+busybox
WebSocket
. .
The Parrot robots work by creating an 802.11ac themselves - smartphones can connect to that and control though applications built with the Parrot SDK.This disables internet access for the smartphones - we needed to “hack” it.
HACKING CONNECTION
$ telnet 192.168.1.1
accesses the BusyBox environment to edit config files /etc/init.d/init_manager and dragon.conf
Drones will connect to your open WiFi at reboot, where they will find Android phones.
The server doesn’t need to provide much functionality, but rather route events in a soft-realtime fashion to the players, with litte computation.• WebSockets help avoid TCP RTT• nodeJS because it can scale to gazillions of
eventsWe have 4 players at at time :)
A BROKER SERVER
SENSEEACH OTHER
WHERE IS MY DRONE?
BLE BEACON/FINGERPRINTING IR MARKER
TANGO
LET DRONES “SEE”
The hardware platform features a front camera for still photos and streaming.Specs not great - we’ll need to design around.
Camera Spec
640x480
15 FPS
Fixed focus
Motion JPEG
COLOR BLOBS
Using OpenCV as resource for highly-optimized algorithm library, we tried to detect color blobs by • thresholding• finding contours• centroidsand applying some heuristics
FIDUCIAL MARKERS
We also res
B&W Square markers are much more robust and can apply adequate Hamming-distance encodings.As an added benefit, they are more robust with respect to luminosity.
IMAGE PROCESSING PIPELINE
PARROT SDK
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
Each of the incoming frames from the drones has to go through heavy processing to detect a marker.
66 ms
3D MODELROTOTRASLATE COORDINATES
HARDER, BETTER, FASTER, STRONGER
OUT OF THE JAVALAND
First problem relates to the sheer computation power needed to conver binary image formats - Java implementation is just too slow.We had to port algorithm implementation to RenderScript to have a fully native implementation that performs acceptably.
JVM RS RS_MULTI
AVG 283,60 1,73 3,24
STDDEV 38,17 2,49 1,52
MEDIAN 262,11 0,86 2,78
MAX 447,57 10,04 15,18
OUT OF THE GC LAND
The main source of delays in real-time applications like games is the Garbage Collector - we’re talking half second freeze on recent hardware.This means we cannot allocate memory in the game loop - Android memory profiler tools will help you discover and eliminate:• Bitmap.create()• new Array[] • Object creations / Autoboxing
FULL CONTROL FLOW
PARROT SDK(CAMERA)
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
3D MODELROTOTRASLATE COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING(GYRO)
BROKER SERVER
OTHERPLAYERSACTIONS
FULL CONTROL FLOW
PARROT SDK
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
3D MODELROTOTRASLATE COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING
BROKER SERVER
66 ms
33 ms
~ 15 s16 ms
~ 50 ms
STEERING(GYRO)
OTHERPLAYERSACTIONS
Handling all the different events coming in at different rates would a very complex task to handle in a fully procedural way in a standard game loop:
REACTIVE STREAMS FTW
while (true) {
acquire_image();
process_markers();
get_steer_input();
get_other_players_input();
apply_cmds_to_drone();
apply_game_events();
update_ui();
}
.
.
.
.
REACTIVE STREAMS FTW
Rx provides common semanthics to express and manipulate a collection of asynchronous events
RXJAVA
http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android
(2014)
Plenty of good introductions to RxJava in the InterWebs
Exercises
https://github.com/jhusain/learnrxjava
KOTLIN MAY BE THE ONE
We may have found a winner.
Kotlin is a deep language with many modern features, including Functional approaches.
The three best features for Kotlin on Android:
1. Overhead is at compile time, Runtime Overhead is minimal
2. Closures and SAM
3. Java Interoperability is a breeze
One things I’m still undecided if is good or bad: Nullability
FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI
REFERENCES
https://github.com/frog/a-kart
http://designmind.frogdesign.com/2016/07/hacking-a-multi-game-user-drone-race/
SIMONE LIPPOLIS
EMANUELE DI SAVERIO
@simonelippolis
@hazam
GAME & UI DESIGN, SERVER DEVELOPMENT
MOBILE DEVELOPMENT, COMPUTER VISION
FEDERICO BRESSAN
ELENA MARENGONIRACETRACK, DRONE SKINDESIGN
RACETRACK, DRONE SKINDESIGN
THANKS!
