ee3682011spring_guestlecture_part2

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 1

Mobile image processing – Part 2

Mobile application development on AndroidProject 1: “Hello Image”, reading/writing/modifying imagesProject 2: “Viewfinder EE368”, processing viewfinder framesProject 3: “CVCamera”, utilizing OpenCV functionsMobile application examples


Recognizing video at a glance

(1) User snaps a photo of screen.

(2) Our system identifies video andframe within the video.

(3) User resumes video on the phone.


http://www.stanford.edu/~dmchen/mvs.html

Recognizing video at a glance


Mobile device versus personal computer

Feature Motorola DROID Typical PCScreen size 4.6 in × 2.4 in 25 in × 16 in

Processor speed 550 MHz 3.0 GHzRAM 256 MB 4 GB

Permanent storage 133 MB internal8 GB external flash 500 GB disk

Internet access Typically 3G orWiFi at hotspots WiFi or wired Ethernet

Telephony Core feature Supplementary feature

Camera 5 MP camera embedded External webcam


“Viewfinder EE368” project

Goals of this projectLearn how to access frames from the viewfinderLearn how to modify camera parametersLearn how to augment the viewfinder frames

Step-by-step instructions available in Tutorial #1:http://ee368.stanford.edu/Android


Android manifest file

…

…<application android:icon="@drawable/icon"

android:label="@string/app_name"><activity android:name=".ViewfinderEE368"

android:label="@string/app_name"android:screenOrientation="landscape“android:theme="@android:style/Theme.NoTitleBar">

<intent-filter><action android:name="android.intent.action.MAIN" /><category android:name="android.intent.category.LAUNCHER" />

</intent-filter></activity></application>

<uses-sdk android:minSdkVersion=“7" /><uses-permission android:name="android.permission.CAMERA"/>

Set the proper hardware permissions

Landscape mode without title bar

Main Activity


Class hierarchy for “Viewfinder EE368”

Viewfinder EE368

(Activity)

Preview(View)

Draw on Top

(View)

Manage two views and manage program open/close

Handle incoming viewfinder frames and adjust camera parameters

Augment viewfinder frames with a new layer of information on top


Viewfinder class: full screen mode

Viewfinder EE368

Icon and title bars visible Icon and title bars hidden

public void onCreate(Bundle savedInstanceState) {...getWindow().setFlags(

WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN

);requestWindowFeature(Window.FEATURE_NO_TITLE);...

}



Viewfinder EE368

(Activity)

Preview(View)

Draw on Top

(View)

Manage two views and supervise program open/close




Timeline of events on the mobile device

Time

New frame New frame

Capture

Augment

Preview

DrawOnTop

Capture

Augment


Preview class: frames go down two paths

Preview(View)

Viewfinder frames

…

Display on phone screen

Forward to custom callback function

myCamera.setPreviewCallback(new PreviewCallback() {public void onPreviewFrame(byte[] data, Camera camera){ Pass data to DrawOnTop class }

});

Data in YCbCr 4:2:0 formatw

h

w/2 w/2

h/2

h/2

YCb Cr


Preview class: set camera parameters

Preview(View)

Camera.Parameters parameters = myCamera.getParameters();parameters.setPreviewSize(640, 480);parameters.setPreviewFrameRate(15); parameters.setSceneMode(Camera.Parameters.SCENE_MODE_NIGHT); parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);myCamera.setParameters(parameters);



Viewfinder EE368

(Activity)

Preview(View)

Draw on Top

(View)

Manage two views and manage program open/close




DrawOnTop class: YCbCr to RGB conversion

84.27625641.516

25608.298

58.13525612.208

25629.100

25608.298

921.22225658.408

25608.298

−⋅

+⋅

=

+⋅

−⋅

−⋅

=

−⋅

+⋅

=

CbYB

CrCbYG

CrYR

w

h

w/2 w/2

h/2

h/2

YCb Cr

w

h

RGB


DrawOnTop class: create some paint brushes

myPaintRed = new Paint();myPaintRed.setStyle(Paint.Style.FILL);myPaintRed.setColor(Color.RED);myPaintRed.setTextSize(25);

myPaintGreen = new Paint();myPaintGreen.setStyle(Paint.Style.FILL);myPaintGreen.setColor(Color.GREEN);myPaintGreen.setTextSize(25);

myPaintBlue = new Paint();myPaintBlue.setStyle(Paint.Style.FILL);myPaintBlue.setColor(Color.BLUE);myPaintBlue.setTextSize(25);


DrawOnTop class: draw histogram bars

Bin 0 Bin 1 Bin 2 Bin 3 Bin 4

// Draw red rectangleRect rect = new Rect(left x, top y,

right x, bottom y);myCanvas.drawRect(rect, myPaintRed);


Viewfinder EE368 demo

http://ee368.stanford.edu/Android


Real-time debugging using DDMS (1)


Real-time debugging using DDMS (2)

File system on device

Messages on the device

0123456789ABC

Device


Taking a screenshot of the phone (1)

0123456789ABC


Taking a screenshot of the phone (2)


Class Project: Mobile Image Completion

H. Wang and A. Lin, Spring 2010http://ee368.stanford.edu/Project_10


Class Project: Mobile HDR Imaging

J. Mathe and T. Wong, Spring 2010http://ee368.stanford.edu/Project_10


Mobile image processing – Part 2

Mobile application development on AndroidProject 1: “Hello Image”, reading/writing/modifying imagesProject 2: “Viewfinder EE368”, processing viewfinder framesProject 3: “CVCamera”, utilizing OpenCV functionsMobile application examples

ee3682011spring_guestlecture_part2

Documents

ee368 mobile image processing

david chen

viewfinder class

viewfinder ee368 project

topbernd girod

eduandroidbernd girod

glancebernd girod

mobile application development