The THIRD EYECORRELATING IMAGE-SOUND-VIDEO

Gunjan Gupta 10BEC112Romil Shah 10BEC093Sagar Patel 10BEC072

Mentor : Prof. Sachin Gajjar

1

Outline

Motivation

Technological Objective

Present Scenario

Bionics: Concept and Cyborg

Our Innovation

Approach (Pros and Cons)

Applications

Algorithms

References

2

Wah Taj !! 3

Beautiful… 4

Again Nice to See!! 5

Oh, This bus is going to LAL DARWAJA 6

BUT CAN EVERYBODY SEE THIS BEAUTY ?

CAN EVERYONE SAY THE BUS NUMBER BY SEEING IT ?

NO!!ALL PEOPLE DON’T HAVE SAME EYE-

SIGHT or EYE-VISION !

7

WHAT FOR THESE GUYS ?? 8

A project to build a system that converts visual input into audio signals which may lead to a practical product to help the blind or visually impaired

navigate

9

Technological Objectives

The main objective is to obtain a device for the whole Visually Impaired(V.I.) community- totally blind, partially sighted, and people in a situationof progressive loss of vision.

To integrate the developments of image segmentation, imageenhancement, 3D Real World perception, audio representation of spaceinto a unique basic prototype with the purpose of assisting visuallyimpaired people in mobility in outdoor spaces.

To develop new ways of interfacing with V. I. people- to develop anacoustic interface (Ear-plugs for presentation) for the blinds.

To develop a system able to calculate a person’s head orientation in astatic and dynamic situation.

10

Technological Objectives

To integrate the 3D modelling system and the intelligent system for thedetection of risks and obstacles

To integrate in a common platform all the data acquired from theenvironment from different sources- artificial vision system, 3D Real WorldRecognition System.

To finally develop a prototype which can be commercialised

During the project a simple device able to make a direct transformationfrom the acquisition system into an acoustic representation will be createdas a final product

11

Gadgets/Technology for V.I. Person 12

Brainport Vision Device

Electric lollipop that allows blind people to 'see'using their tongue

Converts images captured by a tiny camera intoa series of electrical tingles, which can be felt onthe tongue

Nerves then send these messages to the brain,which turn the tingles back into pictures

Costs around 6000 Pounds ( 5 Lakh INR)

Gadgets/Technology for V.I. Person

Devices that alert you when a pot of liquid begins to boil

Gadgets that make a sound when a cup or container you're pouring liquidinto is nearly full

Knives with an adjustable guide to help you cut slices of even thickness

Tactile watches and alarm clocks

13

Gadgets/Technology for V.I. Person 14

Cyborg / Bionics

An organism “to which exogenous components have been added for the purpose of adapting to new environments”.

Recent meaning of the term bionics refers to merging organism and machine.

This approach results in a hybrid system combining biological and engineering parts, which can also be referred as a cybernetic organism (cyborg).

15

Natural-Artificial Feedback 16

Neil Harbisson: A Real Cyborg17

What is Image Processing?

In electrical engineering and computer science, image processing is any form of signalprocessing for which the input is an image, such as a photograph or video frame; theoutput of image processing may be either an image or, a set of characteristics orparameters related to the image.

Most image-processing techniques involve treating the image as a two-dimensionalsignal and applying standard signal-processing techniques to it.

In other words it is basically the transformation of data from a still or video camerainto either a decision or a new representation.

All such transformations are done for achieving some particular goal. The input datamay be a live video feed, the decision may be that a face has been detected, and a newrepresentation may be conversion of a color image into a gray scale image.

Our Innovation

Locating everyday items using Image-Video Processing.

Detection of potential hazards (location and motion of objects, persons, obstacles).

Face Detection and Tagging.

Text to Speech Conversion.

Color Detection

Detecting and analyzing in-front objects (living and non-living) and perform operations on them.

19

Approach

We are using OpenCV with Panda Board for this application.

Interfacing OpenCV and Panda Board along with webcam to generate real time images.

Coding to find distance of objects or any obstacle and provide output as an audio signal.

Provide with inbuilt images and search for them. Coding to match the images and provide the respective name of object as sound signal.

Advance further to develop for general usage amongst people.

20

Advantage & Disadvantage

Advantage

Detection of known objects if they are lost.

Can read newspaper, books, or rather any reading material.

Can easily detect color and hence can be useful in various scenarios.

Face Detection and Face tagging mode.(He can easily know person and canalso save new face)

Most of the software are freeware, so cheap in cost.

Disadvantage

A fully-assembled device (Panda Board + Webcam) is to be carried for all theoperations.

Bulkier in comparison to Mobile.

21

Hardware/Software

The Panda Board is a low-power, low-cost single-boardcomputer development platform based on the TexasInstruments OMAP4430 system on a chip (SoC). It is a communitysupported development platform.

The Panda Board ES is a newer version based on the OMAP4460 SoC, withthe CPU and GPU running at higher clock rates. Like its predecessor, it is acommunity supported development platform. We are working with PandaBoard ES.

22

Pandaboard

Pandaboard Overview

OMAP4 (Cortex-A9) CPU based open development platform.

OMAP4430 Application processor

1GB low-power DDR2

Display HDMI v1.3 Connector (Type A) to drive HD displays, DVI-D Connector,

3.5" audio in/out and HDMI Audio out

Full size SD/MMC card

Built in 802.11 & Bluetooth v2.1+EDR

On board 10/100 Ethernet

Expansion: 1xUSB OTG, 2xUSB HS host ports, General purpose expansion header

What OS does Pandaboard support?

Debian

Ubuntu

Android

Fedora

Sabayon

Xubuntu

Angstrom

OpenCV

OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications.

The intent behind OpenCV was to provide a platform that a student can readily utilize for developing applications, instead of reinventing the basic functions from scratch.

Why use OpenCV?

It is free.

It is fast.

Good documentation in the form of documentation, tutorials, user groups, forums etc.

Lot of pre-built functions and algorithms to get a head start.

There is an active development on interfaces for other languages like Ruby, Python, Matlab etc.

Pretty fast.

License

OpenCV is released under a BSD license, it is free for both academic and commercial use.

There are only a few algorithms inside it that may have been patented and those are not free.

KINECT

Kinect SENSOR

Photo Albumby Gunjan Gupta

BLOCK DIAGRAM

31

Face Recognition 32

Object Detection

Dynamic (Moving)Stationary

33

Detection by Segmentation34

Segmentation Algorithm

Segmentation

Discontinuity

Point Detection Line Detection Edge Detection

Similarity

Thresholding Region GrowingRegion splitting

and merging

35

Using MATLAB inbuilt function

Using our ownalgorithm, we appliedlinear differentiator. Itworks fast and is moreefficient incomparison toMATLAB INBUILTALGO

36EDGE DETECTION

Application

This project is applied mainly in the commercial field of helping visuallyimpaired people with lower eyesight or completely BLIND

Industrial applications can be devised and enhanced like robots andmachineries

Concept can be used for Autonomous Cars

For Security, Identifying and Tracking

37

Projects Implemented

1. Capturing Image using hand gesture and uploading it directly to Google Drive (Internet needed)

2. Face Detection and Person Identity Detection for running lifetime of prog

3. Optical Character Recognition

4. Cartoonifying Real Time Images

5. Colour Detection and Diff Sound for Diff Colour

6. Ball Detection

7. Head Pose Estimation

8. Augmented Reality based Position Change Tracking

9. Combined Project of OpenGL and OpenCV for Image Operation

10. Number Recognition in real-time

11. Text to Speech Synthesis using C++

12. Kinect with WINDOWS

13. Kinect with Ubuntu

14. Sound Marking

References

A collaborative European project CASBliP, or Cognitive Aid System for Blind People

Andhajan Mandal, Ahmedabad, Gujarat, India

Electric lollipop that allows blind people to 'see' using their tongue-Brainport Vision Device

OMAP4460 Pandaboard ES System Reference Manual

Neil Harbisson-http://www.ted.com/talks/neil_harbisson_i_listen_to_color.html

AMTS-www.amts.co.in

39

THANK YOU