smart image and video search presentation
TRANSCRIPT
![Page 1: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/1.jpg)
Smart Image And Video SearchEnterprise Image and Video Analytics
![Page 2: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/2.jpg)
The problem
• Government has piles of video, analysts miss things after looking at it for 20 minutes !
• What to do?
Help!
We are collecting video and images EVERYWHERE !!!!!!!!
![Page 3: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/3.jpg)
How to search image/video for what you are looking for
identify people suspected IED sites
changes in geospatial data
geographic features
IED circuit boardsyellow ships
trains
planesmini vans
known terrorists
activity …
![Page 4: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/4.jpg)
The Enterprise Solution• Only piXserve by piXlogic offers a generalized enterprise
class image/video search capability• Enterprise class 64bit Java application for Windows Server
2003+• Indexes all objects in images and frames of video and also
attempts to classify those objects• Supports 90+ image or video formats• Generates a light weight XML index and leaves the original
information where it lives and can place index in RDBMS• Special focus on text in image and facial recognition• Offers a JMS based event and alerting system• Supports a web services REST API
![Page 5: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/5.jpg)
piXlogic A New Approach
Background
Foreground
Connected Background
Face
Tree
Sky
“Seeing”
Synthesize descriptions
Discern shapes
Vectorize shapes
LocationSizeColorGeometryRelationshipsEtc.
“Comparing” Direct ComparisonsRelationshipsContext
“Understanding”
![Page 6: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/6.jpg)
Think About Text Indexing
By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest. Confucius
At their most Basic they Index Text ….
By three methods we may learn wisdom….
Confucius
<Keywords>
Enter Query:
Search
Common Text Indexing Engines:
<Phrases>“three methods”“learn wisdom”“by imitation”“by experience”
</Keywords>
</Phrases>
Its All About Metadata
![Page 7: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/7.jpg)
Text Result Set
Rank Date Description . 98% 01/01/2011 Confucius dreams .. 87% 04/12/2010 Eastern philosophy and culture84% 07/23/2009 Harvard study on ancient China 73% 02/12/2009 Fortune cookies run amok65% 03/01/2008 Medical studies calm fears in …58% 01/01/2007 Confucius dreams .. 47% 04/12/2007 Eastern philosophy and culture44% 07/23/2004 Harvard study on ancient China 43% 02/12/2000 Fortune cookies run amok35% 03/01/1999 Medical studies calm fears in …
![Page 8: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/8.jpg)
Image/Full Motion Video Indexing
Foreground
Background
Background
WE LOOK FOR SHAPES AT THE PIXEL LEVEL
![Page 9: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/9.jpg)
How Pixel Indexing WorksEnterprise Pixel Indexing Engine:
Indexes Key Objects …
…
Key Objects
people" id="61" <ImageIndex ImageURL="C:\Data\Sample\quebec2008-1.jpg" ImageDate=" 2 6 11016" ImageSize="57489323"><ImageText Name="1" Sz="4" X="66" Y="1" W="2" H="2" L="0" U="0" V="0" R="400852085d086608" FC="41002e002e005600" FL="2e002e002e007400" FN="2e00310031003400" <ImageText Name="2" Sz="4" X="67" Y="1" W="2" H="2" L="0" U="0" V="0" R="7c08890891089808" FC="52002e0055002e00" FL="2e0074002e002e00" FN="2e0031002e003100" <ImageText Name="3" Sz="3" X="11" Y="2" W="4" H="3" L="0" U="0" V="0" R="65018801a401" FC="48004c005a00" FL="6e002e007a00" FN="2e0031003200" …
Synthetic Metadata
Enter Query:
Search
Query By ImageEnter Text
![Page 10: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/10.jpg)
Date Result 01/01/2011
04/12/2010
07/23/2009
02/12/2009
03/01/2008
01/01/2007
08/14/2005
Image/Video Result Set
04/14/2009
![Page 11: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/11.jpg)
How Do We Search
• Select a database or databases to search• Select search term type– Image only– Keyword– Text in image– Text and image– Coordinates– Faces
![Page 12: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/12.jpg)
How Does It Work
![Page 13: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/13.jpg)
Segment Images
…
Break up images into smaller and smaller pieces
![Page 14: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/14.jpg)
Fill In Missing Information• Uses an ontology called Notions:– Apriori information about the world– Attempts to classify objects in an image or frame
of video.– Reasons using the notions to identify objects– System has over 70 notions
PeopleTree
Helicopter
Beach
Sky
![Page 15: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/15.jpg)
Auto-Tagging in piXserve• Add as much meaningful metadata as possible at the
time of indexing (“Auto Tagging”)• Several mechanisms:– “Name Faces”: name a person and autotag when the
person is seen in the image/video frame– “Keyword Recommendations”: add keywords based on
how similar images have been tagged in the past– “Objects-2D”: autotag for a specific item based on a
single example (can use multiple views for 3D objects) – “Notions”: autotag generic classes of items (conceptual
tagging)
![Page 16: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/16.jpg)
Advanced Capabilities
• Text in image or video– Not OCR– Uses the same techniques as we use to find any
object in image or video frames– Tuned for languages – English, European, Some
Chinese, Japanese, Korean, Russian• Faces (biometric)– No pre-registration – just uses images– Very high precision and recall using the NIST standard
test >98%
![Page 17: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/17.jpg)
2D Objects Library
• Topic specific filtering– Specific library can be built to filter for specific
information – as images/frames of video arrive they are
compared to this information and specific tagging / matching occurs
– E.g. light arms, ships, clothing, hair styles etc.
![Page 18: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/18.jpg)
Text In ImageEnter Query:
Search
Query By Text in Image
HEALTH
Enter Text
![Page 19: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/19.jpg)
RESTful API
• All Server features & functions • All User interface functions• Data access•Machine to machine integration• API Guide
![Page 20: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/20.jpg)
Alerting Engine
• Create standing queries• When an event in an image triggers an alert
two things happen:– A JMS message is sent – to whatever program or
system designated– An email message is sent to whomever has been
designated to receive the alert
![Page 22: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/22.jpg)
SUPPORTING MATERIALS
![Page 23: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/23.jpg)
The Industry
• Video monitoring software packages integrated with DVR support such as Object Video, Image IQ, 3VR, Virage and others can recognize certain objects, perimeters, and activities ..
• They are security focused .. Often with some ability to search faces and possibly license tags
![Page 24: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/24.jpg)
The Industry (Cont’d)• Other packages (COTS & GOTS) are starting to address generalized image and
video search• Numenta – offers an image search/pattern matching toolkit (not enterprise .. and
each solution must be hand crafted)• Pixsy – offers image search to match images – for internet advertising and
shopping online• IDEE Tin-Eye – offers image on the internet – more advertising based approaches• KEVA (Sierra Nevada Corp/PNNL) is a GOTS parallel processing engine for
searching video – government only and fairly early stage• Google and Facebook – Both have image and video tagging algorithms in their
general search and/or analytics suites, with some specific object search capabilities – but web focused only
• There are others .. But NONE are enterprise class generalized Google like search engines
• NERVVE
![Page 25: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/25.jpg)
Notions/Classesartificial" id="1" white" id="2" flower" id="3" sky" id="4" gray" id="5" vegetation" id="6" lawn" id="7" skin" id="8" road" id="9" vertical" id="10" landscape" id="11" red" id="12" green" id="13" blue" id="14" aqua" id="15" yellow" id="16" gun" id="65"
smooth" id="19" circle" id="20" polygonal" id="21" sparse" id="22" face" id="23" horizontal" id="24" document" id="25" map" id="26" diagram" id="27" text" id="28" bikini" id="29" exposed body" id="30" line" id="31" adult" id="32" mountain" id="60" people" id="61" fire" id="62"
rectangle" id="37" pentagon" id="38" clutter" id="39" star" id="40" zebra" id="41" silver" id="42" pumpkin" id="43" bride" id="44" sea" id="45" snow" id="46" sand" id="47" beach" id="48" fireworks" id="49" sunset" id="50" soccer ball" id="59" airplane" id="66"
purple" id="17" orange" id="18" child" id="33" drawing" id="34" flat" id="35" triangle" id="36"flash" id="51" indoor" id="52" outdoor" id="53" near" id="54" dark" id="55" helicopter" id="56" building" id="57" car" id="58" crowd" id="63" 3D drawing" id="64"
![Page 26: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/26.jpg)
Segment the Image or Video Frame
Indexing breaks the image up into smaller and smaller pieceswhich are more easily mathematically described
![Page 27: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/27.jpg)
Medical Uses
Ductal CarcinomaPerspective 1
Ductal CarcinomaPerspective 2
![Page 28: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/28.jpg)
Medical Uses
Frontal Lobe InjuryPerspective 1
Frontal Lobe InjuryPerspective 2
![Page 29: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/29.jpg)
Physical CONOP
video/imagesource
continuousstreams or files
contentStore
db
Store inDAM system
indexingservers
direct indexing from source
indirect indexingfrom DAM system
Web Server
Web Server
Web Server
analyst
analyst
analyst
analyst
analystnetwork
Index held in memory
searching
downstreamanalytical systems
analysis
Alert Server
![Page 30: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/30.jpg)
application
faces
Logical CONOPS 1Watching The Border
continuousvideo/image
source
automatic continuous
indexing
indexal
erts
reporting
metadata• name• watch list• vehicle make• vehicle model• tag state• tag number
analysts
downstreamanalyticalsystems
correlationFilter
indexes
vehicles
license plates
Filter indexesFilter
indexes
![Page 31: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/31.jpg)
Conceptual CONOPS 1Watching The Border
vehicle border crossing
cameras
Tag - state & text extracted Vehicle - make/model extractedFace - biometrics
vehicle
tag face
Indexes-watch list-vehicle types-state license plates
-State: Florida-Tag: SAMPLE
-Daniel San Diego-U.S. Most Wanted
-Make: Honda-Model: Civic
![Page 32: Smart image and video search presentation](https://reader036.vdocument.in/reader036/viewer/2022081507/5552504bb4c905954f8b508b/html5/thumbnails/32.jpg)
NET EMULATED UAV
use video filespiXlogic index
video and searches
piXlogic matches and
clips video on target
event
piXlogic apitransfers clip
to flumedirectory
flume transmits matches over
MITRE network emulator
1
activity
2 3 4 5
RADAR INFRARED VIDEO VIDEO IMAGERY
Query:• Buildings• Ships• Aircraft
TNT - CONOPS
Alert Results
network
pixserve
flume
net emulator
commsLo
cal N
etw
ork
FIELD ANALYSTS