Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 1
Introduction to Imaging and Multimedia
Fall 2016 Ahmed Elgammal
Rutgers University
§ Introduction to Imaging and Multimedia § This course has two integrated components:
§ Digital Multimedia § Image Processing and Computer Vision
§ You can think of it as an introduction to multimedia with a special focus on digital imaging and video
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 2
§ What is Multimedia ?
What is Multimedia § When different people mention the term multimedia, they often
have quite different, or even opposing, viewpoints. § A PC vendor: a PC that has sound capability, a DVD-ROM
drive, and perhaps the superiority of multimedia-enabled microprocessors that understand additional multimedia instructions.
§ A consumer entertainment vendor: interactive smart TV with hundreds of digital channels available, or a cable TV-like service delivered over a high-speed Internet connection.
§ A Computer Science (CS) student: applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity.
§ Evolving definition…
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 3
Multimedia in Computer Science § Digital Multimedia - Computational Multimedia
Digital Multimedia is the field concerned with computer-controlled integration of text, graphics, images, videos, audio, and any other medium where every type of information can be represented, transmitted and processed digitally.
Digital Multimedia: Historical Perspective § The word multimedia was coined in the beginning of
the 1990s § Mainly after the success of digital audio recording on
CDs § The next anticipated step was to create digital content
involving image, text, video, along with the audio § Multimedia CD-ROMs (late 90s) : ex, Encyclopedia
Britannica, games with simple graphics, § Experience only limited to a single user interacting
with a PC § Things have changed dramatically since then.
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 4
Example MM Applications § Multimedia involves multiple modalities of text, audio, images,
drawings, animation, and video. Examples of how these modalities are put to use: § Video teleconferencing. § Distributed lectures for higher education. § Tele-medicine. § Co-operative work environments. § Searching in (very) large video and image databases for target visual
objects. § “Augmented” reality: placing real-appearing computer graphics and
video objects into scenes. § Including audio cues for where video-conference participants are
located. Taking into account gaze direction and attention of participants as well
Example MM Applications § Building searchable features into new video, and enabling
very high to very low-bit-rate use of new, scalable multimedia products.
§ Making multimedia components editable. allow the user side to decide what components, video, graphics, etc., are actually viewed; allow the client to move components around or delete them. Making components distributed.
§ Building “inverse-Hollywood” applications that can recreate the process by which a video was made. This then allows storyboard pruning and concise video summarization.
§ Using voice-recognition to build an interactive environment, say a kitchen-wall web browser
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 5
Multimedia life cycle § Four main phases in multimedia content life cycle
§ Multimedia Generation, Authoring, Capturing… § Multimedia Representation, and processing § Multimedia Retrieval: answering user queries. § Multimedia Delivery
Capturing (digitization)
Authoring Editing
Indexing Archival
Client
Retrieval
Sens
ors /
inpu
t
MM Delivery
MM Generation MM Archival MM Retrieval
User ?
User ?
User ?
User ?
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 6
Inherent Quality of Multimedia Data § Digital: it is always bits and bytes § Voluminous: the storage and transmission bandwidth
limitations require that the data be compressed § Interactive § Real-time and synchronization: very small and
bounded delay while transmitting information. Intra-media and inter-media synchronization
§ Multimedia and Computer Science: Multimedia is in the
intersection among different areas § Graphics, § HCI, § Visualization, § Computer vision, § Data compression, § Algorithms, § Networking, § Database systems, § Data mining § Architecture and operating systems, § ….
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 7
Multimedia Research Topics and Projects To the computer science researcher, multimedia consists of a wide
variety of topics: § Multimedia processing and coding: multimedia content
analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc.
§ Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases.
§ Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems. Multi-modal interaction and integration: “ubiquity” web-everywhere devices, multimedia education including Computer Supported Collaborative Learning and design, and applications of virtual environments.
Multimedia Systems § A Multimedia System is a system capable of
processing multimedia data and applications. § A Multimedia System is characterized by the
processing, storage, generation, manipulation and rendition of Multimedia information.
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 8
Components of a Multimedia System
§ Capture devices § e.g. Video Camera, Microphone, Digitising/Sampling Hardware, etc.
§ Storage Devices § e.g. Hard disks, CD-ROMs, DVD, Blu-ray, etc
§ Communication Networks § Internet, wireless internet, etc.
§ Computer Systems § e.g. Desktop machines, Workstations, smart phones, tablets
§ Rendering Devices § e.g. CD-quality speakers, HDTV, Hi-Res monitors, Color printers etc.
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 9
Challenges for Multimedia Systems § Distributed Networks § Temporal relationship between data
§ Render different data at same time continuously. § Sequencing within the media (e.g. playing frames in correct
order/time frame in video) § Synchronization — inter-media scheduling
§ E.g. Video and Audio — Lip synchronization is clearly important for humans to watch playback of video and audio and even animation and audio.
Hardware § All contemporary personal computers are quite
capable of displaying MM content § Even Almost all cell phones § For authoring content, a more powerful machine is
needed § Bandwidth is still and always will be an issue
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 10
Multimedia System Desirable Features
§ Very High Processing Power § Special Hardware/Software needed (e.g. GPUs) § Efficient I/O § Large Storage and Memory § High Speed Network Support
Multimedia History § Before the digital age…
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 11
Multimedia: Historical Perspective 3
storage and distribution. The process mostly involves state-of-the-artcompression algorithms and standards for audio, video, images, and graphics.
• Distribution—Distribution involves how multimedia content is distributed viavarious media, such as wired cables, optical networks, satellite, wirelessnetworks, or any combination thereof, to specific platforms ranging fromtelevision, computers, personal digital assistants (PDAs), and so on.
This threefold view is not new and has been used for information in general—creating or gathering information, storing or recording it, and distributing it tothe end user. The table shown in Figure 1-1 gives an evolutionary perspective onthe type of information that people have grown accustomed to through the ages, the
Time and Type of Storage Mode of Age era information medium distribution
Prehistoric 15,000 BC Sounds to Rock –communicate, surfaces,gestures, cave wallspainting
Ancient 500 BC Alphabets, Invention of People drawing paper delivering
messages,horseback
Middle Ages 400–1000 AD Letters, Books Beginning ofwriting a postal
system
Renaissance 1300–1800 AD News, Books, Printing paintings, libraries press, steammagazine engines,
automobiles
Modern world 1900 AD Morse code, Film, Telegramradio, magnetic service,photographs, tapes, wirelessmovies phonograph radio waves
Electronic 1950–1980 Telephone, Electronic Radio and TVtelevision, fax, memory, broadcasting,computers cassette tapes, satellite
LP records communication
Digital 1980 to Computers, Hard disks, Ethernet,present day digital video, CD-ROMs, wireless
surround DVDs networks,sound optical
networks
Figure 1-1 A brief evolution of information
Copyright 2009 Cengage Learning, Inc. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part.
Licensed to:
Multimedia: Text
Parchment Papyrus Stone
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 12
Paper § Han Dynasty of China (202 BC). Raw material such as tree bark was finely chopped, mixed
with water, spread onto screens, and dried. Well guarded secret § Introduced in Europe in 600AD through the Middle East § First paper mill in Europe was in Spain, in 1120. More mills appeared in Italy in
about the 13th century. They used hemp and linen rags as a source of fiber. Paper is recorded as being manufactured in both Italy and Germany by 1400.
Printing Press
§ In 1451, Johannes Gutenberg and Johannes Fust went into partnership and produced a forty-two line Bible and a thirty-two line Latin Grammar
§ By 1465, printing presses based on Gutenberg's moveable type could be found in Italy, by 1470 in Paris. London followed in 1480. By 1499, there were presses in Stockholm, Constantinople and Lisbon.
§ By 1500, Europe contained in excess of nine million volumes, of thirty thousand titles all of which came off the presses of more than one thousand printers.
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 13
Multimedia Data: Text § Text
§ Input: keyboard, touch pad § Stored and input character by character. § Storage of text is 1 or 2 bytes per character. § Other forms of data (e.g. Spreadsheet files, XML) may store format as
text (with formatting). § Format: Raw text or formatted text
e.g HTML, Rich Text Format (RTF), Word Processing, or a programming language source
Sequential: may have natural implied sequence e.g. HTML format sequence, Sequence of Java program statements.
§ Size Not significant compared with other multimedia data.
Images
Niepce 1825 (8 hours)
Daguerre 1838 (10 mn)
1888 Eastman's Kodak camera on the market with the slogan "You press the button, we do the rest"
Maxwell 1861 (color)
Now: digital CCD 1969 at AT&T, CMOS
Camera obscura Mozi 400BC Aristotle 350BC Ibn al-Haitham, 1000AD
Painting on some media: From Prehistoric times till today
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 14
Multimedia Data: Images § Images
§ Still pictures which (uncompressed) are represented as a bitmap (a grid of pixels).
§ Input: Digital camera, scanner or generated by graphics editing programs § Analog sources will require digitizing. § Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale,
Color Map) or 24 Bits per pixel (True Color) § Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512 24 bit
image takes 3/4 Mb with no compression. § Storage increases with image size § Compression is commonly applied
Stored music
15th century
1598
1870 Player piano
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 15
Stored audio
1860:phonautograph
Modern period: music
Then digital
1877-1887:
Phonograph and gramophone
1904
Read-Write !
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 16
Multimedia Data: Audio § Audio
§ Audio signals are continuous analog signals. § Input: microphones and then digitized and stored § usually compressed. § CD Quality Audio requires 16-bit sampling at 44.1 KHz § 1 Minute of Mono CD quality audio requires 5 Mb.
Multimedia Data: Video § Video
§ Input: usually captured by a video camera. § There are a variety of video (analog and digital) formats § Raw video can be regarded as being a series of single images. § There are typically 25, 30 or 50 frames per second. § a 512x512 size monochrome video images take 25*0.25 = 6.25Mb for 1
second to store uncompressed. § Digital video clearly needs to be compressed.
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 17
How many images and videos are online?
§ Cameras are everywhere now: in our pockets, watching over us at different scales, even inside our bodies. § Need tools for processing, annotation, archiving,…
§ Images and movies are everywhere: § In 2009: 2.5 B photos uploaded to Facebook every
month § That’s 30 B photos per year on FB alone § In 2011: 6 B per month – 90 B total § Youtube: in 2011, 48 hours of video are uploaded every
minute, resulting in nearly 8 years of content uploaded every day. In 2014, the number is 100 hours/minute.
Environment
user
Traditional Multimedia Interface vs. Ubiquitous Interface
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 18
Multimedia in a Ubiquitous Computing World
§ Multimedia is becoming available all the time around us. § Ubiquitous computing focus on integrating computation into the
environment, rather than having computers as distinct objects. § Ubiquitous computing covers wide range of research topics,
such as distributed computing, mobile computing, and sensor networks.
§ Mobile phones now are the most commonly used multimedia devices.
What this course is not about § This course is not about
§ Web design § Multimedia tools § Computer graphics and animation § Internet technology
Introduction to Imaging and Multimedia 9/28/16
A. Elgammal, Rutgers 19
Resources § Multimedia Systems Ch 1 § Some Slides by Prof G. Medioni @ USC