Lin-Kai Chen VClab

Content Adaptation Framework:Bringing the Internet to Information

Appliances

Content-Based Transcoding of Images

in the Internet

IBM T.J Watson Research Center

J.R Smith,R Mohan and Chung-Sheng Li

Lin-Kai Chen VClab

Introduction

• This paper present Content Adaptation Framework for customizing internet content for diverse client devices.

• Converting content items into alternate modalities and resolutions so as to meet the characteristics of the client devices.

• Different modalities: video,image,text,audio.• Different fidelities: reduced resolution and

summaries.

Lin-Kai Chen VClab

Internet

Universal Access

Via

Content

Adaptation

Information

ContentIn the

office

On the road

Lin-Kai Chen VClab

Introduction(Cont.)

• The content can be adapted on the fly or transcoded in advance.

• This content adaptation can be performed at server client or proxy.

Lin-Kai Chen VClab

Content adaptation Framework

Lin-Kai Chen VClab

Content adaptation Framework(Cont.)

• The components are:• Analysis： To extract meta-data, such as resourc

e requirements and the type and purpose of the content, used in guiding subsequent transcoding and selection process.

• Transcoding： Based on the capabilities of client devices,different transcoding modules are employed to generate versions of the content in different resolutions and modalities.

Lin-Kai Chen VClab

Content adaptation Framework(Cont.)

• Selection:Selects the resolutions or modalities that best meet the client capabilities.

• Rendering:The selected content is then rendered in a suitable delivery format(eg.HTML).

• Description:The InfoPyramid is used to represent the multiple resolutions and modalities of the transcoded content.

Lin-Kai Chen VClab

Lin-Kai Chen VClab

Content Analysis

• Resource requirement– Content size in bits.

– Display size:height,width,and area.

– Streaming bit-rate.

– Color requirements.

– Software requirements:support for specific codecs.

– Hardware requirements.

Lin-Kai Chen VClab

Content Analysis

• Content type and purpose– The type of content decides which specific transcoding

algorithm are applied.

– One focus of the content analysis is images as they currently account for the largest amount of web traffic in terms of bits.

– Analyzing images to determine their type and purpose.

Lin-Kai Chen VClab

Image content analysis

Image type

Decision tree

Features

Training set

Parameters

Real time trans.

X[m.n]

Xr, Xg, Xb

μsPv, Vv μh, μv

Color B/W Gray

B/W graphicphoto

Purpose class

Related text

Lin-Kai Chen VClab

Image Content Analysis

• Image type classes:– BWG : b/w graphic.

– BWP : b/w photo.

– GRG : gray graphic.

– GRP : gray photo

– SCG : simple color graphic

– CCG : complex color graphic

– CP : color photo.

Lin-Kai Chen VClab

Image Content Analysis

• Image purpose classes– ADV: advertisement, ie. Banner ads.

– DEC: decoration, ie. Background textures.

– BUL: bullets, points,dots.

– RUL: rules,lines,separators.

– MAP: maps, ie,images with click focus.

– INF: information:icons,logos.

– NAV: navigation, ie,arrows

– CON: content related: news photos.

Lin-Kai Chen VClab

Image Content Analysis

• Image type classification– Decision tree as a classifier.

– It consists of 5 decision points, each of which utilizes a set of features extracted from the images.

– Classification parameters from the training set of 1282 images.

Lin-Kai Chen VClab

Lin-Kai Chen VClab

Color vs. non-color

• Each image X[m,n] has xr, xg, xb.

• Using the measurement of the mean saturation per pixel μs. The saturation channel ys .

ys = max(xr, xg, xb) – min(xr, xg, xb)

• μs discriminates well between color and non-color images.

nm

ss nmyMN ,

],[1

Lin-Kai Chen VClab

B/W vs. Gray

• Intensity channel

• Intensity entropy

• ,where

•

bgrv xxxy 106030 ...

][log][ kpkpPkv 2

255

0

nm

v

otherwise

nmykif

MNkp

,

],[][

0

11

Lin-Kai Chen VClab

B/W vs. Gray

• Intensity variance

nm vvv nmy

MNV

,)],[( 21

nm

vv nmyMN ,

],[1

Lin-Kai Chen VClab

BWG vs. BWP

• It uses the minimum of the mean number of intensity switches in horizontal and vertical scans of images.

Lin-Kai Chen VClab

GRG vs. GRP

• Using the intensity switch measure Wv and intensity entropy Pv .

• As to switch measure, it has a much lower value for b/w graphics.

Lin-Kai Chen VClab

SCG vs. CCG vs. CP

• The images are transformed to HSV and vector quantized.

• The process generates a 166-HSV color representation of the image y166 ,where each pixel refers to an index in the HSV color look-up table.

• The mean color switch per pixel Wv

• Color entropy P166

Lin-Kai Chen VClab

Lin-Kai Chen VClab

Decision tree : correct classification rate(%)

Decision1

non-colorvs.

Color

Decision2B/W

vs.Gray

Decision3B/W

Graphicvs.

Photo

Decision4Gray

Graphicvs.

Photo

Decision5Color

Graphic

vs.Photo

88.2 95.3 97.8 77.5 68.2

Image type class correct classification rate(%)

B/Wgraphic

B/Wphoto

Graygraphi

c

Grayphoto

SCG CCG Colorphoto

91 91 65 78 52 42 70

Lin-Kai Chen VClab

Image purpose classification

• Image purpose is detected by applying a rule induction engine on the image type,document context and related text.

• Context for images.– BAK: background, ie <body background=…..>

– INL: inline,ie <img src=…>

– ISM: is map

– REF: referenced , ie <a href=…>

– LIN: linked, ie <a href=….><img src=…></a>

Lin-Kai Chen VClab

Image purpose classification

• Text terms:extracted from the ‘alt’ tag, the image URL address strings,and the text nearby the images in the web documents.

• D={‘ad’,’texture’,’bullet’,’map’,’logo’,’icon’}

Lin-Kai Chen VClab

Client device characteristics

Lin-Kai Chen VClab

Transcoding policy

• Transcoding functions– Size:minify,crop,and subsample.

– Fidelity:JPEG,GIF compress, quantize, reduce resolution, smooth,sharpen…etc.

– Color content: reduce color, convert to gray,b/w.

– Substitution: attributes, text,type,purpose, and remove image.

Lin-Kai Chen VClab

Transcoding policy

Lin-Kai Chen VClab

Transcoding policy

• For compression, JPEG works well for gray photo graphics but not for graphics.For GIF, the reverse is true.

• The transcoding policies also make use of the image purpose analysis.

Lin-Kai Chen VClab

Conclusion

• The content adaptation can be performed at various nodes in the internet:the server,proxy and client.

• They proposed the content discription scheme based on the InfoPyramid, to the MPEG-7 and W3C standards bodies.

• The system utilizes transcoding policies based on the content classes to transcode and adapt images.