adaptation use case for structured scalable meta-formats (ssm) … · 2018-09-13 · elementary...

Adaptation Use Cases for Structured Scalable Meta-formats (SSM) Version 2.0

Debargha Mukherjee, Geraldine Kuo, Shih-ta Hsiang, Amir Said

Imaging Systems Laboratory, Hewlett Packard Laboratories, Palo Alto, CA Email: {debargha, gkuo, hsiang, said}@hpl.hp.com

Abstract Recently, an end-to-end framework called SSM, for fully content agnostic adaptation

of scalable bit-streams or certain non-scalable bit-streams operating in scalable modes, has been proposed. SSM develops metadata to describe a scalable bit-stream in terms of adaptations that can be conducted on it to obtain lower versions, and also allows recipients to specify usage environment constraints based on which a piece of content is to be adapted by a network adaptation engine. Based on this framework, a fully format agnostic universal adaptation engine software has been developed. In this report, we present adaptation use cases involving several different standard and non-standard bit-stream formats (MPEG-4, JPEG2000, MC-EZBC), and show how the universal adaptation engine adapts them. The SSM schema and software versions considered is 2.0.

1. Introduction The SSM framework [1][2][3][4][5], enables creation of universal adaptation engines

that process resource description metadata (XML) and recipient constraints (XML) for decision making and adaptation of a wide class of resource bit-streams that can be adapted by dropping bit-stream segments followed by minor editing operations, broadly referred to as scalable bit-streams. The strength of the approach is that the entire adaptation process is conducted by a universal content non-specific adaptation engine, with the only inputs changing for different types of content being the descriptor and constraints XMLs. There is no need for external style sheets or external modules specific for the content for either adaptation or decision making, as long as the bit-stream is scalable as per the above definition.

Universal adaptation engines may reside in network nodes, such as in edge servers, content-servers, or midstream routing servers, and can often be combined in a chain as shown in Figure 1. In this generic architecture, the scalable bit-stream resource, as well as the resource desription metadata moves downstream, while constraints, originating from the recipients move upstream.

From the above generic delivery architecture, an elemental universal adaptation engine can be isolated. The interface to the universal adaptation engine is shown in Figure 2. The three inputs to the adaptation engine are: (1) resource description metadata (XML), (2) resource bit-stream, (3) recipient constraints (XML). These inputs are processed in a fully format agnostic manner to make appropriate decisions, and adapt both the resource and the resource description metadata for use in a subsequent

adaptation stage. The outputs are therefore: (1) adapted resource description metadata, and (2) adapted resource bit-stream.

This interface is very well defined, and for all practical purposes, the adaptation engine can be regarded as a format agnostic black box. Just by changing the descriptor and the constraints appropriately, adaptations on a wide range of scalable formats are possible.

In order to demonstrate fully format-agnostic adaptation, we use four different use cases involving the following bit-streams: MPEG-4 Visual ES [6][7], MPEG-4 Visual Texture Coding [6], MC-EZBC [8][9][10][11][12], and JPEG2000 [13]. MC-EZBC is a non-standard fully scalable video codec from RPI.

All the XML files related to these use cases are compliant with SSM version 2.0 [5]. All the descriptors, schemas, the resource bit-streams, and adaptation engine software executables, could be found online at:

http://www.hpl.hp.com/research/ssm/SSM_UseCases.zip.

This report walks the reader through the four cases while refering to XML documents, and bit-streams in this package. All the executables are on the Windows platform.

2. MPEG-4 Visual Elementary Streams The first use case considered is based on frame-dropping on MPEG-4 visual

Adaptation Engine

Recipient

Recipient

Adaptation Engines do not understand the specifics of the data, only the meta-format. They process information about all outbound connection(s).

Originator Adaptation

Engine

SSM-compliant scalable media bit-stream

Resource description meta-data (XML)

Outbound Constraints from recipients (XML)

Recipient

Figure 1. Media-Type-Independent Adaptation and Delivery Chain

Input SSM Adaptation

Engine

Outbound Constraints XML

Input resource description XML

Adapted resource description XML

Adapted SSM

Figure 2. Adaptation Engine external model

elementary streams. We uses two different bitstreams in the validation: akiyo.mpg4 and foreman.mpg4. In each case, it was assumed that each GOP has a violence rating between 1 and 5. Furthermore, the bit-stream was arranged in multiple temporal resolution layers based on its encoding structure.

2.1 Use Cases for akiyo.mpg4 The akiyo bit-stream resource is defined to have exactly one parcel in the SSM

structure, and there is one component ssm:comp:akiyo in the parcel. The component has two tiers, the first based on violence rating and the second based on temporal resolution, as shown in Figure 3.

The following is the resource description XML for akiyo.mpg4 (SSMDescription_akiyo.xml):

<?xml version="1.0" ?> <SSMDescription xsi:schemaLocation="SSMDescription SSMDescription.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMDescription" xmlns:ssm="SSMCommon"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:akiyo"> <tierInfo numTiers="2"> <tierElemInfo numLayers="5" origLayers="5" exclusiveFlag="false" tier="0" /> <tierElemInfo numLayers="4" origLayers="4" exclusiveFlag="false" tier="1" /> </tierInfo> <atomToc start="0" addressType="absolute"> <atomTocEntry indices="0 0" start="18" length="4641" /> <atomTocEntry indices="0 1" start="4659" length="98" /> <atomTocEntry indices="0 2" start="4757" length="16" /> <atomTocEntry indices="0 3" start="4773" length="23" /> <atomTocEntry indices="0 1" start="4796" length="178" /> <atomTocEntry indices="0 2" start="4974" length="53" /> <atomTocEntry indices="0 3" start="5027" length="39" /> <atomTocEntry indices="0 1" start="5066" length="235" /> <atomTocEntry indices="0 2" start="5301" length="66" /> <atomTocEntry indices="0 3" start="5367" length="65" /> <atomTocEntry indices="0 1" start="5432" length="274" /> <atomTocEntry indices="0 2" start="5706" length="74" /> <atomTocEntry indices="0 3" start="5780" length="68" />

Violence rating

1 2 3 4 5

I

P

B1

B2

Temporal resolution

Figure 3. SSM structure for akiyo sequence

<feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:akiyo" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>18</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dims="2" dimToTierMap="0 1" distType="nonDecreasing">23172 31075 35674 40273 41737 56234 64834 73782 59944 80442 92728 105598 78327 107315 124371 142290 92014 124972 144520 165034</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData /> <globalCombAvarData> <globalCombAvar avar="ssm:avar:totalVideoTimeInSec"> <ssm:constant value="10”/> </globalCombAvar> </globalCombAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

As seen in the descriptor, the first tier has 5 layers. The 5 layers indicate the violence rating for the atoms. All the atoms that have violence rating equal to 1 are in layer-0. All the atoms that have violence rating equals to 2 are in layer-1, and so on. The violence rating value could be obtained from the adaptation variable ssm:avar:violenceRating. If the outbound adapted resource contains all the layers for the first tier, the violence rating would be 5; if the outbound adapted resource contains 4 layers, the violenceRating would be 4, and so on.

The second tier has 4 layers. Akiyo has an encoding structure where a GOP is 15 frames, and the sequence of VOP encoding modes are {I, B, B, P, B, B, P, B, B, P, B, B, P, B, B}. All the I-VOPs are in layer-0, all the P-VOPs are in the layer-1, all the first B-VOPs of a contiguous pair of B-VOPs are in the layer-2, and all the second B-VOPs of the pair B-VOPs are in the layer-3. The frames per second value could be obtained from the adaptation variable ssm:avar:framesPerSecond. If the outbound adapted resource contains all 4 layers, there would be 30 frames per second. If the outbound adapted resource contains 3 layers, there would be 20 frames per second, and so on.

All the atoms get their indices based on their violence rating, and their type of VOP. Violence ratings were arbitrarily assigned to GOPs in the akiyo sequence to emulate the case of a real video with violent sections.

An adaptation variable ssm:avar:codesize is also defined to return the codesize of the resource based on different adaptation bounding boxes. If all layers from both of the tiers are included, the codesize would be 165034. If 3 layers for the first tier, and 2 layers for the second tier are included, the codesize would be 80442.

A global combination variable ssm:avar:totalVideoTimeInSec is defined to denote a constant 10s indicating the duration of the unadapted video.

Use Case 1A The first use case demonstrated is a multi-step adaptation. The first adaptation step

will remove all B-VOPs and the second adaptation step will remove the atoms that have violent rating more than 3.

The first adaptation outbound constraints XML (SSMAdaptReq_ves1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="15" lowLimit="0"> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

The outbound constraints XML has one limit constraint that requires the value of adaptation variable ssm:avar:framesPerSecond to be less than or equal to 15 and greater or equals to 0. The resultant resource will include all layers from the first tier, since the outbound constraints XML does not limit any of the violence rating; and it will include 2 layers from the second tier, and dropped the layers that include all the B-VOPs.

Here is the command line for the first step adaptation:

SSMAdaptEngine.exe -di SSMDescription_akiyo.xml -ar SSMAdaptReq_ves1.xml -do SSMDescription_akiyo_ves1.xml -bi akiyo.mpg4 –bo akiyo_ves1.mpg4

Here is the adapted resource description XML for akiyo after the first adaptation (SSMDescription_akiyo_ves1.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:akiyo"> <tierInfo numTiers="2"> <tierElemInfo exclusiveFlag="false" numLayers="5" origLayers="5" tier="0" /> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="4" tier="1" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0 0" length="4641" start="18" /> <atomTocEntry addressType="relative" indices="0 1" length="98" start="4659" /> <atomTocEntry addressType="relative" indices="0 1" length="178" start="4757" /> <atomTocEntry addressType="relative" indices="0 1" length="235" start="4935" />

</featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData /> <globalCombAvarData> <globalCombAvar avar="ssm:avar:totalVideoTimeInSec"> <ssm:constant value="10”/> </globalCombAvar> </globalCombAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

The second step of the adaptation will drop all atoms that have violence rating greater than or equal to 4. Here is the outbound constraints XML (SSMAdaptReq_ves2.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="3" lowLimit="1"> <ssm:adapVar avar="ssm:avar:violenceRating" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

The outbound constraints XML has one limit constraint that requires the value of adaptation variable ssm:avar:violenceRating to be less or equals to 3 and greater or equals to 1. The result resource will only include 3 layers from the first tier.

Here is the command line for the second step adaptation:

SSMAdaptEngine.exe -di SSMDescription_akiyo_ves1.xml -ar SSMAdaptReq_ves2.xml -do SSMDescription_akiyo_ves2.xml -bi akiyo_ves1.mpg4 -bo akiyo_ves2.mpg4 If adapted descriptor SSMDescription_akiyo_ves2.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_akiyo_ves1.xml -ar SSMAdaptReq_ves2.xml -bi akiyo_ves1.mpg4 -bo akiyo_ves2.mpg4

Here is the adapted resource XML for akiyo after the first and second adaptation steps (SSMDescription_akiyo_ves2.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:akiyo"> <tierInfo numTiers="2">

<feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:akiyo" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>18</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0 1" dims="2" distType="nonDecreasing">23172 31075 41737 56234 59944 80442</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData /> <globalCombAvarData> <globalCombAvar avar="ssm:avar:totalVideoTimeInSec"> <ssm:constant value="10”/> </globalCombAvar> </globalCombAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

Use Case 1B In this use case, we will limit the violence rating as well as codesize. Here is the

outbound constraints XML (SSMAdaptReq_akiyo1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="4" lowLimit="1"> <ssm:adapVar avar="ssm:avar:violenceRating" /> </limitConstraint> <limitConstraint highLimit="1" lowLimit="1"> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:adapVar avar="ssm:avar:totalVideoTimeInSec" /> <ssm:constant value="12500" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolIsGEZ" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There are two limit constraints in the outbound constraints XML. The first one will drop all the atoms that have violence rating greater than 4. The second limit constraint will ensure that the available bandwidth should be sufficient to transmit the entire video within the time the video is supposed to be played for. In other words, bandwidth (12500 bytes/s) × totalVideoTimeInSec (10 s) will be greater or equal to the codesize. Note, while the above approach may make sense in a streaming scenario assuming adequate buffering, if the time of interest is download time, the line:

<ssm:adapVar avar="ssm:avar:totalVideoTimeInSec"/>

in SSMAdaptReq_akiyo1.xml could be replaced with something like:

<ssm:constant value="15" />

indicating 15s is the maximum allowable download time at 12500 bytes/s bandwidth.

After the adaptation, 4 layers from the first tier, and 3 layers from the second tier will be included in the outbound adapted resource. The codesize with such adaptation point is 124371 bytes and it is less than 12500 times total video time, which is 10 s.

Here is the command line for the adaptation:

SSMAdaptEngine.exe -di SSMDescription_akiyo.xml -ar SSMAdaptReq_akiyo1.xml -do SSMDescription_akiyo1.xml -bi akiyo.mpg4 -bo akiyo1.mpg4 If adapted descriptor SSMDescription_akiyo1.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_akiyo.xml -ar SSMAdaptReq_akiyo1.xml -bi akiyo.mpg4 -bo akiyo1.mpg4

Here is the adapted resource description XML (SSMDescription_akiyo1.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:akiyo"> <tierInfo numTiers="2"> <tierElemInfo exclusiveFlag="false" numLayers="4" origLayers="5" tier="0" /> <tierElemInfo exclusiveFlag="false" numLayers="3" origLayers="4" tier="1" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0 0" length="4641" start="18" /> <atomTocEntry addressType="relative" indices="0 1" length="98" start="4659" /> <atomTocEntry addressType="relative" indices="0 2" length="16" start="4757" /> <atomTocEntry addressType="relative" indices="0 1" length="178" start="4773" /> <atomTocEntry addressType="relative" indices="0 2" length="53" start="4951" /> <atomTocEntry addressType="relative" indices="0 1" length="235" start="5004" /> <atomTocEntry addressType="relative" indices="0 2" length="66" start="5239" /> <atomTocEntry addressType="relative" indices="0 1" length="274" start="5305" /> <atomTocEntry addressType="relative" indices="0 2" length="74" start="5579" /> <atomTocEntry addressType="relative" indices="1 0" length="4670" start="5653" /> <atomTocEntry addressType="relative" indices="1 2" length="416" start="10323" /> <atomTocEntry addressType="relative" indices="1 1" length="591" start="10739" /> <atomTocEntry addressType="relative" indices="1 2" length="153" start="11330" />

<codecOffsetData /> </SSMDescription>

Use Case 1C In this use case, we will select all atoms that have violence rating equals 1 or 3. Here

is the outbound constraints XML (SSMAdaptReq_akiyo2.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0" requestType="unstructured"> <adapVarDriven> <limitConstraint highLimit="1" lowLimit="1"> <ssm:constant value="3" /> <ssm:adapVar avar="ssm:avar:violenceRating" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolNOT" /> <ssm:constant value="1" /> <ssm:adapVar avar="ssm:avar:violenceRating" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolNOT" /> <ssm:operation operator="boolOR" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

In the outbound constraints XML, there is one limit constraint and no optimization constraint. We use requestType equal to unstructured. It means that we will include all the atoms that fulfill the limit constraint, and ignore the SSM dependency model. The constraint will ensure that the selected atoms will have violence rating equal to either 1 or 3. The outbound adapted resource will include all the atoms in the first layer and third layer of the first tier.


SSMAdaptEngine.exe -di SSMDescription_akiyo.xml -ar SSMAdaptReq_akiyo2.xml -do SSMDescription_akiyo2.xml -bi akiyo.mpg4 –bo akiyo2.mpg4 If adapted descriptor SSMDescription_akiyo2.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_akiyo.xml -ar SSMAdaptReq_akiyo2.xml -bi akiyo.mpg4 –bo akiyo2.mpg4

Here is the adapted resource description file (SSMDescription_akiyo2.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData>

<featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0 1" dims="2" distType="nonDecreasing">23172 31075 35674 40273 41737 56234 64834 73782 59944 80442 92728 105598</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData /> <globalCombAvarData> <globalCombAvar avar="ssm:avar:totalVideoTimeInSec"> <ssm:constant value="10”/> </globalCombAvar> </globalCombAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

2.2 Use Cases for foreman.mpg4 For the foreman resource, we define a SSM structure with multiple parcels, that is

more appropriate in a streaming scenario. There are five parcels, each containing one component ssm:comp:foreman covering 2 seconds of video. As in the akiyo use case, we indicate the violence ratings and temporal resolution of the parcels. However, unlike the akiyo case, it is not necessary to organize a parcels into 2 tiers, unless there are more than one violence ratings within a parcel. The SSM structure for the parcels is shown in Figure 4.

The following is the resource description XML for foreman.mpg4 (SSMDescription_foreman.xml):

<?xml version="1.0" ?> <SSMDescription xsi:schemaLocation="SSMDescription SSMDescription.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMDescription" xmlns:ssm="SSMCommon"> <parcelData> <parcel parcelID="0"> <componentData>

Violence rating

1

I

P

B Temporal resolution

2

I

P

B

5 4

I

P

B

1

I

P

B

2

I

P

B

Parcel 0

Parcel 1

Parcel 2

Parcel 3

Parcel 4

Figure 4. SSM structure for Foreman sequence

</globalFeatureData> <globalCombAvarData> <globalCombAvar avar="ssm:avar:parcelTimeInSec"> <ssm:constant value="2”/> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:overallRateToLagrangian" numAxes="1"> <axisValues axis="0" grid="65994 107849 375672 533422 572100" /> <content> <marginalDist dims="1" dimToTierMap="0">0.0007 0.0007 0.0004 0.0002 0.0001 0 0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

Foreman has an encoding structure where a GOP is 30 frames, and the sequence of VOP encoding modes are {I, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B, P, B. Each 2-sec parcel covers two GOPs.

As seen above, all but the second parcel (parcelID=”1”) have only one tier with 3 layers. All the I-VOPs are in layer-0, all the P-VOPs are in the layer-1, and all the B-VOPs are in the layer-2. The frames per second value can be obtained from a global feature variable ssm:avar:framesPerSecond. If the outbound adapted resource contains all 3 layers, there would be 30 frames per second. If the outbound adapted resource contains 2 layers, there would be 15 frames per second, and so on. Each parcel also provides an adaptation variable for the violence rating ssm:avaar:violenceRating. The values for the violence rating adaptation variables are constants for all parcels except the second. For example, in the first parcel (parcelID=”0”), the violence rating is always 1.

In the component in the second parcel (parcelID=”1”), there are two tiers. The first tier has 2 layers. All the atoms that have the same violence rating will belong to the same layer. The violence rating value is obtained from the adaptation variable ssm:avar:violenceRating. If the outbound adapted resource contains both of the layers for the first tier, the violence rating would be 5; if the outbound adapted resource contains 1 layer, the violenceRating would be 2. The second tier has 3 layers and is the same as the tier in other parcels. All the I-VOPs are in layer-0, all the P-VOPs are in the layer-1, and all the B-VOPs are in the layer-2. The frames per second value could be obtained from the parcel feature variable ssm:avar:framesPerSecond. Note that this name is common with the global feature variable, defined for one tier only, the parcel definition takes precedence. If the adapted resource contains all 3 layers in tier 1, there would be 30 frames per second. If the adapted resource contains 2 layers, there would be 15 frames per second, and so on.

In all the parcels, we also define an adaptation variable ssm:avar:codesize to return the codesize of the resource based on different adaptation options.

A global combination variable ssm:avar:parcelTimeInSec is defined to denote a constant 2s indicating the duration of a parcel of unadapted video.

There is also one global lookup table defined ssm:avar:overallRateToLagrangian. It takes one argument which is the overallRate and the LUT will return the lagrangian parameter to use to optimize each parcel in order to obtain the given overall rate.

Use Case 2A In this use case, we use the same multi-step outbound constraints XML files used for

akiyo.mpg4. The first adaptation step will remove all B-VOPs and the second adaptation step will remove the atoms that have violent rating more than 3. Refer to Use Case 1A for the outbound constraints XML files.

The first step adaptation will remove all B-VOPs. Here is the command line for the first step adaptation:

SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_ves1.xml -do SSMDescription_foreman_ves1.xml -bi foreman.mpg4 –bo foreman_ves1.mpg4

Here is the adapted resource description XML for foreman after the first adaptation (SSMDescription_foreman_ves1.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="6445" start="18" /> <atomTocEntry addressType="relative" indices="1" length="1737" start="6463" /> <atomTocEntry addressType="relative" indices="1" length="1948" start="8200" /> <atomTocEntry addressType="relative" indices="1" length="1752" start="10148" /> … <atomTocEntry addressType="relative" indices="1" length="925" start="30568" /> <atomTocEntry addressType="relative" indices="0" length="7131" start="31493" /> <atomTocEntry addressType="relative" indices="1" length="1436" start="38624" /> <atomTocEntry addressType="relative" indices="1" length="1610" start="40060" /> <atomTocEntry addressType="relative" indices="1" length="1710" start="41670" /> … </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="constant">1</marginalDist> </featureDist>

</feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">12576 62803</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> <parcel parcelID="1"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="2"> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="2" tier="0" /> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="3" tier="1" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0 0" length="7065" start="62821" /> <atomTocEntry addressType="relative" indices="0 1" length="1512" start="69886" /> <atomTocEntry addressType="relative" indices="0 1" length="1687" start="71398" /> <atomTocEntry addressType="relative" indices="0 1" length="2013" start="73085" /> <atomTocEntry addressType="relative" indices="0 1" length="1821" start="75098" /> <atomTocEntry addressType="relative" indices="0 1" length="1823" start="76919" /> … <atomTocEntry addressType="relative" indices="0 1" length="2393" start="92736" /> <atomTocEntry addressType="relative" indices="0 1" length="2131" start="95129" /> <atomTocEntry addressType="relative" indices="1 0" length="6650" start="97260" /> <atomTocEntry addressType="relative" indices="1 1" length="2772" start="103910" /> <atomTocEntry addressType="relative" indices="1 1" length="2817" start="106682" /> <atomTocEntry addressType="relative" indices="1 1" length="1879" start="109499" /> <atomTocEntry addressType="relative" indices="1 1" length="1343" start="111378" /> <atomTocEntry addressType="relative" indices="1 1" length="1177" start="112721" /> … </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">2 5</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:framesPerSecond"> <components>

<refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">21132 77492</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData> <globalFeature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:foreman" numLayers="3"/> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">1 15 30</marginalDist> </featureDist> </globalFeature> </globalFeatureData> <globalCombAvarData> <globalCombAvar avar="ssm:avar:parcelTimeInSec"> <ssm:constant value="2”/> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:overallRateToLagrangian" numAxes="1"> <axisValues axis="0" grid="65994 107849 375672 533422 572100" /> <content> <marginalDist dims="1" dimToTierMap="0">0.0007 0.0007 0.0004 0.0002 0.0001 0 0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

The second adaptation step will remove the atoms that have violent rating more than 3. Here is the command line for the second step adaptation:

SSMAdaptEngine.exe -di SSMDescription_foreman_ves1.xml -ar SSMAdaptReq_ves2.xml -do SSMDescription_foreman_ves2.xml -bi foreman_ves1.mpg4 –bo foreman_ves2.mpg4 If adapted descriptor SSMDescription_foreman_ves2.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_foreman_ves1.xml -ar SSMAdaptReq_ves2.xml -bi foreman_ves1.mpg4 -bo foreman_ves2.mpg4

Here is the adapted resource XML for foreman after the first and second adaptation steps (SSMDescription_akiyo_ves2.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

… </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="constant">1</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">10974 118318</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> <parcel parcelID="4"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="10174" start="216080" /> <atomTocEntry addressType="relative" indices="1" length="2040" start="226254" /> <atomTocEntry addressType="relative" indices="1" length="2244" start="228294" /> <atomTocEntry addressType="relative" indices="1" length="2324" start="230538" /> <atomTocEntry addressType="relative" indices="1" length="2403" start="232862" /> <atomTocEntry addressType="relative" indices="1" length="2034" start="235265" /> … <atomTocEntry addressType="relative" indices="1" length="1756" start="251618" /> <atomTocEntry addressType="relative" indices="1" length="2123" start="253374" /> <atomTocEntry addressType="relative" indices="0" length="10958" start="255497" /> <atomTocEntry addressType="relative" indices="1" length="1335" start="266455" /> <atomTocEntry addressType="relative" indices="1" length="1722" start="267790" /> <atomTocEntry addressType="relative" indices="1" length="1833" start="269512" /> <atomTocEntry addressType="relative" indices="1" length="1890" start="271345" /> <atomTocEntry addressType="relative" indices="1" length="1670" start="273235" /> … </atomToc> </component> </componentData>

Use Case 2B In this use case, we will limit the violence rating as well as codesize. The outbound

constraints XML is very similar to the one for akiyo in Use Case 1B. Here is the outbound constraints XML (SSMAdaptReq_foreman1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="4" lowLimit="1"> <ssm:adapVar avar="ssm:avar:violenceRating" /> </limitConstraint> <limitConstraint highLimit="1" lowLimit="1"> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:adapVar avar="ssm:avar:parcelTimeInSec" /> <ssm:constant value="50000" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolIsGEZ" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There are two limit constraints in the outbound constraints XML. The first one will drop all the atoms that have violence rating greater than 4. The second limit constraint will ensure that the available bandwidth should be sufficient to transmit a parcel of video within the time it is supposed to be played for. In other words, bandwidth (50000 bytes/s) × parcelTimeInSec (2 s) will be greater or equal to the codesize. The limit constraints will apply to all parcels in the resource description file.

The adapted resource will become:

parcelID=”0”: 3 layers for the first tier.

parcelID=”1”: 1 layer for the first tier. 3 layers for the second tier.


parcelID=”3”: 1 layer for the first tier.



SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_foreman1.xml -do SSMDescription_foreman1.xml -bi foreman.mpg4 –bo foreman1.mpg4 If adapted descriptor SSMDescription_foreman1.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_foreman1.xml -bi foreman.mpg4 -bo foreman1.mpg4

Here is the adapted resource description XML (SSMDescription_foreman1.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="2" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="6445" start="18" /> <atomTocEntry addressType="relative" indices="1" length="1737" start="6463" /> <atomTocEntry addressType="relative" indices="1" length="1948" start="8200" /> <atomTocEntry addressType="relative" indices="1" length="1752" start="10148" /> <atomTocEntry addressType="relative" indices="1" length="1770" start="11900" /> <atomTocEntry addressType="relative" indices="1" length="1865" start="13670" /> … <atomTocEntry addressType="relative" indices="1" length="1596" start="28972" /> <atomTocEntry addressType="relative" indices="1" length="925" start="30568" /> <atomTocEntry addressType="relative" indices="0" length="7131" start="31493" /> <atomTocEntry addressType="relative" indices="1" length="1436" start="38624" /> <atomTocEntry addressType="relative" indices="1" length="1610" start="40060" /> <atomTocEntry addressType="relative" indices="1" length="1710" start="41670" /> <atomTocEntry addressType="relative" indices="1" length="1785" start="43380" /> <atomTocEntry addressType="relative" indices="1" length="1439" start="45165" /> … </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="constant">1</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">12576 62803</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData />

</parcel> <parcel parcelID="1"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="2"> <tierElemInfo exclusiveFlag="false" numLayers="1" origLayers="2" tier="0" /> <tierElemInfo exclusiveFlag="false" numLayers="3" origLayers="3" tier="1" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0 0" length="7065" start="62821" /> <atomTocEntry addressType="relative" indices="0 2" length="1079" start="69886" /> <atomTocEntry addressType="relative" indices="0 1" length="1512" start="70965" /> <atomTocEntry addressType="relative" indices="0 2" length="849" start="72477" /> <atomTocEntry addressType="relative" indices="0 1" length="1687" start="73326" /> <atomTocEntry addressType="relative" indices="0 2" length="800" start="75013" /> … </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">2</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="1" dims="1" distType="nonDecreasing">1 15 30</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0 1" dims="2" distType="nonDecreasing">7065 34439 48920</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> <parcel parcelID="2">

<globalFeature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:foreman" numLayers="3"/> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">1 15 30</marginalDist> </featureDist> </globalFeature> </globalFeatureData> <globalCombAvarData> <globalCombAvar avar="ssm:avar:parcelTimeInSec"> <ssm:constant value="2”/> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:overallRateToLagrangian" numAxes="1"> <axisValues axis="0" grid="65994 107849 375672 533422 572100" /> <content> <marginalDist dims="1" dimToTierMap="0">0.0007 0.0007 0.0004 0.0002 0.0001 0 0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

Use Case 2C In this use case, we will use the lookup table ssm:avar:overallRateToLagrangian to

look up the lagrangian mapping value for a given overall rate and minimize the (ssm:avar:codesize *lagrangian mapping value) + (30-ssm:avar:framesPerSecond).

Here is the outbound constraints XML (SSMAdaptReq_foreman2.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="30" lowLimit="15"> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> </limitConstraint> <optimizationConstraint optimize="minimize"> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> <ssm:constant value="30" /> <ssm:operation operator="subtract" /> <ssm:constant value="200000" /> <ssm:adapVar avar="ssm:avar:overallRateToLagrangian" />

<ssm:adapVar avar="ssm:avar:codesize" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> </optimizationConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There is one limit constraint and one optimization constraint in the outbound constraints XML file. The limit constraint will make sure that the adaptation variable ssm:avar:framesPerSecond lies between 15 and 30 inclusive. The optimization constraint will lookup a Langrangian parameter value using input argument overall rate to be 200000 bytes and minimize (ssm:avar:codesize × Lagrangian parameter) + (30 - ssm:avar:framesPerSecond)

The adapted resource will become:


parcelID=”1”: 1 layer for the first tier. 3 layers for the second tier. That is violence rating 5 is removed.





SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_foreman2.xml -do SSMDescription_foreman2.xml -bi foreman.mpg4 –bo foreman2.mpg4 If adapted descriptor SSMDescription_foreman2.xml is not required, skip the -do <> part: SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_foreman2.xml -bi foreman.mpg4 -bo foreman2.mpg4

Here is the adapted resource description XML (SSMDescription_foreman2.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="3" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="6445" start="18" /> <atomTocEntry addressType="relative" indices="1" length="1737" start="6463" /> <atomTocEntry addressType="relative" indices="2" length="674" start="8200" /> <atomTocEntry addressType="relative" indices="1" length="1948" start="8874" /> <atomTocEntry addressType="relative" indices="2" length="799" start="10822" /> …

</component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">2</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="1" dims="1" distType="nonDecreasing">1 15 30</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0 1" dims="2" distType="nonDecreasing">7065 34439 48920</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> <parcel parcelID="2"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="3" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="6565" start="137161" /> <atomTocEntry addressType="relative" indices="2" length="529" start="143726" /> <atomTocEntry addressType="relative" indices="1" length="1078" start="144255" /> <atomTocEntry addressType="relative" indices="2" length="385" start="145333" /> <atomTocEntry addressType="relative" indices="1" length="1588" start="145718" /> <atomTocEntry addressType="relative" indices="2" length="699" start="147306" /> … <atomTocEntry addressType="relative" indices="1" length="2026" start="182717" /> <atomTocEntry addressType="relative" indices="2" length="1091" start="184743" /> <atomTocEntry addressType="relative" indices="0" length="6162" start="185834" /> <atomTocEntry addressType="relative" indices="2" length="1048" start="191996" />

</feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">10974</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> <parcel parcelID="4"> <componentData> <component compID="ssm:comp:foreman"> <tierInfo numTiers="1"> <tierElemInfo exclusiveFlag="false" numLayers="3" origLayers="3" tier="0" /> </tierInfo> <atomToc addressType="absolute" start="0"> <atomTocEntry addressType="relative" indices="0" length="10174" start="258664" /> <atomTocEntry addressType="relative" indices="2" length="1400" start="268838" /> <atomTocEntry addressType="relative" indices="1" length="2040" start="270238" /> <atomTocEntry addressType="relative" indices="2" length="1716" start="272278" /> <atomTocEntry addressType="relative" indices="1" length="2244" start="273994" /> <atomTocEntry addressType="relative" indices="2" length="1807" start="276238" /> … <atomTocEntry addressType="relative" indices="1" length="2123" start="317730" /> <atomTocEntry addressType="relative" indices="2" length="1339" start="319853" /> <atomTocEntry addressType="relative" indices="0" length="10958" start="321192" /> <atomTocEntry addressType="relative" indices="2" length="295" start="332150" /> <atomTocEntry addressType="relative" indices="1" length="1335" start="332445" /> <atomTocEntry addressType="relative" indices="2" length="467" start="333780" /> <atomTocEntry addressType="relative" indices="1" length="1722" start="334247" /> <atomTocEntry addressType="relative" indices="2" length="595" start="335969" /> … </atomToc> </component> </componentData> <featureData> <feature avar="ssm:avar:violenceRating"> <components> <component compID="ssm:comp:foreman" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="constant">2</marginalDist> </featureDist> </feature> <feature avar="ssm:avar:codesize"> <components> <component compID="ssm:comp:foreman" /> </components>

<refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">21132 77492 117008</marginalDist> </featureDist> </feature> </featureData> <combAvarData /> <resourceEditData /> </parcel> </parcelData> <globalFeatureData> <globalFeature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:foreman" numLayers="3"/> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dimToTierMap="0" dims="1" distType="nonDecreasing">1 15 30</marginalDist> </featureDist> </globalFeature> </globalFeatureData> <globalCombAvarData> <globalCombAvar avar="ssm:avar:parcelTimeInSec"> <ssm:constant value="2”/> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:overallRateToLagrangian" numAxes="1"> <axisValues axis="0" grid="65994 107849 375672 533422 572100" /> <content> <marginalDist dimToTierMap="0" dims="1">0.0007 0.0007 0.0004 0.0002 0.0001 0 0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

3. Scalable MPEG-4 Visual Textures In the second use case, we consider MPEG-4 still textures, which is one of the coding

modes supported in MPEG-4. An image Rubik_Lena, a MPEG-4 visual still texture image bit-stream is used as the example resource. The image is supposed to be wrapped around a rubik cube for viewing. The objective is to create an adapted bit-stream from the original that decodes different parts of the image differently based on viewing angle and viewing distance. In particular, there are 9 squares on each of six faces of the cube, to yield a total of 54 different zones that are to be decoded differently. The viewing angle and distance together give a desired PSNR value that drives how the square is to be decoded.

The MPEG-4 still texture decoded at full resolution is shown in Figure 5. Its size is 1440x960. The distance and angle values that will be used for each zone (A: angle, D: distance), provided as inputs for the CE, is shown in the table below. Note that each zone is 160x160 in size.

A: -115 D: 17.5

A: 158 D: 19.6

A: 159 D: 19.7

A: 157 D: 19.9

A: -115 D: 18.6

A: 81 D: 17.4

A: 66 D: 17.5

A: 81 D: 18.5

A: -99 D: 17.2

A: -99 D: 18.4

A: 157 D: 19.7

A: 156 D: 19.6

A: -99 D: 19.7

A: 81 D: 18.4

A: 81 D: 18.3

A: -115 D: 17.4

A: -23 D: 17.2

A: 66 D: 18.4

A: -23 D: 16.9

A: -22 D: 17.1

A: -100 D: 17.4

A: 155 D: 19.7

A: 66 D: 17.3

A: 67 D: 17.1

A: 80 D: 19.4

A: 158 D: 19.8

A: -98 D: 19.6

A: -21 D: 17.2

A: -24 D: 17.3

A: 65 D: 20.1

A: 157 D: 19.5

A: -114 D: 19.7

A: -25 D: 17.2

A: -114 D: 17.3

A: -99 D: 18.6

A: -113 D: 19.5

A: 65 D: 18.8

A: -114 D: 18.5

A: 81 D: 17.5

A: -22 D: 17.3

A: -99 D: 18.5

A: -99 D: 17.3

A: 156 D: 19.8

A: 66 D: 18.6

A: -24 D: 17.1

A: -99 D: 19.8

A: -114 D: 18.4

A: 82 D: 17.6

A: -23 D: 17.4

A: 81 D: 19.5

A: 81 D: 19.6

A: -114 D: 20

A: 65 D: 19.9

A: 66 D: 19.7

Figure 5. Rubik_Lena decoded at full resolution

Although for viewing purposes, the image is divided into 54 zones of size 160x160 each, for the purpose of coding, the image is divided into smaller tiles of size 32x32. Further, each tile is encoded using 5 resolution levels, each with 12 bitplane layers, to yield a total of 60 quality layers. Each quality layer is obtained by scanning the 32x32 tiles row-wise from the top to the bottom.

For the purpose of decision making, 25 tiles that comprise a viewing zone are grouped together as a SSM parcel. Each parcel is organized as a 5x12 component, based on the (level, bitplane) combination. There are 54 parcels in the resource. The structure is shown in Figure 6.

We next present three use cases. In the first use case, the receiving terminal knows exactly what the desired PSNR is given a particular distance and angle, so in the outbound constraints XML file, the desired PSNR is represented as a constant. In the second use case, the content creator provides a lookup table for the desired PSNR; so in the outbound constraints file, distance and angle are provided as input arguments for the lookup table to find the desired PSNR. In the third use case, the content creator provides an adaptation variable function to calculate the desired PSNR; so in the outbound constraints file, distance and angle are provided as input arguments for the adaptation variable to get the desired PSNR.

Use Case 3A The following is the resource description XML for Rubik_Lena

(SSMDescription_VTC1.xml):

<?xml version="1.0" ?> <SSMDescription xsi:schemaLocation="SSMDescription SSMDescription.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMDescription" xmlns:ssm="SSMCommon"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:Rubik_Lena"> <tierInfo numTiers="2"> <tierElemInfo numLayers="5" origLayers="5" exclusiveFlag="false" tier="0" /> <tierElemInfo numLayers="12" origLayers="12" exclusiveFlag="false" tier="1" /> </tierInfo> <atomToc start="0" addressType="absolute">

Bitplane

1 2 3 4 5

1

2

3

4

Level

5

7 8 9 10 6 12 11

All 54 parcels

Figure 6. SSM structure for Rubik_Lena_1440x960

<globalCombAvarData /> <globalLUTAvarData /> <globalCreatorLimitConstraints /> <codecOffsetData /> </SSMDescription>

There are 54 parcels in the resource description. Each parcel maps to a viewing zone in the bitstream and has one component ssm:comp:Rubik_Lena. Each viewing zone consists of 25 32x32 tiles. Each component has two tiers. The first tier has 5 layers to indicate the levels, and the second tier has 12 layers to indicate the bitplanes. The indices in each atomTocEntry will indicate the information: (level, bitplane) in each atom. For a specific (level, bitplace) combination, there are 25 atoms in one parcel, except for the first parcel, where there are 24 atoms for each (level, bitplace) combination because the decoder requires that the first packet in each quality level to be always included for correct decoding.

There are two adaptation variables defined in the resource description. One is ssm:avar:decodedPSNR which will return the decoded PSNR given a (level, bitplane) adaptation decision. The other one is ssm:avar:computeTime which will return the compute time given (level, bitplane) adaptation decision.

In this use case, the receiving terminal knows exactly what the desired PSNR is given a particular distance and angle, so in the outbound constraints XML file, the desired PSNR is represented as a constant. Also, we assume a maximum decoding time of 0.4 sec uniformly allocated for each parcel.

Here is the outbound constraints XML (SSMAdaptReq_VTC1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0"> <ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="10.0" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> <parcel parcelID="1"> … </parcel> … …

<parcel parcelID="53"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0"> <ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="26.20" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

In the outbound constraint file, there are 54 parcels mapping to 54 parcels in the resource description file. Each parcel uses adaptation variable driven adaptation. There are two limit constraints. One is to limit the compute time to a maximum of 0.40. The other is to limit the value of (1.1 × desired PSNR)-ssm:avar:decodedPSNR to a positive number. In other words, this means that the decoded PSNR should not exceed 10% of the desired PSNR. If the desiredPSNR has been obtained, there is no reason to continue decoding. There is one optimization constraint which will maximize the value of ssm:avar:decodedPSNR.


SSMAdaptEngine.exe -di SSMDescription_VTC1.xml -ar SSMAdaptReq_VTC1.xml -bi Rubik_Lena_1440x960-MQ-TD.cmp –bo Rubik_Lena_1440x960-adapted1.cmp If the adapted descriptor is needed, it can be specified using -do <> option.

The following table shows for all the 54 zones, the desired PSNR, the decoded PSNR obtained from adaptation variable ssm:avar:decodedPSNR for the adaptation decisions made, and the adaptation decision (dr: desired PSNR, dc: decoded PSNR, lb: (level, bitplace) for adaptation decision):

A: -115 D: 17.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 158 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 159 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.9 dr:10.0 dc: 10.0 lb:(0,0)

A: -115 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.4 dr:17.96 dc: 19.50 lb:(1,3)

A: 66 D: 17.5 dr:26.25 dc:28.14 lb:(4,6)

A: 81 D: 18.5 dr:17.95 dc:19.50 lb:(1,3)

A: -99 D: 17.2 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 18.4 dr:17.95 dc: 19.50 lb:(1,3)

A: 81 D: 18.3 dr:17.95 dc: 19.50 lb:(1,3)

A: -115 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: -23 D: 17.2 dr:39.81 dc: 31.44 lb:(4,10)

A: 66 D: 18.4 dr:26.23 dc: 28.14 lb:(4,6)

A: -23 D: 16.9 dr:40.75 dc: 31.44 lb:(4,10)

A: -22 D: 17.1 dr:40.22 dc: 31.44 lb:(4,10)

A: -100 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 155 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 17.3 dr:26.26 dc: 28.14 lb:(4,6)

A: 67 D: 17.1 dr:25.93 dc: 28.14 lb:(4,6)

A: 80 D: 19.4 dr:18.82 dc: 19.95 lb:(1,12)

A: 158 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -98 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -21 D: 17.2 dr:40.00 dc: 31.44 lb:(4,10)

A: -24 D: 17.3 dr:39.41 dc: 31.44 lb:(4,10)

A: 65 D: 20.1 dr:26.48 dc: 28.14 lb:(4,6)

A: 157 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: -25 D: 17.2 dr:39.62 dc: 31.44 lb:(4,10)

A: -114 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -113 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 18.8 dr:26.55 dc: 28.14 lb:(4,6)

A: -114 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.5 dr:17.96 dc: 19.50 lb:(1,3)

A: -22 D: 17.3 dr:39.59 dc: 31.44 lb:(4,10)

A: -99 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 18.6 dr:26.23 dc: 28.14 lb:(4,6)

A: -24 D: 17.1 dr:40.03 dc: 31.44 lb:(4,10)

A: -99 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 82 D: 17.6 dr:17.07 dc: 17.73 lb:(4,1)

A: -23 D: 17.4 dr:39.18 dc: 31.44 lb:(4,10)

A: 81 D: 19.5 dr:17.94 dc: 19.50 lb:(1,3)

A: 81 D: 19.6 dr:17.94 dc: 19.50 lb:(1,3)

A: -114 D: 20 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 19.9 dr:26.53 dc: 28.14 lb:(4,6)

A: 66 D: 19.7 dr:26.20 dc: 28.14 lb:(4,6)

The decoded bitstream obtained by running the decoder mq-decoder.exe (using command line: MQ-decoder.exe -vtc Rubik_Lena_1440x960-adapted1.cmp out-adapted1.yuv 1440 960 5 12 ) is shown in Figure 7.

Use Case 3B

Figure 7. Decoded bit-stream for Use case 3A

In this use case, we use a lookup table for the desired PSNR, so in the outbound constraint file, distance and angle are provided as input arguments for the lookup table to find the desired PSNR. This is useful in situations where the content creator wants to control the mapping between viewing (distance, angle) to desired PSNR.

The resource description XML file (SSMDescription_VTC2.xml) used in this use case is identical to the one used in the first use case (SSMDescription_VTC1.xml) except one additional global combination variable and one additional lookup table defined as follow:

<globalCombAvarData> <globalCombAvar avar="ssm:avar:desiredPSNRLUT" numArguments="2"> <ssm:argument number="1" /> <ssm:operation operator="magnitude" /> <ssm:argument number="0" /> <ssm:adapVar avar="ssm:avar:DistAngToDesiredPSNRLUT" /> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:DistAngToDesiredPSNRLUT" numAxes="2"> <axisValues axis="0" grid="1 5 10 15 20 30 50 100" /> <axisValues axis="1" grid="0 15 30 45 60 75 90" /> <content> <marginalDist dims="2" dimToTierMap="0 1">55.64 55.64 50.34 48.55 39.12 29.35 25.72 10.0 10.0 55.64 55.64 50.34 48.55 39.12 29.35 25.72 10.0 10.0 54.98 54.98 49.87 47.03 38.67 29.04 24.76 10.0 10.0 52.56 52.56 49.1 46.46 38.05 28.97 24.15 10.0 10.0 50.23 50.23 47.82 45.68 37.12 28.32 23.29 10.0 10.0 31.68 31.68 31.34 30.83 27.78 28.17 23.23 10.0 10.0 27.89 27.89 26.47 26.13 25.64 22.45 20.25 10.0 10.0 24.51 24.51 23.12 22.58 21.95 18.74 17.65 10.0 10.0 20.35 20.35 20.14 19.23 18.26 16.41 15.68 10.0 10.0 20.35 20.35 20.14 19.23 18.26 16.41 15.68 10.0 10.0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData>

The global combination variable ssm:avar:desiredPSNRLUT takes two arguments. Argument-0 is the distance, and argument-1 is the angle. If the angle is less than 0, we will take the magnitude of it. The distance value and angle magnitude will be used as the input arguments for the lookup table ssm:avar:DistAngToDesiredPSNTLUT to get the desired PSNR.

This lookup table ssm:avar:DistAngToDesiredPSNTLUT has two axes. The first axis has 8 grid values, and is used by the input distance argument value. The second axis has 7 grid values, and is used by the input angle magnitude argument value. The lookup table uses the default linear method to get the desired PSNR.

The following is the outbound constraints XML (SSMAdaptReq_VTC2.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0">

<ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="-115" /> <ssm:constant value="17.5" /> <ssm:adapVar avar="ssm:avar:desiredPSNRLUT" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> <parcel parcelID="1"> … </parcel> … … <parcel parcelID="53"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0"> <ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="66" /> <ssm:constant value="19.7" /> <ssm:adapVar avar="ssm:avar:desiredPSNRLUT" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

The outbound constraint XML is similar to the one in the first use case (SSMAdaptReq_VTC1.xml) except the way we get the desired PSNR. In this use case, we get the desired PSNR by passing the distance and angle values as input argument to the adaptation variable ssm:avar:desiredPSNTLUT which will in turn use the lookup table ssm:avar:DistAngToDesiredPSNTLUT.


SSMAdaptEngine.exe -di SSMDescription_VTC2.xml -ar SSMAdaptReq_VTC2.xml -bi Rubik_Lena_1440x960-MQ-TD.cmp –bo Rubik_Lena_1440x960-adapted2.cmp If the adapted descriptor is needed, it can be specified using -do <> option

The following table shows the desiredPSNR obtained from adaptation variable ssm:avar:desiredPSNRLUT, the decoded PSNR obtained from adaptation variable ssm:avar:decodedPSNR for the adaptation decisions made, and the adaptation decision (dr: desired PSNR, dc: decoded PSNR, lb: (level, bitplace) for adaptation decision):

A: -115 D: 17.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 158 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 159 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.9 dr:10.0 dc: 10.0 lb:(0,0)

A: -115 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.4 dr:17.96 dc: 19.50 lb:(1,3)

A: 66 D: 17.5 dr:26.25 dc:28.14 lb:(4,6)

A: 81 D: 18.5 dr:17.95 dc:19.50 lb:(1,3)

A: -99 D: 17.2 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 18.4 dr:17.95 dc: 19.50 lb:(1,3)

A: 81 D: 18.3 dr:17.95 dc: 19.50 lb:(1,3)

A: -115 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: -23 D: 17.2 dr:39.81 dc: 31.44 lb:(4,10)

A: 66 D: 18.4 dr:26.23 dc: 28.14 lb:(4,6)

A: -23 D: 16.9 dr:40.75 dc: 31.44 lb:(4,10)

A: -22 D: 17.1 dr:40.22 dc: 31.44 lb:(4,10)

A: -100 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 155 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 17.3 dr:26.26 dc: 28.14 lb:(4,6)

A: 67 D: 17.1 dr:25.93 dc: 28.14 lb:(4,6)

A: 80 D: 19.4 dr:18.82 dc: 19.95 lb:(1,12)

A: 158 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -98 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -21 D: 17.2 dr:40.00 dc: 31.44 lb:(4,10)

A: -24 D: 17.3 dr:39.41 dc: 31.44 lb:(4,10)

A: 65 D: 20.1 dr:26.48 dc: 28.14 lb:(4,6)

A: 157 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: -25 D: 17.2 dr:39.62 dc: 31.44 lb:(4,10)

A: -114 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -113 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 18.8 dr:26.55 dc: 28.14 lb:(4,6)

A: -114 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.5 dr:17.96 dc: 19.50 lb:(1,3)

A: -22 D: 17.3 dr:39.59 dc: 31.44 lb:(4,10)

A: -99 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 18.6 dr:26.23 dc: 28.14 lb:(4,6)

A: -24 D: 17.1 dr:40.03 dc: 31.44 lb:(4,10)

A: -99 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 82 D: 17.6 dr:17.07 dc: 17.73 lb:(4,1)

A: -23 D: 17.4 dr:39.18 dc: 31.44 lb:(4,10)

A: 81 D: 19.5 dr:17.94 dc: 19.50 lb:(1,3)

A: 81 D: 19.6 dr:17.94 dc: 19.50 lb:(1,3)

A: -114 D: 20 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 19.9 dr:26.53 dc: 28.14 lb:(4,6)

A: 66 D: 19.7 dr:26.20 dc: 28.14 lb:(4,6)


Use Case 3C In this use case, we use a combination adaptation variable used as a function to

calculate the desired PSNR. So, in the outbound constraints XML, distances and angles are provided as input arguments for the adaptation variable to get the desired PSNR.

The function used is obtained by modeling the desired PSNR using nne-parameters (p0, p1, p2, p3, p4, p5, p6, p7, p8) as follows:

desiredPSNR(D, A) = p0 + p1.D1/4 + p2.D

1/2 + p3.D3/4 + p4.A + p5.A. D1/4 + p6.A.D1/2 +

p7.A.D3/4 + p8.A2,

where A is the angle in degrees in [0,90] and D is the distance in [0, ��parameters pi are obtained by a least-squares fit.

The resource description XML file (SSMDescription_VTC3.xml) used in this use case is identical to the one used in the first use case (SSMDescription_VTC1.xml) except two additional global combination defined as follow:

<globalCombAvarData> <globalCombAvar avar="ssm:avar:DistAngToDesiredPSNRFn" numArguments="4"> <ssm:constant value="-60.2376513" /> <ssm:argument number="0" /> <ssm:constant value="217.199868" /> <ssm:operation operator="multiply" /> <ssm:argument number="1" /> <ssm:constant value="-120.8535486" /> <ssm:operation operator="multiply" /> <ssm:argument number="2" /> <ssm:constant value="18.9931968" /> <ssm:operation operator="multiply" /> <ssm:argument number="3" /> <ssm:constant value="1.104080" /> <ssm:operation operator="multiply" /> <ssm:argument number="0" /> <ssm:argument number="3" /> <ssm:constant value="-2.48911506" /> <ssm:operation operator="multiply" /> <ssm:operation operator="multiply" /> <ssm:argument number="1" /> <ssm:argument number="3" /> <ssm:constant value="1.39537781" /> <ssm:operation operator="multiply" /> <ssm:operation operator="multiply" /> <ssm:argument number="2" /> <ssm:argument number="3" /> <ssm:constant value="-0.22203052" /> <ssm:operation operator="multiply" /> <ssm:operation operator="multiply" /> <ssm:argument number="3" /> <ssm:operation operator="sqr" /> <ssm:constant value="-0.00304881" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> <ssm:operation operator="add" /> </globalCombAvar> <globalCombAvar avar="ssm:avar:desiredPSNRFn" numArguments="2"> <ssm:constant value="10" /> <ssm:argument number="1" /> <ssm:operation operator="magnitude" /> <ssm:argument number="0" />

<ssm:operation operator="sqrt" /> <ssm:operation operator="sqrt" /> <ssm:argument number="0" /> <ssm:operation operator="sqrt" /> <ssm:operation operator="multiply" /> <ssm:argument number="0" /> <ssm:operation operator="sqrt" /> <ssm:argument number="0" /> <ssm:operation operator="sqrt" /> <ssm:operation operator="sqrt" /> <ssm:adapVar avar="ssm:avar:DistAngToDesiredPSNRFn" /> <ssm:argument number="1" /> <ssm:operation operator="magnitude" /> <ssm:constant value="-90" /> <ssm:operation operator="add" /> <ssm:operation operator="boolIsLEZ" /> <ssm:operation operator="selector" /> </globalCombAvar> </globalCombAvarData>

The global combination variable ssm:avar:desiredPSNRFn takes two arguments. Argument-0 is the distance (D), and argument-1 is the angle (A). If the magnitude of the input angle value is greater than 90, we will return 10 as the desired PSNR value. Otherwise, we then use these two argument values to generate four new computed values (D1/4, D1/2, D3/4, A) and use them as input arguments to another combination variable ssm:avar:DistAngToDesiredPSNRFn to get the desired PSNR.

The following is the outbound constraints XML (SSMAdaptReq_VTC3.xml):

Figure 8. Decoded bit-stream for use case 3B

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0"> <ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="-115" /> <ssm:constant value="17.5" /> <ssm:adapVar avar="ssm:avar:desiredPSNRFn" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> <parcel parcelID="1"> … </parcel> … … <parcel parcelID="53"> <adapVarDriven> <limitConstraint highLimit="0.40" lowLimit="0"> <ssm:adapVar avar="ssm:avar:computeTime" /> </limitConstraint> <limitConstraint highLimit="10000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> <ssm:constant value="66" /> <ssm:constant value="19.7" /> <ssm:adapVar avar="ssm:avar:desiredPSNRFn" /> <ssm:constant value="1.1" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:decodedPSNR" /> </optimizationConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

The outbound constraint XML is similar to the one in the second use case (SSMAdaptReq_VTC2.xml) except the way we get the desired PSNR. In this use case, we get the desired PSNR by passing the distance and angle values as input argument to the adaptation variable ssm:avar:desiredPSNTFn.


SSMAdaptEngine.exe -di SSMDescription_VTC3.xml -ar SSMAdaptReq_VTC3.xml -bi Rubik_Lena_1440x960-MQ-TD.cmp –bo Rubik_Lena_1440x960-adapted3.cmp If the adapted descriptor is needed, it can be specified using -do <> option

The following table shows the desiredPSNR obtained from the adaptation variable ssm:avar:desiredPSNRFn, the decoded PSNR obtained from adaptation variable ssm:avar:decodedPSNR for the adaptation decisions made, and the adaptation decision (dr: desired PSNR, dc: decoded PSNR, lb: (level, bitplace) for adaptation decision):

A: -115 D: 17.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 158 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 159 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.9 dr:10.0 dc: 10.0 lb:(0,0)

A: -115 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.4 dr:16.95 dc: 17.73 lb:(4,1)

A: 66 D: 17.5 dr:24.41 dc:26.53 lb:(3,12)

A: 81 D: 18.5 dr:16.83 dc:17.73 lb:(4,1)

A: -99 D: 17.2 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 157 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 18.4 dr:16.84 dc: 17.73 lb:(4,1)

A: 81 D: 18.3 dr:16.86 dc: 17.73 lb:(4,1)

A: -115 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: -23 D: 17.2 dr:38.45 dc: 31.44 lb:(4,10)

A: 66 D: 18.4 dr:24.14 dc: 26.53 lb:(3,12)

A: -23 D: 16.9 dr:38.71 dc: 31.44 lb:(4,10)

A: -22 D: 17.1 dr:38.73 dc: 31.44 lb:(4,10)

A: -100 D: 17.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 155 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 17.3 dr:24.47 dc: 26.53 lb:(3,12)

A: 67 D: 17.1 dr:24.01 dc: 26.41 lb:(3,8)

A: 80 D: 19.4 dr:17.25 dc: 18.82 lb:(1,2)

A: 158 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -98 D: 19.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -21 D: 17.2 dr:38.83 dc: 31.44 lb:(4,10)

A: -24 D: 17.3 dr:38.17 dc: 31.44 lb:(4,10)

A: 65 D: 20.1 dr:24.07 dc: 26.48 lb:(3,9)

A: 157 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 19.7 dr:10.0 dc: 10.0 lb:(0,0)

A: -25 D: 17.2 dr:38.05 dc: 31.44 lb:(4,10)

A: -114 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 18.6 dr:10.0 dc: 10.0 lb:(0,0)

A: -113 D: 19.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 18.8 dr:24.46 dc:26.53 lb:(3,12)

A: -114 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: 81 D: 17.5 dr:16.94 dc: 17.73 lb:(4,1)

A: -22 D: 17.3 dr:38.56 dc: 31.44 lb:(4,10)

A: -99 D: 18.5 dr:10.0 dc: 10.0 lb:(0,0)

A: -99 D: 17.3 dr:10.0 dc: 10.0 lb:(0,0)

A: 156 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: 66 D: 18.6 dr:24.08 dc: 26.48 lb:(3,9)

A: -24 D: 17.1 dr:38.34 dc: 31.44 lb:(4,10)

A: -99 D: 19.8 dr:10.0 dc: 10.0 lb:(0,0)

A: -114 D: 18.4 dr:10.0 dc: 10.0 lb:(0,0)

A: 82 D: 17.6 dr:16.37 dc: 17.73 lb:(4,1)

A: -23 D: 17.4 dr:38.28 dc: 31.44 lb:(4,10)

A: 81 D: 19.5 dr:16.73 dc: 17.73 lb:(4,1)

A: 81 D: 19.6 dr:16.72 dc: 17.73 lb:(4,1)

A: -114 D: 20 dr:10.0 dc: 10.0 lb:(0,0)

A: 65 D: 19.9 dr:24.13 dc: 26.53 lb:(3,12)

A: 66 D: 19.7 dr:23.77 dc: 25.78 lb:(3,6)


4. MC-EZBC The third use case involves MC-EZBC, which is a fully scalable video codec from

RPI that has been proposed for a new scalable video standard in MPEG-4. We have explored the use of SSM for adaptation of a MC-EZBC bit-stream. Each GOP in the sample bit-stream is 16 frames long, and coded in a fully scalable manner using a 3-tier scalability structure of dimensions 5x6x5. The first tier corresponds to five temporal layers, the second tier corresponds to six spatial resolution layers within each temporal layer, and the third tier corresponds to five quality layers within each spatial layer. Each GOP has a header that contains information as to how many of temporal and spatial layers are included. Further, each spatial layer is preceded by a length field. When an adaptation is made on a GOP, not only should appropriate parts of the bit-stream be removed, but the information in the header and length fields must be updated appropriately to enable correct decoding.

The following is the resource description XML for the MC-EZBC example foreman.bit (SSMDescription_foreman.xml):

<?xml version="1.0" ?> <SSMDescription xsi:schemaLocation="SSMDescription SSMDescription.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMDescription" xmlns:ssm="SSMCommon"> <parcelData> <parcel parcelID="0"> <componentData>

Figure 9. Decoded bit-stream for use case 3C

… <parcel parcelID="17"> … </parcel> </parcelData> <globalFeatureData> <globalFeature avar="ssm:avar:framesPerSecond"> <components> <component compID="ssm:comp:myGOP" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dims="1" dimToTierMap="0" distType="nonDecreasing">1.875 3.75 7.5 15 30</marginalDist> </featureDist> </globalFeature> <globalFeature avar="ssm:avar:diagResolution"> <components> <component compID="ssm:comp:myGOP" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dims="1" dimToTierMap="1" distType="nonDecreasing">14 28 57 114 227 455</marginalDist> </featureDist> </globalFeature> <globalFeature avar="ssm:avar:distortion"> <components> <component compID="ssm:comp:myGOP" /> </components> <refFeatureValue>1</refFeatureValue> <emptyFeatureDist>0</emptyFeatureDist> <featureDist emptyComponentTiers=""> <marginalDist dims="1" dimToTierMap="2" distType="nonDecreasing">64 32 16 8 4 </marginalDist> </featureDist> </globalFeature> </globalFeatureData> <globalCombAvarData> <globalCombAvar avar="ssm:avar:parcelTimeInSec"> <ssm:constant value="0.5333" /> </globalCombAvar> </globalCombAvarData> <globalLUTAvarData> <globalLUTAvar avar="ssm:avar:avgBWToLagrangianMap" numAxes="1"> <axisValues axis="0" grid="0 40000 65000 77000 140000 300000 750000" /> <content> <marginalDist dims="1" dimToTierMap="0">1 0.1 0.00008 0.00004 0.00003 0.00002 0.00001 0.0 0.0</marginalDist> </content> </globalLUTAvar> </globalLUTAvarData> <globalCreatorLimitConstraints /> <codecOffsetData>

<offsetReference addressType="absolute" start="170"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="169" /> </offsetReference> <offsetReference addressType="absolute" start="343"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="661" /> </offsetReference> <offsetReference addressType="absolute" start="1008"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="1965" /> </offsetReference> <offsetReference addressType="absolute" start="2977"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="5996" /> </offsetReference> <offsetReference addressType="absolute" start="8977"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="17629" /> </offsetReference> <offsetReference addressType="absolute" start="26610"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="43918" /> </offsetReference> <offsetReference addressType="absolute" start="72863"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="91" /> </offsetReference> … <offsetReference addressType="absolute" start="7101246"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="30360" /> </offsetReference> <offsetReference addressType="absolute" start="7131610"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="31" bitPos="1" signed="false" start="-4" value="87301" /> </offsetReference> </codecOffsetData> </SSMDescription>

In the SSM description of the resource bit-stream, each GOP is considered as a SSM parcel for adaptation purposes. Each parcel has a single component ssm:comp:myGOP with 5x6x5 layers in each. Each parcel has an adaptation variable ssm:avar:codesize defined to specify the codesize. Each parcel also has two resource edit elements. One is used to store the number of temporal layers on the GOP header of the bitstream, and the other one is used to store the number of spatial layers on the GOP header of the bitstream.

The first tier corresponds to five temporal layers. The value for frames per second could be obtained using the adaptation variable ssm:avar:framesPerSecond. If all 5 layers are included in the adapted resource, there will be 30 frames per second. If 4 layers are included in the adapted resource, there will be 15 frames per second, and so on.

The second tier corresponds to six spatial resolution layers within each temporal layer. The value for spatial resolution could be obtained using the adaptation variable ssm:avar:diagResolution. If all 6 layers are included in the adapted resource, the value of

spatial resolution would be 455. If 5 layers are included in the adapted resource, the value of spatial resolution would be 227, and so on.

The third tier corresponds to 5 quality layers within each spatial layer. The value of distortion could be obtained using the adaptation variable ssm:avar:distortion. If all 5 layers are included in the adapted resource, the value of distortion would be 4. If 6 layers are included in the adapted resource, the value of distortion would be 8, and so on.

There is one global combination variable ssm:avar: parcelTimeInSec defined. It defines the constant value 0.5333 second as the parcel time.

There is one global lookup table ssm:avar:avgBWToLagrangianMap defined. It is used to map average bandwidth over the duration of the video to an optimal Lagrangian parameter.

In the codec offset section, each offset reference points to the beginning of each spatial layer, and the offset entry within each offset reference points to the length field that precedes the spatial layer. The value of the offset entry would be the length. There are in total 18*5*6=240 offset reference within the codec offset section.

Use Case 4A This use case demonstrates adaptation where specific constraints on temporal

resolution, spatial resolution, and distortion are provided in the outbound constraints XML.

The following is the outbound constraints XML (SSMAdaptReq_mcezbc1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel> <adapVarDriven> <limitConstraint highLimit="15" lowLimit="0"> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> </limitConstraint> <limitConstraint highLimit="400" lowLimit="0"> <ssm:adapVar avar="ssm:avar:diagResolution" /> </limitConstraint> <limitConstraint highLimit="20" lowLimit="4"> <ssm:adapVar avar="ssm:avar:distortion" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

In this outbound constraints XML, there is one adaptation variable driven parcel defined and it will be used for all 18 parcels in the resource description XML. There are three limit constraints defined in the profile parcel. The first constraint limits the value of frames per second to be within 0 and 15. The second constraint limits the value of spatial resolution to be within 0 and 400. The third constraint limits the value of distortion to be within 4 and 20.


SSMAdaptEngine.exe -di SSMDescription_foreman.xml -ar SSMAdaptReq_mcezbc1.xml -bi foreman.bit –bo foreman_adapted1.bit If the adapted descriptor is needed, it can be specified using -do <> option

The adapted resource will have the structure of dimensions 4x5x5 after adaptation. The value of the frames per second would be 15 which is within the range 0 and 15. The value of spatial resolution would be 227 which is within the range 0 and 400. The value of distortion would be 4 which is within the range of 4 and 20.

Use Case 4B This use case demonstrates a scenario where the bandwidth available for every parcel

is limited, and at the same time, the spatial resolution must be within a specified range for the first parcel and be maintained the same as that chosen for the first for all subsequent parcels. The bandwidth constraint translates to a codesize requirement that the parcel codesize must not exceed bandwidth times the parcel time.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel> <adapVarDriven> <limitConstraint highLimit="500" lowLimit="300"> <ssm:adapVar avar="ssm:avar:diagResolution" /> </limitConstraint> <limitConstraint highLimit="1" lowLimit="1"> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:adapVar avar="ssm:avar:parcelTimeInSec" /> <ssm:constant value="360000" /> <ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolIsGEZ" /> </limitConstraint> </adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> <parcel> <adapVarDriven> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:diagResolution" /> <ssm:adapVar avar="ssm:avar:diagResolution" previous="true" /> <ssm:operation operator="subtract" /> </limitConstraint> <limitConstraint highLimit="1" lowLimit="1"> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:adapVar avar="ssm:avar:parcelTimeInSec" /> <ssm:constant value="360000" />

<ssm:operation operator="multiply" /> <ssm:operation operator="subtract" /> <ssm:operation operator="boolIsGEZ" /> </limitConstraint> </adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> </profile> </SSMAdaptReq>

There are two parcels in the outbound constraints XML. The first profile parcel will be applied to the first parcel in the resource description XML. The second profile parcel will be applied to the remaining parcels in the resource description XML.

In the first profile parcel, there are two limit constraints defined. The first constraint limits the spatial resolution to be within 300 and 500. The second constraint limits the codesize to be less or equal to (bandwidth * ssm:avar:parcelTimeInSec). We use constant 360000 as the bandwidth.

In the second profile parcel, there are also two limit constraints defined. The first constraint makes sure that the spatial resolution for this parcel is the same as the spatial resolution for the previous parcel. The second constraint limits the codesize to be less or equal to (bandwidth * ssm:avar:parcelTimeInSec). We use constant 360000 as the bandwidth.



Based on these limit constraints the adaptation engine makes decisions on adaptation points on a per GOP basis, with an implicit optimization constraint on symmetry of the adaptation. Each adaptation point is a triple (t, s, q), indicating the number of temporal (t), spatial (s) and quality (q) layers included, with a maximum of (5,6,5). The adaptation points for the 18 GOPs are as follows: (3,6,5), (3,6,5), (3,6,5), (3,6,5), (4,6,4), (5,6,4), (5,6,4), (5,6,4), (4,6,4), (3,6,5), (3,6,5), (3,6,5), (5,6,4), (4,6,4), (4,6,4), (3,6,5), (4,6,4), (3,6,5). Note that for each GOP the full spatial resolution with 6 layers, corresponding to a diagonal resolution of 455 pixels, are transmitted. However, the temporal and quality layers vary to meet the bandwidth constraint. The actual average bandwidth achieved by this adaptation is 333,973 bytes/sec. Note however that the constraint is strictly met for all GOPs.

Use Case 4C This use case demonstrates a scenario where as in the above use case, the spatial

resolution must be within a specified range for the first parcel, and be maintained the same as that chosen for the first for all subsequent parcels. Within this constraint, for all parcels the following is minimized: (log(ssm:avar:distortion) + ssm:avar:framesPerSecond�� ssm:avar:codesize�� The

first term may be considered to correspond to a measure of overall distortion, so that the

overall met�� XML specify an optimization constraint, which is minimization metric specified using stack expressions involving the necessary adaptation variables and two constants, �� ptation engine makes the optimal choice of number of temporal and SNR

layers to include in each parcel based on a minimization of this metric.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel> <adapVarDriven> <limitConstraint highLimit="500" lowLimit="200"> <ssm:adapVar avar="ssm:avar:diagResolution" /> </limitConstraint> <optimizationConstraint optimize="minimize"> <ssm:adapVar avar="ssm:avar:distortion" /> <ssm:operation operator="log" /> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> <ssm:operation operator="inverse" /> <ssm:constant value="12" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> <ssm:constant value="0.00002" /> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> </optimizationConstraint> </adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> <parcel> <adapVarDriven> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:diagResolution" /> <ssm:adapVar avar="ssm:avar:diagResolution" previous="true" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="minimize"> <ssm:adapVar avar="ssm:avar:distortion" /> <ssm:operation operator="log" /> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> <ssm:operation operator="inverse" /> <ssm:constant value="12" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> <ssm:constant value="0.00002" /> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" />

</optimizationConstraint> </adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> </profile> </SSMAdaptReq>


In both of the profile parcels, the limit constraints will ensure that the value of the spatial resolution for the first parcel is within 200 and 500, and subsequent parcels will have the same value of the spatial resolution as the first parcel.

The optimization constraint will minimize the value of:

(log(distortion)+12 / framesPerSecond) + 0.00002*codesize



For the Foreman sequence, wi�� ×10-5, the adaptation points chosen for the 18 GOPs are as follows: (3,5,5), (4,5,4), (4,5,4), (4,5,4), (4,5,4), (3,5,5), (5,5,4), (4,5,5), (3,5,5), (3,5,5), (4,5,4), (3,5,5), (4,5,4), (5,5,3), (5,5,3), (4,5,4), (5,5,4), (5,5,4). Note that the spatial resolution constraints ensure that 5 spatial resolution layers, one less than full resolution, are chosen for all GOPs. However, the temporal and quality layers are change depending on the minimization metric.

Use Case 4D The fourth use case is similar to the third one, except that the Lagrangian parameter

is obtained by interpolation on a look-up-table provided by the content creator in the resource descriptor XML to map average bandwidth over the duration of the video to an optimal Lagrangian parameter. The average bandwidth essentially provides the overall ��!�� "��!��#-up table is to be used to minimize the metric (log(ssm:avar:distortion) + ssm:avar:framesPerSecond�� ssm:avar:codesize for each parcel. Since the content

creator already knows the R-D characteristics of the entire sequence, he/she can provide this information in the look-up table ssm:avar:avgBWToLagrangianMap to enable an adaptation engine to find the R-D optimal adaptation for each parcel to yield an overall rate close to the desired. The bandwidth to Lagrangian map is typically sufficiently smooth, so that not much is lost by interpolation as opposed to the exact. Note that this solves a classic problem where the optimum Lagrangian parameter needed to obtain a desired rate is unknown at the time of encoding, but by making appropriate choices at transmission time in the network, it may still be possible to deliver a near optimal stream for the desired rate, to multiple users.

In addition, as in the previous two cases, the spatial resolution must be within a specified range for the first parcel, and be maintained the same as that chosen for the first for all subsequent parcels.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel> <adapVarDriven> <limitConstraint highLimit="500" lowLimit="200"> <ssm:adapVar avar="ssm:avar:diagResolution" /> </limitConstraint> <optimizationConstraint optimize="minimize"> <ssm:adapVar avar="ssm:avar:distortion" /> <ssm:operation operator="log" /> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> <ssm:operation operator="inverse" /> <ssm:constant value="12" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> <ssm:constant value="100000" /> <ssm:adapVar avar="ssm:avar:avgBWToLagrangianMap" /> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> </optimizationConstraint> </adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> <parcel> <adapVarDriven> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:diagResolution" /> <ssm:adapVar avar="ssm:avar:diagResolution" previous="true" /> <ssm:operation operator="subtract" /> </limitConstraint> <optimizationConstraint optimize="minimize"> <ssm:adapVar avar="ssm:avar:distortion" /> <ssm:operation operator="log" /> <ssm:adapVar avar="ssm:avar:framesPerSecond" /> <ssm:operation operator="inverse" /> <ssm:constant value="12" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> <ssm:constant value="100000" /> <ssm:adapVar avar="ssm:avar:avgBWToLagrangianMap" /> <ssm:adapVar avar="ssm:avar:codesize" /> <ssm:operation operator="multiply" /> <ssm:operation operator="add" /> </optimizationConstraint>

</adapVarDriven> <adapVarStore> <ssm:storedAdapVar avar="ssm:avar:diagResolution" /> </adapVarStore> </parcel> </profile> </SSMAdaptReq>


In both of the profile parcels, the limit constraints will ensure that the value of the spatial resolution for the first parcel is within 200 and 500, and subsequent parcels will have the same value of the spatial resolution as the first parcel.

The optimization constraint will minimize the value of: (log(distortion)+12 / framesPerSecond�� $�� #�!� ��ssm:avar:avgBWToLagrangianMap with 100000 as the input average bandwidth argument.



For the Foreman��%�� &�� "��''�'''��(��maps to a Lagrangian value 2.6×10-5 by interpolation, the adaptation points chosen for the 18 GOPs are as follows: (5,5,2), (5,5,3), (5,5,3), (5,5,3), (5,5,3), (5,5,2), (5,5,3), (5,5,3), (5,5,2), (3,5,5), (5,5,2), (3,5,5), (5,5,3), (5,5,3), (5,5,3), (5,5,3), (5,5,3), (5,5,3). Note that the spatial resolution constraints ensure that 5 spatial resolution layers, one less than full resolution, are chosen for all GOPs. However, the temporal and quality layers are change depending on the minimization metric. The average bandwidth obtained by this adaptation is 99,627 bytes/s, very close to the desired.

5. JPEG2000 The fourth use case involves JPEG2000 codestreams. JPEG2000 is a new, fully

scalable image coding standard, which allows immense flexibility in downward adapatation of an image, based on quality, resolution, color, regions of interest, and any combination thereof. A test bit-stream river.j2c is used for the experiment.

The following is the resource description XML for river.j2c (SSMDescription_river.xml):

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <SSMDescription xmlns="SSMDescription" xmlns:ssm="SSMCommon" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SSMDescription SSMDescription.xsd"> <parcelData> <parcel parcelID="0"> <componentData> <component compID="ssm:comp:river">

<offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="16" signed="false" start="0" value="11" /> </offsetReference>  <offsetReference addressType="absolute" start="119"> <offsetEntry addressType="relative" invalidPointerHandling="moveUp" length="32" signed="false" start="0" value="214734" /> </offsetReference> </codecOffsetData> <sequenceData>  <sequence startValue="0" stepValue="1"> <countField addressType="absolute" length="8" start="42" write="false" /> <countField addressType="relative" length="8" start="3" write="false" /> <countField addressType="relative" length="8" start="3" write="false" /> <writeField addressType="absolute" length="16" start="40" /> </sequence>  <sequence startValue="0" stepValue="1"> <countField addressType="absolute" length="16" start="131" /> <countField addressType="absolute" length="16" start="345" /> <countField addressType="absolute" length="16" start="556" /> <countField addressType="absolute" length="16" start="693" /> <countField addressType="absolute" length="16" start="1164" /> <countField addressType="absolute" length="16" start="1656" /> <countField addressType="absolute" length="16" start="2125" /> <countField addressType="absolute" length="16" start="2623" /> <countField addressType="absolute" length="16" start="3156" /> <countField addressType="absolute" length="16" start="3687" /> … <countField addressType="absolute" length="16" start="214824" /> <countField addressType="absolute" length="16" start="214831" /> </sequence> </sequenceData> </SSMDescription>

The bit-stream is organized as a single parcel with a single component ssm:comp:river. There are 4 tiers in the component with 3x2x3x3 layers.

The first tier is used for quality layers. Relative quality values, increasing sequentially layer by layer (1, 2,3), is obtained from adaptation variable ssm:avar:layer.

The second tier is used for level/resolution layers. There are two resolution layers. The adaptation variable ssm:avar:level provides level values 1 or 2 for the two levels. Alternatively, the actual resolution information is obtained from adaptation variables ssm:avar:resolutionX and ssm:avar:resolutionY.

The third tier is used for color information. There are three color layers, corresposnding to three color image components. The adaptation variable ssm:avar:color indicates whether a color or grayscale representation is obtained. If all three layers in the color tier are included in the adapted outbound resource, ssm:avar:color will be 1, otherwise it will be 0. There is also one global creator limit constraint to disallow the case where two layers in the color tier are included in the adapted outbound resource.

The fourth tier is used for region of interest. There are 3 layers in the tier. The first layer comprises region A, which is a rectangular portion of the full image depicting a castle. The second layer comprises region B, which is a rectangular portion of the full image, depicting a boat. The third layer comprises the remaining regions that are neither in region A nor in B. This tier is designated exclusive of type lastAll, so that, if the last layer of this tier is selected, the entire image, will be included in the adapted version. We define adaptation variables ssm:avar:region for selection of the region: A, B or all, denoted respectively 1, 2 or 3.

There is also a adaptation variable ssm:avar:codesize available for different adaptation purposes.

There are some satellite atoms that are shared by all regions. One or more coordinates of these atoms are made –1 to indicate they are don’t care coordinates. For any regular atom that is included in an adapted version, all its available satellite atoms (obtained by converting one or more coordinates into –1), are also included.

In the resource edit section, it is necessary to specify required post-adaptation updates on certain fields in the bitstream header, including Xsiz, Ysiz, XTsiz, YTsiz, numLayers, mct, and nDecompLevels.

The codec offset section is used to update certain length fields in the bitstream header including the length of SIZ, the length of COD, the length of QCD, and the length of the tile(Psot).

In the sequence data section, two sequences are defined. The first sequence is meant to update the number of color components field (Csiz) in the header, while the second sequence, which is more important, updates the sequential NSOP counters in each packet.

Use Case 5A This use case will extract a color version of region A (castle)with reduced quality

and/or resolution.

The following is the outbound constraints XML (SSMAdaptReq_river1.xml):

<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0" requestType="SSM"> <adapVarDriven> <limitConstraint highLimit="2" lowLimit="0"> <ssm:adapVar avar="ssm:avar:layer" /> </limitConstraint> <limitConstraint highLimit="200" lowLimit="0"> <ssm:adapVar avar="ssm:avar:resolutionX" /> <ssm:adapVar avar="ssm:avar:resolutionY" /> <ssm:operation operator="maximum" /> </limitConstraint> <limitConstraint highLimit="1" lowLimit="0"> <ssm:adapVar avar="ssm:avar:color" /> </limitConstraint>

<limitConstraint highLimit="1" lowLimit="1"> <ssm:adapVar avar="ssm:avar:region" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There is a single profile in the outbound constraints XML, with a single parcel. Four limit constraints defined. The first constraint specifies a high limit on quality layers, the second constraint limits the maximum value of resolutionX and resolutionY, the third constraint limits the color, and the fourth constraint designates the region of interest to be extracted.


SSMAdaptEngine.exe -di SSMDescription_river.xml -ar SSMAdaptReq_river1.xml -bi river.j2c –bo river_adapted1.j2c If the adapted descriptor is needed, it can be specified using -do <> option

The adapted resource corresponds to an adaptation point of (2,1,3,1).

Use Case 5B This use case will extract a grayscale version of region B (boat) with reduced quality.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0" requestType="SSM"> <adapVarDriven> <limitConstraint highLimit="1" lowLimit="0"> <ssm:adapVar avar="ssm:avar:layer" /> </limitConstraint> <limitConstraint highLimit="400" lowLimit="0"> <ssm:adapVar avar="ssm:avar:resolutionX" /> <ssm:adapVar avar="ssm:avar:resolutionY" /> <ssm:operation operator="maximum" /> </limitConstraint> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:color" /> </limitConstraint> <limitConstraint highLimit="2" lowLimit="2"> <ssm:adapVar avar="ssm:avar:region" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There is a single profile in the outbound constraints XML, with a single parcel. Four limit constraints defined. The first constraint limits the number of layers, the second

constraint limits the maximum value of resolution X and resolution Y, the third constraint limits the color, and the fourth constraint designates the region of interest to be extracted.



The adapted resource corresponds to an adaptation point of (1,2,1,2).

Use Case 5C This use case will extract a grayscale version of the full image with reduced

resolution.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0" requestType="SSM"> <adapVarDriven> <limitConstraint highLimit="2" lowLimit="0"> <ssm:adapVar avar="ssm:avar:layer" /> </limitConstraint> <limitConstraint highLimit="400" lowLimit="0"> <ssm:adapVar avar="ssm:avar:resolutionX" /> <ssm:adapVar avar="ssm:avar:resolutionY" /> <ssm:operation operator="maximum" /> </limitConstraint> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:color" /> </limitConstraint> <limitConstraint highLimit="3" lowLimit="3"> <ssm:adapVar avar="ssm:avar:region" /> </limitConstraint> </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There is a single profile in the outbound constraints XML, with a single parcel. Four limit constraints defined. The first constraint limits the number of quality layers, the second constraint limits the maximum value of resolution X and resolution Y, the third constraint limits the color, and the fourth constraint designates the region of interest to be extracted.



The adapted resource corresponds to an adaptation point of (2,1,1,3). Because the fourth tier (region) is exclusive and with type set to lastAll, the adapted resource will have all layers for the region tier included.

Use Case 5D This use case will extract a grayscale version of the full image with the number of

quality layers to be at least 2, codesize no more then 100000 bytes, and try to maximize the resolution.


<?xml version="1.0" ?> <SSMAdaptReq adaptType="terminal" xsi:schemaLocation="SSMAdaptReq SSMAdaptReq.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="SSMAdaptReq" xmlns:ssm="SSMCommon"> <profile> <parcel parcelID="0" requestType="SSM"> <adapVarDriven> <limitConstraint highLimit="3" lowLimit="2"> <ssm:adapVar avar="ssm:avar:layer" /> </limitConstraint> <limitConstraint highLimit="100000" lowLimit="0"> <ssm:adapVar avar="ssm:avar:codesize" /> </limitConstraint> <limitConstraint highLimit="0" lowLimit="0"> <ssm:adapVar avar="ssm:avar:color" /> </limitConstraint> <limitConstraint highLimit="3" lowLimit="3"> <ssm:adapVar avar="ssm:avar:region" /> </limitConstraint> <optimizationConstraint optimize="maximize"> <ssm:adapVar avar="ssm:avar:resolutionX" /> <ssm:adapVar avar="ssm:avar:resolutionY" /> <ssm:operation operator="maximum" /> </ optimizationConstraint > </adapVarDriven> </parcel> </profile> </SSMAdaptReq>

There is a single profile in the outbound constraints XML, with a single parcel. Four limit constraints and one optimization constraint defined. The first limit constraint limits the number of quality layers, the second limit constraint limits the codesize, the third limit constraint limits the color, and the fourth limit constraint designates the region of interest to be extracted. The optimization constraint tries to maximize the resolution.



The adapted resource corresponds to an adaptation point of (3,1,1,3). Because the fourth tier (region) is exclusive and with type set to lastAll, the adapted resource will have all layers for the region tier included.

6. Conclusion In this report we have presented comprehensive results showing how the SSM

framework can be used to accomplish fully format-agnostic adaptation of real bit-streams. Both existing standardized bit-streams as well as future non-standard bit-streams have been considered. The adaptation engine itself is regarded as an invariant black box in all cases, while the resource description metadata and the constraints drive the adaptation process appropriately for each use case. Since the core adaptation engine does not need to change, only the driving XML inputs to it, the engine can be deployed in servers today, and it would continue to adapt for all present and future scalable bit-streams.

7. References [1] Debargha Mukherjee, Amir Said, “Structured Content Independent Scalable Meta-

formats (SCISM) for Media Type Agnostic Transcoding: Response to CfP on DIA / MPEG-21,” ISO/IEC JTC1/SC29/WG11, MPEG2002/M8689.

[2] Debargha Mukherjee et. al. “Structured scalable meta-formats version 1.0 for content agnostic digital item adaptation,” ISO/IEC JTC1/SC29/WG11 MPEG2002/M9131, Dec 2002.

[3] Debargha Mukherjee et. al. “Proposals for end-to-end digital item adaptation using structured scalable meta-formats,” ISO/IEC JTC1/SC29/WG11 MPEG2002/M8898, Oct 2002.

[4] D. Mukherjee, A. Said, “Structured scalable meta-formats (SSM) version 2.0 for content agnostic digital item adaptation – Principles and Complete Syntax,” Hewlett Packard Technical Report, March 2003.

[5] D. Mukherjee, G. Kuo, A. Said, “Structured scalable meta-formats (SSM) for digital item adaptation,” Hewlett Packard Technical Report, HPL-2002-326, Nov 2002.

[6] (MPEG-4) Information technology – Coding of audio-visual objects – Part 2: Visual, ISO/IEC 14496-2-2001.

[7] (MPEG-4) Information technology – Coding of audio-visual objects – Part 3: Audio, ISO/IEC 14496-3-2001.

[8] J. Woods, P. Chen, Shih-ta Hsiang, D. Mukherjee, G. Kuo, A. Said, “Fully scalable MC-EZBC in the Structured Scalable Meta-formats (SSM) framework,” ISO/IEC JTC1/SC29/WG11 MPEG2002/M9290, Dec 2002.

[9] Shih-Ta Hsiang and John W. Woods, "Embedded video coding using motion compensated 3-D subband/wavelet filter bank", Packet Video Workshop, Sardinia, Italy, May 2000.

[10] Shih-Ta Hsiang and John W. Woods, "Embedded video coding using invertible motion compensated 3-D subband/wavelet filter bank," Signal Processing: Image

Communications, Vol, pp. 705-724, May 2001.

[11] Shih-Ta Hsiang, “Highly Scalable Subband/Wavelet Image and Video Coding,” Ph.D. Thesis, Rensselaer Polytechnic Institute, Troy, New York, May 2002.

[12] J. W. Woods and Peisong Chen, “Improved MC-EZBC with quarter-pixel motion vectors,” ISO/IEC JTC1/SC29/WG11, MPEG2002/M8366.

[13] David S. Taubman and M. W. Marcellin, “JPEG2000: Image Compression Fundamentals, Standards and Practice,” Kluwer Academic Publishers, 2002.

adaptation use case for structured scalable meta-formats (ssm) … · 2018-09-13 · elementary...

Documents