lncs 4266 - real-time watermark embedding for high …hklee.kaist.ac.kr/publications/2006 iwsec(with...

Real-Time Watermark Embedding for HighResolution Video Watermarking

In-Koo Kang1, Dong-Hyuck Im1, Young-Ho Suh2, and Heung-Kyu Lee1

1 Department of EECS, Korea Advanced Institute of Science and Technology,Guseong-dong, Yuseong-Gu, Deajeon, Republic of Korea

{ikkang, iammoni, hklee}@mmc.kaist.ac.kr2 Digital Content Research Division, Electronics and Telecommunications Research

Institute, Gajeong-dong, Yuseong-gu, Daejeon, Republic of [email protected]

Abstract. This paper addresses implementation issues for real-time wa-termark embedding scheme of High Definition(HD) resolution videos onpersonal computers. In most watermark applications, an embedding pro-cedure should be built at low costs and at the same time the embeddedwatermarks should have robustness against signal and image process-ing as well as malicious attacks. This paper provides some performanceoptimization guidelines and a simplified Human Visual System (HVS)method for fast and robust watermark embedding. This work demon-strates a real-time watermark embedding process including HD MPEG-2 video decoding, watermark embedding and displaying on Intel archi-tecture personal computers. Experimental results show optimized em-bedding performances and robustness against several malicious attackscommonly happened to videos.

1 Introduction

In proportion to developments of digital infrastructures and industries, digitalwatermarking technologies are also advanced to protect the copyright of thosecontents from illegal distributions and reproductions. In these days, people caneasily access digital contents from various sources, furthermore, their requests forhigh quality contents are rising rapidly. In watermark applications, watermark-ing schemes should be implemented at low cost, especially real-time embeddingsystems are required for authentication, fingerprinting, copy protection systems.Real-time watermarking systems for high quality contents are not an exceptionany more.

In this work, we implemented a real-time watermark embedding system for1920×1080high resolution MPEG-2 video contents on an Intel Pentium� system.We will present some general tips for system optimization in section 2. In the nextsection, we will define our watermark embedding scope and specify implementa-tion details and optimization issues for a real-time embedding scheme. In section4, experimental results are presented to verify our embedding scheme is suitablefor real-time processing and robust against video attacks.

H. Yoshiura et al. (Eds.): IWSEC 2006, LNCS 4266, pp. 227–238, 2006.c© Springer-Verlag Berlin Heidelberg 2006

228 I.K. Kang et al.

processor N

instruction

instruction

instruction

data

data

.

.

.

.

.control unit

processor 1 memory 1

memory N

Fig. 1. SIMD structure : A SIMD structure includes several processors and thoseprocessors are controlled by one control unit. Even if all processors are assigned oneinstruction from one control unit, they use different data from different memories.(Insome cases, processors share memories.)

2 General Optimization Guidelines

Many multimedia applications in these days are implemented for a real-timeprocessing according to their purposes. However, due to intensive computing vol-umes for multimedia data, applications are used to be implemented with appli-cation specific integrated circuits(ASICs) or digital signal processors. Things arechanged owing to the introduction of Single-Instruction-Multiple-Data(SIMD)extensions to general-purpose processors. With this, the following optimizationsare possible [5]:

1. Use a current generation compiler that will produce an optimized applica-tion. This will help you generate good codes from the start.

2. Maximize memory access performance:– Minimize memory references,– Maximize register usages,– Prefetch data,– Arrange code to minimize instruction cache misses and optimize prefetch,– Align frequently executed branch targets on 16-byte boundaries,– Make sure all data are aligned

3. Minimize branching penalties:– Minimize branch instructions, for instance, unroll small loops,– Arrange code to minimize the misprediction in the branch prediction

algorithm, for example, forward conditional branches are usually nottaken and backward conditional branches are usually taken.

4. Use software pipelining to schedule latencies and functional units. Unrollsmall loops to schedule more instructions.

With multimedia instruction set extensions such as Intel’s MMX(Multi-MediaeXtension) technology, we can speed up the computing time up to eight times

Real-Time Watermark Embedding for High Resolution Video Watermarking 229

faster[1][4]. The SIMD technique is the reason to make it possible. Earlier ver-sion of CPUs which do not use the SIMD technique process one data usingone instruction, where as in SIMD one instruction can handle several data con-currently. That is to say, MMX technology offers much greater capabilities ofspeed up to multimedia data owing to eight data, each of eight-bits, concurrencyprocessing power using 64-bits data buffers. Nevertheless, advantages of MMXtechnique, this technique cannot be adopted to all of implementation areas be-cause the MMX technique is best-suitable to simple and steady data structuresor iterative data processing like multimedia data. This method, for that reason,is regarded as a second majority technique to improve system performances, notmain methodology for data process enhancements. Under this circumstances,it is an important work to recognize possible areas of MMX optimization forperformance improvements. The followings are three basic steps [5]:

1. Understand where the application spends most of its execution time: Thebenefit of optimizing computationally-intensive parts is larger than thatof optimizing non-intensive parts. We should start optimizations from themost computationally-intensive components.In our watermarking system, wefound the most time-spending part that bothers the real-time embeddingprocessing and modified it.

2. Understand which algorithm is the best for MMX technology in this ap-plication: Matching the algorithms to MMX instruction capabilities is thekey to extracting the best performance. Once the computationally-intensivesections of code are identified, an evaluation should be done to determinewhether the current algorithm or a modified one will give the best perfor-mance. There could be many algorithms for a same application. In somecases, it is possible to improve performance by changing the types of opera-tions in the algorithm.

3. Understand where data values in the application can be converted to integer(fixed-point) while maintaining the required range and precision if the datavalues are not originally of an integer type: The MMX instructions offer thebest support for 8-bit and 16-bit integer data types. While some DSP can bedone in the integer domain, some must be done in the floating-point domain.MMX can provide significant speedup in certain DSP and multimedia appli-cation, even over hand-optimized floating-point assembly code. MMX seemswell-suited for image processing applications because of the large amountof contiguous, 8-bit data to process and precision requirements that rarelyextend beyond 8 bits.

3 Optimization Issues for Real-Time Embedding

In this section, first we will define real-time scope of our watermarking sys-tem and describe optimization issues and our solutions for those problems. Foran optimum system, we analyzed time complexities of a watermarking systemimplemented using pure C code. Then, we decided which parts of the system


should be rebuilt to reduce the execution time suitable for real-time processingand modified those modules.

3.1 Real-Time Performance Scope

First in this session, we would like to define the term “real-time” in our work.The term “real-time” is cited in many applications in slightly different meanings.The word in our work defines that a watermark embedding process should notbe recognizable during video playing time. That means watermark embeddingtime should be shorter than a frame processing time. In case of 30 fps(framesper second) framerates videos, a frame is decoded and displayed on a screenin around 0.03 second. It is obvious that watermark embedding time should bemore shorter than 0.03 second. Furthermore, we manipulate 1920×1080 size HDcontents to embed watermarks.

Our real-time watermarking system consists of three major parts as shown infigure 2: a decoding unit of HD MPEG-2 video bitstreams into raw frames, a wa-termark embedding unit and a video display unit on a screen. More specifically,our watermarking unit includes subunits such as a HVS masking function, wa-termark multiplication to HVS masked frames and addition of the watermarksto raw frames on a spatial domain, etc. In our real-time scheme, these threeunits are scope of real-time processing.

We included a video display unit to our scope for two reasons. One is that ourreal-time solution could be used in other applications. For example, in digitalfingerprinting systems, the unique fingerprint code assigned to each differentcustomer is embedded to contents at the end-users’ computers, not on suppliers’servers. In this case, the fingerprint codes should be embedded while videos arebeing displayed on a screen. The other is our solution could be adopted to real-time watermark detection systems. In most of watermarking systems, watermarkdetection complexities are lower than embedding complexities. That means if aembedding scheme can be executed in real-time, a detection routine can be alsoexecuted in real-time.

Encodings of watermarked raw frames to MPEG-2 video streams are beyondof our scope. A real-time encoding to HD MPEG-2 bitstreams on personal com-puters is too heavy work and it cannot be implemented by software approaches,without any MPEG-2 encoding specific hardware. It, however, should be notedthat our embedding scheme including decodings of HD MPEG-2 stream, HVSmasking function and watermark addition to decoded frames have merits inwatermark applications. Because all these steps should be performed withintheoretically 0.03 second in case of 30 fps videos and software approaches ofwatermarking systems are more efficient over hardware approaches when modi-fications or updates are needed.

3.2 Use of the MMX Technique to Reduce Computing Time

The MMX instructions offer the best support for 8-bit and 16-bit integer datatypes. MMX can provide significant speedup in certain DSP and multimedia


WMGenerator

InsertPayload

Tiling

HVSLocal

ScaleingADD

Watermark Embedder

WatermarkedFrame

MMX Output

Key

Payload

Frames

Input Hold

Decoder Display

Fig. 2. High-level view of real-time watermarking system. Our real-time scope consistof a HD MPEG-2 decoding, a watermark embedding and displaying of video. In thewatermark embedder, routines that need fast processing are implemented with MMXinstructions.

applications. MMX seems well-suited for image processing applications becauseof the large amount of contiguous, 8-bit data to process and precision require-ments that rarely extend beyond 8 bits. With the MMX technique, the followingoptimization guideline will be applicable for real-time programming [4][5][6][7]:

1. Do not intermix MMX instructions and floating-point instructions. MMXinstructions do not mix well with floating-point instructions. MMX registersand states are aliased onto the floating-point registers and state, so no newregisters or states are introduced by MMX.

2. MMX code sections should end with “emms” instructions if floating-pointoperations are to be used later in the program.

3. MMX shift/pack/unpack instructions do not mix well with each other. Ingeneral, two MMX instructions can be executed at the same clock. However,only one MMX shift/pack/unpack instruction can be executed at one clockbecause there is only one shifter unit.

4. MMX multiplication instructions “pmull/pmulb/pmadd” do not mix wellwith each other. Currently, there is only one multiplication unit.

5. MMX instructions, which reference memory or integer registers, do not mixwell with integer instructions referencing same memory or registers.

6. It is important to arrange data in the best way for MMX processing, e.g.,structure of array, array of structure, row-wise, or column-wise arrangements.Column-wise processing in general is better than sequential row-wise process-ing.

We applied the MMX technology to several embedding modules that spendmuch time and need to be rebuilt for speedup as shown in figure 2. This includeswatermark tiling, a HVS masking, watermark addition to source frames andother miscellaneous operations such as multiplications or additions as depicted ingrayed rectangles. These modules have some common features which are suitableto be manipulated by the MMX instructions: they are frequently and regularlyused, conduct simple and repeatable operations and need fast processing time.The MMX implementation of such heavy modules produced great performance


improvements especially at HVS (Human Visual System) masking function asdescribed in Table 1.

3.3 Use of Simplified HVS Function

A HVS function plays an important role in watermarking systems. Watermarksembedded in flat areas can be notified easily, so we have to embed watermarksinto those areas weakly. On the other hand, watermarks in edge or texturedareas are not noticeable compared to flat areas, so we can embed watermarksmore strongly into those areas. That says a HVS function adjusts watermarkstrength according to local features of source images. While the HVS functionis the most important part for robust and invisible watermarking methods, it isthe heaviest processing step over the rest of other embedding steps.

In many literatures, a NVF (Noise Visibility Function) is a commonly usedHVS function [2]. The NVF function measures flat areas and edge(textured)areas of source images by calculating local variance and mean values. Despiteof its popularity for the purpose of a HVS mask, this function takes bunch ofprocessing time than that of other embedding modules due to heavy calcula-tions for local mean, local variance values and division operations as shown inequation 1, where σ2

x is a local variance of source images, σ2x max is the max-

imum local variance for a given image and D ∈ [50, 100] is an experimentallydetermined parameter. Even if sophisticated computations of local measures inNVF provide more exact estimation of source images, a number of computationsprevent watermark embedding from real-time processing. We confirm this effectin a table 1. In the pure C code implementation of watermarking system, theHVS function took the most longest time in the embedding processes and itsprocessing time(0.05sec.) is already over the 0.03 second, upper bound frameprocessing time in our application.

NV F (i, j) =1

1 + θσ2x(i, j)

, where θ =D

σ2x max

. (1)

For a real-time embedding, the HVS function should be lightened. To reducethe time complexity, we would rather use an edge detector than NVF functionin our system for following reasons:

– As described above, the NVF function is too heavy in real-time video process-ing.

– The NVF function results in floating point values between 0 and 1, i.e.0 ≤ NV F ≤ 1. That means the NVF function cannot be implemented by theMMX instructions, because the MMX code can process floating point valuedata. Furthermore, floating point data operations decrease the processingtime.

– Edge detectors can be also used as a HVS function, because edge detectorsalso extract highly textured area of source images and watermarks can bestrongly embedded in those area.


Fig. 3. Edge Detectors of our HVS module(compass operators): With a “separable”feature and MMX instructions, we implemented the HVS function at a low complexity.(a) shows four different edge detectors and (b) depicts eight compassed kernels of thesecond kernel of (a).

– Furthermore, we also consider flat regions of source images as well as edgeor highly textured area using equation 2 like that used in a NVF function.

– Some edge detectors have a “separable” property which makes its computa-tion more faster and are also more suitable for a MMX instructions.

In our system, we adopted a compass operator as an edge detector. Compassoperators measure gradients in as selected number of directions. Figure 3 (a)shows four different compass gradients for north-going edges. An anti-clockwisecircular shift of the eight boundary elements of these masks gives a 45 degreerotation of the gradient direction. For example, the eight compass gradientscorresponding to the second operator of figure 3 (a) are shown in figure 3 (b).

This compass operator can be transformed to be separable. A 2D filter is“separable” if the kernel [h2D] can be decomposed into two 1D kernels which areapplied successively. The filtering is performed in one dimension (rows), followedby filtering in another dimension (columns) : [h2D] = [h(V )

1D ] ⊗ [h(H)1D ], where ⊗

stands for a convolution operator. The rows and the columns in the originalimage are thus separately filtered. Whatever the first 1D filtering performed,the output image IS(m, n) is still the same. To be separable, a 2D filter musthave proportional elements on the rows and the columns: mathematically that isseldom true, however, several usual 2D filters are separable. In figure 4, propertiesof separable filters and some examples are presented.

The complexity is low for 2D separable filters because the number of oper-ations (multiplications and additions) is reduced, thus the computation timeis faster. Typically if the kernel size is M × N , we need only (M + N) multi-plications and (M + N − 2) additions instead of (M × N) multiplications and


Fig. 4. Separable filter properties and examples. Ie(M, N) is a original image, hH(n)and hV (m) are 1D horizontal and vertical filters, respectively. IS(m, n) is a 2D filteredimage of Ie(M, N).

(M + N − 1) additions for a non-separable 2D filter. Often the term “MAP”is preferred (multiplications and accumulations per pixel): there are (M + N)MAP for a separable filter instead of (M × N) MAP for a non-separable filter.

When applying edge detectors to get a HVS masked image, the result showsonly edged areas or highly textured areas are strongly highlighted. On the otherhand, pixel values of flat areas approach zero. That means watermarks embeddedin only textured areas are emphasized and easily visible, but the watermarkinformation is nearly lost in flat areas. We should concern both textured areasand flat areas for robust and invisible watermarks. As a consequence of thisconsideration, we adopted the contents-adaptive embedding rule in the NVFmethod and slightly modified it suitable for our HVS [2]:

Λ = (C − HV S) · α + HV S · β, (2)

where C is a constant value that limits the upper bound of HVS energies, α is astrength parameter for edged areas(textured areas) and β is a strength parameterfor flat regions. As a result of this embedding rule, watermarks have a strengthrange between α and β. So watermarks could be embedded with a at least βstrength in the very flat areas and the watermark strength could be controlledby adjusting the parameters α and β according to a tradeoff of a watermarkvisibility and robustness.

4 Experiments

We experimented our embedding scheme on an Intel Pentium IV processor3.6GHz and with a 1 GB RAM. The decoding time of HD bitstreams and the dis-play time after watermark embedding is constant as shown in table 1 and these


values are beyond of our analysis scope because they are already optimizedfor the best performance. We embedded watermarks into 1920×1080 size HDMPEG-2 videos using the same watermarks and embedding ways as the methoddescribed in [3]. We adjusted the Λ value in equation 2 between 1 to 7, thatsays watermarks were embedded in strength 7 on textured areas and strength 1on flat areas. An average PSNR after watermark embedding was around 45dB,so image visual degradations due to watermark embedding were not perceptual.We tested five MPEG-2 HD videos in the fields of movies, entertainment shows,documentaries, etc. which contain more than 20 different scenes and features asshown in figure 5.

Fig. 5. Snapshot examples of test videos: All test videos have 1920×1080 resolu-tions(HD). This test set contains a number of scenes and different features.

First, we measured time complexities of subfunctions for a embedding process.As mentioned earlier, the decoding process and the display process took a con-stant time during all experiments and time complexities for embedding stepssuch as a HVS masking function, multiplication and addition operations arevaried according to the implementation methods and programming languages asshown below:

All the three experiment systems consist of three subfunctions: decoding ofbitstreams to raw frames, watermark embedding and a display of watermarkedframe to a screen. The watermark embedding unit includes taking a HVS mask-ing function to decoded raw frames, multiplication and addition operations forwatermark manipulations and other miscellaneous operations which do not takeserious time for an analysis. The first experiment was designed to measureprocessing time taken for units, each of which is implemented using pure Cprogramming language. In this test, the NVF masking function was used asa HVS method. As we mentioned earlier, the NVF masking method with Ccode implementation is inappropriate for a HD real-time video watermarking


Table 1. Time complexity comparisons between experiment systems that implementedwith pure C code and MMX code. Execution time taken for each subfunction is shownin the table. The total time is the time complexity taken for watermark processing perframe.

[sec.]

0.02800.0020.0020.006

(our HVS)System 3.

0.02100.0010.0020System 2.

0.0770

0.0048

0.0030.0060.050

(NVF)

0.0132

System 1.

miscellaneousMUL, ADD

functionHVS function

Total TimeDisplay Time

Watermark Embedding TimeDecoding

time

0.02800.0020.0020.006

(our HVS)System 3.

0.02100.0010.0020System 2.

0.0770

0.0048

0.0030.0060.050

(NVF)

0.0132

System 1.

miscellaneousMUL, ADD

functionHVS function

Total TimeDisplay Time

Watermark Embedding TimeDecoding

time

application: the NVF process time is 0.050 second and the total processing timeis 0.077 second, beyond of the time limit per frame.

A MMX code implementation considerably drops processing time comparedwith C code implementation. In the second experiment, all units were imple-mented with MMX code but a HVS function. We observe that the MMX codingconsiderably reduced the time complexity of a whole embedding process andthe real-time watermark embedding is feasible. However, watermarks that weresimply added to source frames without any HVS processing were not robust tovarious well-known watermark attacks.

Our proposed HVS method conducted its job fast under MMX implemen-tation. In the third experiment, we applied an edge detector as a HVS mask,the second one in figure 3 (a) and its compassed ones like (b) with a “sep-arable” feature. Our masking function took only 0.006 second to get maskedimages and the total processing time is 0.028 second, still under the processinglimit time per frame. That means our implementation can decode HD-resolutionbitstreams, embed robust watermarks with HVS masking and display the wa-termarked frames on a screen within 0.03 second. From the results, the totalprocessing time of system 1, 0.077 sec., was decreased to 0.028 sec. of system 3,thus we achieved about 63% performance enhancement.

We tested robustness of our watermarks against several common attacks tovideos. Various video processing attacks in the table 2 are expected to occur toHD videos owing to its high resolution and excellent visual qualities. People mayscale down HD contents to VGA files for efficient playing on personal computers,convert the MPEG-2 format to various MPEG-4 formats to reduce file sizesfor easy manipulations or network transference. Watermarks embedded in ourmethod survived against various kinds of attacks as shown in below table.

It should be noted that we selected attack items by focusing on video ma-nipulations which could be happened frequently to HD videos, so we did notconsider common image(signal) processing attacks or geometric attacks of


Table 2. Summarization of performance of our watermark robustness in the real-timesystem. After various attacks, watermarks were survived with the following normalizedcorrelation values.

0.730.300.60Slow Motion

0.700.500.60MPEG-4 Conversion


0.310.100.18White-noise Addition

0.700.250.58Color Conversion (RGB to Gray)

0.740.250.62Framerate Conversion (30 24 fps)

0.690.200.56Cropping

0.450.140.35Resize to VGA (640x480)

0.450.150.28Resize to VGA with padding (640x360)

0.730.370.63Original (1920x1080)

MaximumMinimumAverage

0.730.300.60Slow Motion



0.310.100.18White-noise Addition

0.700.250.58Color Conversion (RGB to Gray)

0.740.250.62Framerate Conversion (30 24 fps)

0.690.200.56Cropping

0.450.140.35Resize to VGA (640x480)

0.450.150.28Resize to VGA with padding (640x360)

0.730.370.63Original (1920x1080)

MaximumMinimumAverage

[ Normalized Correlation]

to

image watermarking. However, we applied robust watermarking methods suchas spread-spectrum method in [3] or human visual masking method in [2], thuswe expect our watermarks would be robust against those kind of attacks.

5 Conclusion

As digital content markets and infrastructures are emerging, high quality con-tents are becoming the center of market shares. Digital watermarking techniquesshould come up with the market currencies. In this paper, we proposed real-time video watermarking implementation guidelines for HD videos. We sug-gested a system implementation using the MMX technique to decrease processingtime and increase multimedia manipulation efficiency, a simplified HVS methodfor robust watermark embedding. The MMX implementation offers significantspeedup in multimedia data processing and certain DSP instructions becauseMMX instructions are well-suited for manipulating large amount of contiguous8-bit data that are common forms of multimedia data.

Our HVS method provides low time complexities and high performance ca-pabilities. By using an edge detector for the purpose of HVS mask, we achievedthe real-time embedding requirement as well as robustness. The “separable” fea-ture of edge detectors reduces multiplication and addition complexities and itsstructure is well-suited for the MMX implementation.

We measured the processing time of our embedding units and experimentedthe robustness of our watermarks against various video dedicated attacks. Thewatermarking system that was implemented with MMX instructions and ap-plied with the simplified HVS function presented the best performances in both


real-time processing and robustness. With our implementation, watermarks areembedded into HD-resolution high quality videos in real-time, 0.03 second perframe, and robust against video manipulation attacks. All these techniques couldbe applied to other video watermarking schemes and other video signal process-ing applications.

Acknowledgment. This work is financially supported by the Ministry of Ed-ucation and Human Resources Development(MOE),the Ministry of Commerce,Industry and Energy(MOCIE) and the Ministry of Labor(MOLAB) through thefostering project of the Lab of Excellency.

References

[1] A. Peleg and U. Weiser : The MMX Technology Extension to the Intel Architecture,IEEE Micro, Vol. 16, no. 4, pp. 42-50, Aug. 1996.

[2] S. Voloshynovskiy, A. Herrigel, N. Baumgartner and T. Pun : A stochastic ap-proach to content adaptive digital image watermarking, In International Workshopon Information Hiding, Vol. LNCS 1768, pp. 212-236, Springer Verlag, Dresden,Germany, 1999.

[3] Ingemar J. Cox, Joe Kilan, F. Thomson Leighton and Talal Shamoon : Securespread spectrum watermarking for multimedia, in IEEE Trans. on Image Process-ing, Vol. 6, No. 12, Dec., 1997.

[4] The complete guide to MMX technology, McGraw-Hill, 1997.[5] Yen-Kuang Chen, Matthew Holliman, William Macy and Minerva Yeung : Real-

time Detection of Video Watermark o nIntel Architecture, Proceeding of SPIE,Vol. 3971, pp. 198-208, May 2000.

[6] R. Ceolho and M. Hawash : DirexX�, RDX, RSX and MMX Technology : a Jump-start Guide to High Performance APls, MA, Addison-Wesely, April 1998.

[7] Intel Architecture MMX Technology Developer’s Manual, IL, Intel Corporation,Order Number 243006, 1996.

lncs 4266 - real-time watermark embedding for high …hklee.kaist.ac.kr/publications/2006 iwsec(with...

Documents