Lab 4: Image Processing and Photographic Mosaic

EE299 Winter 2008

Due: Exercises 1-2 are due on or before 8 February, and the remainder of the lab is due on or before 15February 2008.

Objective

The purpose of this lab is to learn about digital images and perform some basic image processing in MATLAB.You have already done some work with images in MATLAB in Lab 1. In this lab, you will learn a fewmore tools for handling images and develop a better understanding of the tools that you have used. Morespecifically, this lab covers:

1: representing and displaying images in MATLAB;

2: simple techniques for image filtering and transformation;

3: how to create synthetic color images; and

4: creating a photographic mosaic (an image created from many much smaller images).

There are four exercises total in this lab: exercises 1-3 allow you to practice loading, displaying, and changingimages in MATLAB, and in exercise 4 you will make your photo-mosaic.

Prelab:

To ensure that you can complete everything within the allotted time, before your lab section you should:

• Read the lab before coming to lab.

• Try to finish parts 1-2 (ideally 1-3) during the first week of lab.

• Before the second week of lab, find a digital picture that you would like to turn into a photo-mosaic,and make sure that it is available to download to a lab computer. Ideally, you should pick somethingthat is fairly high resolution to start with. Teammates can have the same picture or different ones.

Optional for more ambitious students: If you want to use their own image collection to build themosaic, you should have that collected in one directory that you can access from the Sieg 232 lab, or burna CD ahead of time.

1 Representing Images in MATLAB

In this section we will talk about how to read, display and write images in MATLAB, as well as howto manipulate subimages. As a preliminary, it is important to understand that a digital image (color orgrayscale) is just a bunch of numbers, but the numbers themselves can be characterized using differentformats. Digital systems typically represent two types of numbers: integer and floating point. There aremany types of each in MATLAB, but the ones we will focus on are double (floating point) and uint8 (8 bit

1

Table 1: Differences between double and uint8 in MATLAB

double uint8type floating point integerbits 64 8

range roughly -1.8e+308 to 1.8e+308 0 to 255

unsigned integer). Most all variables in MATLAB are double by default, and most functions expect doubleand will issue a warning if it is something else. In image storage and display, however, uint8 is usually usedbecause it is more memory efficient. In image processing (where numbers are being multiplied and added),the double representation is often preferred. The differences between double and uint8 in MATLAB areshown in Table 1. You can figure out the type of representation associated with a particular variable ormatrix by using the MATLAB class function. You can also specify the type as an argument in buildingmatrices of ones and zeroes, as we will see in the first exercise.

Recall from Lab1 that there are three general methods used for representing images: intensity, indexed,and “truecolor.” For images that only have one color plane (i.e. NxMx1 matix) and have elements of theuint8 type, MATLAB will interpret the matrix to be an indexed image, and will use each element value asan index into a colormap. If the elements are of the double type, with values between 0 and 1; MATLAB willinterpret this as an intensity image. If this is a grayscale image, 0 represents black and 1 represents white. Ifit is a truecolor image, meaning there are three color plane (i.e. NxMx3 matrix), the [0,1] range representsthe intensity of the three colors red, green and blue; black corresponds to (0,0,0) and white to (1,1,1).

The “colormap” is a K-length vector (where K = 2N given N bits/pixel) for grayscale images, and aKx3 matrix for color images with the columns corresponding to red, green and blue intensities. The valuesin the colormap should also be in the [0,1] range. The colormap is essentially a lookup table.

1.1 Reading, Displaying and Writing Images

As for sound files, there are three ways for getting images into MATLAB:

• Convert an external image file into a MATLAB matrix with the imread command, using either:>> myImage = imread(‘someimage.ext’);>> [myImage map] = imread(‘someimage.ext’);Images can be stored in different formats on your computer, and the filename extension ext specifiesthe format. MATLAB can handle a large number of formats, including jpg (or jpeg), gif, tif (or tiff),pgm, bmp and more, as documented in the MATLAB help documentation. The variable myImage is anMxN array if the file is a grayscale image or MxNx3 for most color formats. The map variable specifiesthe colormap and is only needed for indexed images.

• Load an image that already exists as a MATLAB matrix into your workspace, using the load command,as you did in Lab1 with the “durer” image. The syntax is either:>>load durer;>>load(‘durer’);The signal is put into a previously-defined variable, in this case X, and it has unnormalized doubleformat. (For some tools, you will need to normalize it to the [0,1] range.)

• Create a matrix from scratch in MATLAB, as in the example script makeGrayImage.m, which you willuse in Exercise 1.

Once you have generated a matrix, the original format (jpg, gif, etc.) doesn’t play a role, though the in-dexed/intensity/color difference will matter.

2

There are three ways to display an image: image, imagesc and imshow. For photos, the imshow com-mand is nicer, since it doesn’t give the dimension tick marks on the edges of the image that the othercommands provide (which can be useful for scientific graphics), but it requires that the intensities be in therange [0,1]. If the values of the matrix are not in the [0,1] range, then you should use the imagesc command.Now, let’s say you type:

>>imshow(myImage);then MATLAB will assume that a 2-dimensional (MxN) myImage is grayscale and a 3-dimensional (MxNx3)myImage is color. You can also use the indexed representation of a color image using

>>imshow(myImage,map);in which case myImage is 2-dimensional with integer uint8 (or uint16 ) values and map is a Kx3 (RGB) ma-trix as described above. You can change the colormap for by using the colormap() function. The differentcolormaps in MATLAB can be viewed in the MATLAB help function or by typing “doc colormap” at thecommand line. You can also use non-standard colormaps (try >>[mon map] = imread(‘mondrian.gif’);and look at the resulting map) or make up your own.

Once you have created or modified an image, you can save the image using either>> imwrite(myImage,‘filename.ext’,fmt);>> imwrite(myImage,map,‘filename.ext’,fmt);

where myImage is the matrix your image is stored in, “filename.ext” is the name you want the file to have,and fmt is any format that MATLAB supports (e.g. ‘jpg’, ‘gif’, etc.). By convention, it is nice if ’fmt’ and’ext’ are the same. (Note that the output format need not be the same as the input format.) The mapvariable is only needed if you are using indexed color images.

1.2 Working with SubImages

In Lab1, we learned how to access (and change) a part of an image. We can use the same technique togenerate a new image that is a part of the original image. For example, download the Paolina image fromthe course web page (make sure to get the full image, not the thumbnail) and try the following:

>>pao = imread(‘Paolina.gif’);>>A=pao(1:240,1:256);>>imshow(A);

You should see the upper left quadrant of the Paolina image (since it is 480x512, which you can determineusing the size command). You can use “:” alone to indicate that you want everything in a particulardimension, so pao(:,1:256) corresponds to the left half of the Paolina image.

We can alter parts of an image by doing mathematical operations on that part. The script changePaolina.mon the class webpage shows how to lighten/darken different parts of the image and add lines in specifiedplaces.

We can build bigger images out of smaller images (which will be important for the photomosaic) byconcatenating images. For example, you can put images side-by-side in MATLAB by concatenating them ina row, as in

>>group=[A1 A2 A3];or you can stack them as in

>>group=[A1; A2; A3];Using these together, you can create a bigger image out of subimages, which you will do in the next exercise.

Download the makeGrayImage M-file from the course web page. This is a function that will create animage object with concentric squares. Try:

>>y=makeGrayImage;>>imshow(y);>>figure;>>z = [y y y];>>mapc=colormap(’cool’);

3

>>imshow(z,mapc); This should give you two figures, one grayscale with one block and the other in colorwith 3 blocks.

For another example, look at the jumble.m script on the course webpage which scrambles the Paolinaimage.

Exercise 1: Write one or more new functions similar to makeGrayImage but that generate a differentpattern, and use these in a script to make a quilt image. You can also mix in parts of photos (such asPaolina), but you must have some squares with geometric patterns that you create. Show the image to yourTA using a different color map in the display. Write out the image to a file and upload it to the CollectItEE299 Lab 4 space during the first week of the lab. You should verify that the file you created looksthe way you intended before you submit by clicking on it to display it in Windows.

2 Image Processing

Modifying images using signal processing is similar to modifying sounds, except that you need to keep trackof two or three dimensions instead of one. As with digital sounds, there are many things that you can dowith digital images. For example, you can:

• operate on one pixel at a time, as in the changePaolina.m script,

• create an “echo” effect as with the lamp image we’ll be working with (which is 256x256)>>lampE=lamp;>>lampE(:,8:256)=0.7*lampE(:,8:256)+0.3*lamp(:,1:249);

• or change a pixel based on a region of its neighbors as in the filtering that we will explore next.

Download the lamp image and the filter image.m function from the course webpage. Read the imageinto MATLAB (don’t forget to use the correct extension) and run the script on it. The function returns twofiltered images and displays the original with them. Note that you can run the function with or withoutassigning the returned images to a variable, so either

>>[lampL lampH]=filter image(lamp);>>filter image(lamp);

will work (assuming the image is assigned to variable lamp). In this script, the low pass filter is a simplesmoothing function over a 10x10 region, and the high pass filter is designed using the MATLAB filter designfunction. Both filters are designed using the same 1-dimensional techniques you used for sound files, and thenturned into a 2-dimensional filter by taking a vector outer product. It is also possible to design 2-dimensionalfilters directly, but this is beyond the scope of the class.

Exercise 2: Create a new script that modifies the lamp image and displays a sequence of the original plustwo modified versions of the picture using different processing techniques than those in the filter image.mfunction. You can use filters with different design criteria, or warp the intensities, or add an echo, or anyother variation that is of interest to you. You may copy code from any of the m-files provided with this lab,but you must change some part of it. Explain the techniques that you used to your TA.

3 Color Images

Accessing parts of RGB images is a little trickier because of the third dimension. Now we also have a colorchannel: myColorImage(row, column, channel). For an RGB image there are only three color channels, 1 =Red, 2 = Green, and 3 = Blue. If we want to cut out or operate on a sub-image of all the channels, we cansimply use the ‘:’ to tell MATLAB that we want everything associated with a dimension of the array. Forexample,

>> mySubImage=myColorImage(1:50, 100:120, :);

4

will select a sub-image of 1,100 to 50,120 with all three color planes. If all we wanted was the green channel ofthe full image then we can use myColorImage(:, :, 2);. You can select a single channel and then displayit as a grayscale image using

>> Gpart=myColorImage(:, :, 2);>> imshow(Gpart);

or you can remove the green from the image by using:>> myColorImage(:, :, 2)=0;

Exercise 3: Download the mfile makecolorimage.m from the class webpage. The purpose of this scriptis to demonstrate constructing an RGB image in MATLAB. There are 400 pixels in the image; each pixel’scolor is determined by the three values (between 0 and 1), representing the red, green, and blue intensitiesthat make up the color. When all three values are 1, the pixel appears as white. Setting the red value of apixel to zero causes it to appear as a combination of green and blue.

a) Examine the mfile and the resulting images so you understand how to change the color values forregions of pixels in the image. Then change the image matrix A to add a red region (it doesn’t have tobe a square) somewhere in the image, and display the result.

b) Download the color image parrot.jpg and change the colors in two regions of the image. You canchange colors by modifying the intensity (using scaling, adding or filtering) or simply zeroing (removing)them.

4 Photographic Mosaic

A photographic mosaic is an image that is created from many smaller images. The effect is to recreate somepicture (e.g. a face) by replacing small portions of that picture with another image (which we’ll call a tile)that has the same average color. At a distance, the mosaic will look like the original picture, while up close,the individual tiles can be seen. An example of a photographic mosaic can be seen in the lab and on theclass web site under Lab 4. To learn more about the history of photographic mosaics see Wikipedia’s entryat http://en.wikipedia.org/wiki/Photographic_mosaic.

Using the notion of sub-images discussed earlier, we can split a picture into sub-images and compareeach one’s average color value with a tile-set. The original picture’s sub-image will be replaced with a tilefrom the tile-set that has the closest average color value. The tile-set you will be using contains over 5,000images of various subject matter. While it should give you a good result, typically mosaics that contain atile-set with a theme have a more profound impact artistically. For example, a mosaic of an animal mightbe made of tiles from nature photos.

It is not difficult to make a photographic mosaic in MATLAB, and the functions to make the mosaicare given to you so you have more time to be creative with the project. Some of the parameters are left upto you to choose, such as the number of sub-sub-blocks (“nBlocks”) to split the tiles into. In addition, youare strongly encouraged to look at the code to make sure you understand it and because you may want toenhance or modify the functions.

Exercise 4: Using the procedure outlined next, make two versions of a photographic mosaic from an imageof your choosing. There are 5 tile-sets to choose from: 25x25, 50x50, 75x75, 100x100, and 150x150 pixels.The different tile-sets all characterize the same set of images, but different sizes may be better for differentpictures or final image sizes.

a) Poster printout: Choose a poster size of either 11”x14”, 13”x19”, 16”x20”, or 18”x24”. Calculatethe image dimensions (in pixels) needed to create a print at 300 DPI with the chosen poster size. (Forexample, an 11”x14” poster requires a 3300x4200 pixel image.) Resize your image so that it is this sizeor larger. The image does not have to have the exact dimensions; it can be cropped to fit the posterdimensions. If cropping is needed, use the idea of sub images to cut out the part of the picture that

5

you want. Choose how many tiles per inch you want to have in the mosaic and use the tile-set thatcomes closest. (As an example: the 100x100 tile set is used in the poster hanging in the lab.)

b) Web version: For the mosaic that you will turn in via CollectIt should be suitable for a web site, sochoose the display size in inches (roughly) that would be reasonable on a computer screen. Follow thesame procedure as for the poster, except that you will resize the original image and choose the tile-setbased on a different resolution: monitors typically have 96 to 120 DPI. Explain your tile-set choice toyour TA. (Note: If you create multiple poster mosaics in your team, you need only pick one to be usedfor the web page version.)

Getting Set Up. From the class web page, download the two functions needed to create the mosaic:

• getAverages.m, precomputes a simple representation of images in the tile-set for search, and

• mosaic.m, creates the mosaic.

Next download the image you want to base the mosaic on. Finally, download your desired tile-set fromsccserv.ee.washington.edu (in the /v0/data/images/mosaictiles/ directory). The tile-sets are storedin a mat file as a 4-dimensional matrix, shown in Figure 1, as in tilesetN(rows, columns, rgb, image number).Load the tile-set into MATLAB, e.g. using:

>> load tileset25and the tile-set will be loaded as a variable with the same name as the filename. try displaying a couple ofimages from it with the imshow command (hint, use the ‘:’ to specify you want everything, such as all rows,or columns).

Figure 1: The tile-set layout.

Formatting the Original Image. In order to make the mosaic the correct size, your image may haveto be adjusted to make the mosaic dimensions. If your image is smaller than the size needed for the posterthat you computed, then you will have to enlarge the image using the imresize command. Once it is thesame or larger than the required size, check to see if the image size corresponds to an integer multiple of thetile size in both dimensions. (For example, a 50x50 tile would perfectly cover a 1200x1000 image, but not a1220x980 image.) If not, you need to crop your image, as shown in Figures 2(a) or 2(b). The figure displaysan image of birds with a grid representing the size of the tile-set. Each one of these squares in the grid willbe replaced with a tile in the tile-set. The image does not fit into the squares at the edges, so the imagemust be cropped (or cut) to fit into the grid. In MATLAB, cropping simply involves taking a sub-imagefrom the picture.

6

(a) Image cropped from the bottom-right.

(b) Image cropped from the middle.

Figure 2: Examples of cropping the image to be divisible by the tile-set size.

Formatting the Tile-Set. With the image ready and the tile-set chosen, use getAverages.m to precom-pute the averages for the tile-set, as in:

>> tileAve=getAverages(tileData,nBlocks);The function takes two arguments, “tileData” and “nBlocks”. The first is name of the tile-set that you haveloaded into MATLAB, and the second defines an NxN grid of sub-blocks to split each tile into, shown inFigure 3. The sub-blocks are used in computing the distance between a block in the image and each differenttile. For the example in Figure 3, “nBlocks” was set to three to produce a 3x3 set of sub-blocks for that tile.The getAverages.m function will get the average of each of these sub-blocks for each tile. You can think ofthem as sub-sub-images. If you set “nBlocks” to one, then only a single average (for each color plane) willbe recorded for that tile. The “nBlocks” variable provides a way to control how closely the sub-images inthe picture match the tiles in the tile-set. Try different values to see how it affects your mosaic, but avoidbig numbers since that makes the function much slower. Of course, “nBlocks” must be smaller than the tilesize, but it need not divide evenly into the tile size. Note: You will need to rerun getAverages.m whena new tile-set is loaded or if you want to try a different value for “nBlocks.”

Making the Mosaic. Finally, to make the mosaic, use the function mosaic.m. It takes three arguments,“img”, “tileData”, and “tileAve”. The first is the image from which you will create the mosaic, the secondis the tile-set, and the third argument represents the precomputed averages returned from getAverages.m.

7

Figure 3: “nBlocks” determines the number of sub-sub-blocks that a sub-image is split into. In this case,“nBlocks” = 3, so the sub-image is split into 3 x 3 sub-sub-blocks for purposes of finding the distance ofthat block to images in the tile set.

This function splits the image up into sub-images that are the same size as your tile-set, and then iteratesthrough each one. For each sub-image, the function computes the average in the same way as getAverages.m,and for each sub-sub-image the distance between that average and the precomputed averages of all the tilesin the tile-set is calculated. The distances for each sub-sub-image are summed for that sub-image, and thetile in the tile-set with the minimum cumulative distance is inserted in the sub-image.

Optional: If you want to explore some variations that will require more code changes, some possibilitiesinclude:

• modify the functions to operate on grayscale images;

• prevent the same tile from being repeated immediately after itself; and/or

• create your own tile set (contact the TA for an additional script that can help with this).

Upload your web-sized mosaic to CollectIt EE299 Lab 4.

IMPORTANT: You only get 2 free poster-size printouts per person, so make sure you are happy withyour final result before you print it out on the big color printer. The printer driver is installed only on onemachine in the lab. See your TA for help with printing.

8