processing and 3d visualisation of biological ......images recorded via optical sectioning...

PROCESSING AND 3D VISUALISATION OF

BIOLOGICAL VOLUMES OF CONFOCAL

MICROSCOPY

Thomas Bauer

January 2003

Contents Thomas Bauer

2 GVA-ELAI-UPM®PFC0069-2002

CONTENTS

1 INTRODUCTION...........................................................................................5

1.1 AIM OF THE PROJECT ..................................................................................5

2 STATE OF THE ART.....................................................................................7

2.1 THE CONFOCAL MICROSCOPE .....................................................................7 2.1.1 THE PRINCIPLE OF THE CONFOCAL MICROSCOPE.................................... 7 2.1.2 THE FILE FORMAT................................................................................ 9

2.2 IMAGE IMPROVEMENT ..............................................................................10 2.2.1 DENOISE........................................................................................... 11 2.2.2 DEBLURING ...................................................................................... 11 2.2.3 DECONVOLUTION.............................................................................. 12

2.3 RENDERISATION........................................................................................13 2.3.1 VOLUME RENDERING......................................................................... 14 2.3.2 SURFACE RENDERING........................................................................ 14 2.3.3 RESUMEN ......................................................................................... 15

3 VISUALISATION.........................................................................................17

3.1 USING MATLAB .........................................................................................17 3.1.1 VISUALISATION WITH MATLAB.......................................................... 17

3.2 USING VTK...............................................................................................18 3.2.1 VISUALISATION WITH VTK................................................................ 19 3.2.2 DATA-PIPELINE OF VTK:................................................................... 19

Thomas Bauer Contents

GVA-ELAI-UPM®PFC0069-2002 3

3.3 REGION OF INTEREST................................................................................19 3.3.1 CHOOSING A ROI .............................................................................. 20

4 DATA-PROCESSING...................................................................................23

4.1 DISPLAY IMAGES WITH MATLAB ...............................................................23 4.1.1 DISPLAY A SINGLE SLICE .................................................................... 23 4.1.2 DISPLAY MULTIPLE SLICES ................................................................. 25 4.1.3 CONVERTING SLICES INTO A MOVIE .................................................... 28

4.2 DISPLAY A VOLUME WITH VTK.................................................................30

4.3 CALL THE VTK PROGRAM FROM MATLAB ................................................33

5 FUTURE ENHANCEMENTES.....................................................................37

Contents Thomas Bauer


Thomas Bauer Introduction


1 Introduction

1.1 Aim of the project

The aim of this project is to reconstruct a 3D representation from a serie of 2D images, made with a confocal microscope. The typical size of the data coming from the confocal microscope is about 20 Mb. We want to find an easy way to display the 3D image and to have the possibility to view the volume from diferent angles. We want to enhance the quality of the images by the means of debluring, enhancement of the signal- to-noise ratio and deconvolution. To reach this goals we use the Matlab software whith some aditional Toolboxes (e.g. Image Processing Toolbox, Morphology Toolbox) and the Visualisation Toolkit (VTK).

Introduction Thomas Bauer


Thomas Bauer State of the art


2 State of the art

2.1 The confocal microscope

2.1.1 The principle of the confocal microscope

Mostly used in biomedical imageing, the Laser Scanning Confocal Microscopy (LSCM) is used for obraining high resolution images and their representation in three dimensional reconstructions. LSCM is also known with the name CSLM (Confocal Scanning Laser Microscopy).

In LSCM, a laser light beam is expanded to make optimal use of the optics in the objective. Through a x-y deflection mechanism this beam is turned into a scanning beam, focussed to a small spot by an objective lens onto a fluorescent specimen. The mixture of reflected light and emitted fluorescent light is captured by the same objective and (after conversion into a static beam by the x-y scanner device) is focused onto a photodetector (photomultiplier) via a dichroic mirror (beam splitter). The reflected light is deviated by the dichroic mirror while the emitted fluorescent light passes through in the direction of the photomultiplier. A confocal aperture (pinhole) is placed in front of the photodetector, such that the fluorescent light (not the reflected light!) from points on the specimen that are not within the focal plane (the so called out-of-focus light) where the laser beam was focussed will be largly obstructed by the pinhole. In this way, out-of- focus information (both above and below the focal plane) is greatly reduced. This becomes especially important when dealing with thick specimens. The spot that is focussed on the center of the pinhole is often referred to as the "confocal spot." A simple arrangement of a LSCM is shown in the figure below (Figure 2.1).

State of the art Thomas Bauer


A 2-D image of a small partial volume of the specimen centered around the focal plane (referred to as an optical section) is generated by performing a raster sweep of the specimen at that focal plane. As the laser scans across the specimen, the analog light signal, detected by the photomultiplier, is converted into a digital signal, contributing to a pixel-based image displayed on a computer monitor attached to the LSCM. The relative intensity of the fluorescent light, emitted from the laserhit point, corresponds to the intensity of the resulting pixel in the image (typically 8-bit greyscale). The plane of focus (Z-plane) is selected by a computer-controlled fine-stepping motor which moves the microscope stage up and down. Typical focus motors can adjust the focal plane in as little as 0.1 micron increments. A 3-D reconstruction of a specimen can be generated by stacking 2-D optical sections collected in series.

Figure 2.1: Principle of a confocal microscope



The general setup of an entire LSCM system is shown in figure 2.2. It should be noted that most laser scanning confocal microscopes consist of a confocal unit attached to a conventional fluorescence microscope.

2.1.2 The file format

The files produced by the confocal microscope have a special file format. At the beginning of the file there is placed a header of 76 Byte. In this header you can find information about the file it self and about the images. For example in the header is stored the number of the slices, the dimensions of the slices in pixels, the colour-range and the filename. Atention, the filename is the name that was given to the file from the confocal microscope and it has no correlation with the Windows-filename. The representation of the data in the header is not equal for every information. The size of each information varies, therefore we have to read the header bytewise or wordwise. Below you can find a discription of the header. (Figure 2.3)

Figure 2.2: Confocal microscopy system



Datatype Name byte size(byte) Information

Int nx, ny 0 2*2 image wight and hight in pixels

Int Npic 4 2 number of images in file

Int ramp1_min, ramp1_max 6 2*2 LUT1 ramp min. and max.

NOTE Notes 10 4 no notes=0;has notes=non zero

BOOL byte-format 14 2 bytes=1, words=0

Int N 16 2 image number within file

Char name[32] 18 32 Filename

Int Merged 50 2 merged format

Unsigned short color1 52 2 LUT1 color status

Unsigned short file_id 54 2 valid *.PIC file=12345

Int ramp1_min, ramp1_max

56 2*2 LUT2 ramp min. and max.

Unsigned short color2 60 2 LUT2 color status

BOOL Edited 62 2 image has been editet=TRUE(1)

Int Lens 64 2 integerpart of lensmagnification

Float mag_factor 66 4 4 byte real mag. factor (old) unsigned char dummy[3] 70 6 NOT USED (from old vers.)

2.2 Image improvement

Due to the caracteristics of a confocal microscope it is necessary to improve the produced images and to enhance the quality of the images. To obtain better results, it is indispensable for example to filter the noise in the images or to enhance the signal-to-noise ratio (SNR). Despite the fact we are using the pinhole in the confocal microscope, we still have a lot of out-of- focus light, we also need to reduce or better to eliminate it. Important and usefull techniques are:

- denoise

- debluring

- deconvolution

Figure 2.3: Header of file *.PIC



2.2.1 Denoise

With the technique “denoise” we try to get rid of small objects in the image which do not belong the the original image. To eliminate different kinds of noises there are three types of filters to use, depending on the type of noise, one of the filters can have better results than an other. We can choose between the median-, averaging- and adaptivefilter, in all off the different types it is necessary to define the number of pixels in the neighbourhood to take in account to remove the noise. Below (Figure 2.4) you can see an example for an applied filter to an image with salt & pepper noise, the used filter is a median filter with size 3 x 3 pixels in the neighbourhood.

2.2.2 Debluring

If an image is not very clear and not very sharp we say that it is blured. Debluring means, that we use some kind of filter for example to make it clearer and to improve the contrast of the image. The Blind Deconvolution Algorithm for example can be used effectively to deblur the image, when no information about the distortion (blurring and noise) is known. The algorithm restores the image and it is not necessary to know anything about the source of the distortion.

Figure 2.4: Noise elimination



2.2.3 Deconvolution

Deconvolution is a technique for removing out-of- focus light in a series of images recorded via optical sectioning microscopy. However, besides the in-focus structures the images usually also contain out-of- focus light from other parts of the object, causing haze and severe axial blur. This is even the case for a laser scanning confocal microscope, where most of the out-of- focus light is removed from the image by a pinhole system.

Mathematically, the image produced by any microscopic system can be described as the convolution of the ideal unblurred image of the specimen and the microscope’s so-called point spread function (PSF), i.e., the image of an ideal point light source. With the inverse of this process, called deconvolution, a deblurred image of the specimen can be obtained, provided the point spread function is known or at least, can be estimated somehow.

To deblur the images before further processing, there are existing some different methods of deconvolution. The Lucy-Richardson algorithm for example can only be used, when the PSF of the system is known. The Wiener-Filter is usefull for imagis with motion-noise and it is necessary to have information about the size and the angle of the noise. Furthermore the iterative maximum likelihood algorithm for deconvolution.

Mainly there are existing two variants of powerful iterative maximum likelihood image restoration algorithms, a non-blind one and a blind one. The difference between these two variants is that in the first case a measured or computed point spread function is used, while in the second case the PSF is estimated along with the data itself. As we don’t know the PSF we will use the blind deconvolution.

Figure 2.5: PSF due to point source and pinhole

Point Spread Function (PSF)

Light intensity at (x,y,z) due to a point source at (0,0,0)

Airy disk or Airy diffraction pattern



Maximum-likelihood image restoration can be considered as the standard for deconvolution of 3D optical sections. Although computationally quite expensive, the method is able to significantly enhance image quality, at the same time being very robust and insensitive with respect to noise artifacts. However, it should be noted that, although rejecting most of the out-of- focus light, does not mean that all of it is rejected. Therefore, some noticeable haze remains in the images. Also, the images retain a substantial axial smearing in z-direction, which cannot be removed by any deconvolution algorithm.

2.3 Renderisation

Renderisation is called the method to process a digital image to display it on the computer screen. For example if we have a set of 2D images and we want to have a 3D representation of it, we need to render it first befor diplaying it.

There exist different ways of rendering an image, first you can use a hardware or a software supported rendering method, than you can choose between volume rendering and surface rendering. As we are using Computers without a special renderization hardware bord, we will use only the software methods.

Figure 2.6: Example for deconfolution



2.3.1 Volume rendering

Volume rendering is a computer graphics technique whereby the object or phenomenon of interest is sampled or subdivided into many cubic building blocks, called voxels (or volume elements.) A voxel is the 3D counterpart of the 2D pixel and is a measure of unit volume. Each voxel carries one or more values for some measured or calculated property of the volume (such as intensity values in the case of LSCM data) and is typically represented by a unit cube. The 3D voxel sets are assembled from multiple 2D images (such as the LSCM image stack), and are displayed by projecting these images into 2D pixel space where they are stored in a frame buffer. Volumes rendered in this manner have been likened to a translucent suspension of particles in 3D space.

2.3.2 Surface rendering

In surface rendering, the volumetric data must first be converted into geometric primitives, by a process such as isosurfacing, isocontouring, surface extraction or border following. These primitives (such as polygon meshes or contours) are then rendered for display using conventional geometric rendering techniques.

Figure 2.7: Example for volumerendering



2.3.3 Resumen

Both techniques have their advantages and disadvantages. So for example with the volume rendering you can display every stack of 2D images in 3D, without haveing any information about the size and the geometry of the phenomenon of interest, but it needs a lot of memory and processing power. The surface rendering on the other hand does not need a lot of memory nor processing to display, but it needs to implement one of the techniques to catch the surface of the body or volume we want to display. Therefore we need do process the image before. And finding the exact borders of the phenomenon also needs powerfull algorithms and a fine tune of the filters.

Thomas Bauer Visualisation


3 Visualisation

For the visualisation of the 2D and 3D images we are using Matlab with its different libraries and toolboxes (image processing toolbox, SDC morphology toolbox). Better than Matlab for the 3D visualisation are the libraries of the “Visualisation Toolbox” (VTK). These are much stronger and potent for the reconstruction and renderisation as Matlab. The libraries of VTK are applied with another programming language to implement them, for example you can write your source-code for a program in C++ and use the VTK libraries with their special features for image-processing.

3.1 Using Matlab

Matlab is a program originally designed to deal with matrices, but it is able to do a lot more than just some calculations with matrices. It possesses a lot of toolboxes to expand its capabilities. With the Simulink toolbox for example you can build mathematic models of filters and circuits and simulate them. Furthermore you have a lot of libraries such as the Filer Design-, Image Processing- and Optimization Toolbox. This toolboxes provide you with an abundance of ready-to-use functions, specially programmed for the given problems and you can easily use them and include them in your programs

3.1.1 Visualisation with Matlab

The confocal microscope provides us with a large file of raw data, normally a header and a stack of 76 image-slices. To visualize the single slices of the 3D stack or the whole volume with Matlab we have to process the following steps:

Visualisation Thomas Bauer


- You need to know where the file is saved, to open and to read the data. First we read the header of the file, the most imprtant information for us is to know how many slices are contained within the file and what dimensions every image has (pixels in x and y direction).

- If you want to show just a single slice, you need to know to where you have to “jump” if you don’t want to visualize the first slice of the image stack. The same situation accures if you want to display just a part of the volume.

- If you want to apply a filter or a set of filters to improove the image you have to be aware of waiting a bit, because Matlab is not the most powerfull program.

3.2 Using VTK

The Visualization ToolKit (VTK) is an open source, freely available software system for 3D computer graphics, image processing, and visualization. VTK includes a C++ class library and supports several interpreted interface layers including Tcl/Tk, Java, and Python. VTK has been implemented on nearly every Unix-based platform and PC's (Windows NT and Windows95). The design and implementation of the library has been strongly influenced by object-oriented principles.

The graphics model in VTK is at a higher level of abstraction than rendering libraries like OpenGL or PEX. This means it is much easier to create useful graphics and visualization applications. In VTK applications can be written directly in Tcl, Java, Python, or C++. In fact, using the interpreted languages Tcl or Python with Tk, and even Java with its GUI class libraries, it is possible to build useful applications quite fast.

Finally, the software is a true visualization system, it doesn't just let you visualize geometry. VTK supports a wide variety of visualization algorithms including scalar, vector, tensor, texture, and volumetric methods; and advanced modeling techniques like implicit modelling, polygon reduction, mesh smoothing, cutting, contouring, and Delaunay triangulation. Moreover, there are directly integrated dozens of imaging algorithms into the system so you can mix 2D imaging / 3D graphics algorithms and data.



3.2.1 Visualisation with VTK

To process any data VTK uses a so called data-pipeline. Once your raw data of interest entered in the pipelin, you can influence nearly everything you can imagine. To read in the data you have to choose the right reader for the given data set. Than you can influence the representation of the colors, you can apply various filters, you can define a type of rendering and much more. Below you can find an example of a possible pipeline with ist components.

3.2.2 Data-Pipeline of VTK:

As an example you can find below the data-pipeline from one of our programs to visualize a volume in 3D.

- Imagereader

- Transferfunction to eliminate the dark areas of the image

- Color-Transferfunction for the representation of the colors (grayscale, RGB)

- Volume-Mapper to render the data

- Volume to define how the representation will look like

- Window to display the volume

- Window-Interactor to change the properties of the window (eg: size, colour, light …)

3.3 Region of Interest

An important technique is to make use of the Region of interest (ROI). With this trick you are able to reduce a lot the needed time for processing an image. For example, if you are not taking in account the background of an image and you are applying a filter just on the phenomenons in the foreground you can safe from 50% up to 90% computing time.



3.3.1 Choosing a ROI

To choose the region of interest in our images, we are using various techniques. First we are applying thresholding on our image, to separate the dark backround from the light cells, this is very quick and it doesn’t need a lot of processing time. But with this technique you also define noise and other little light areas in the image as region of interest, that you are not really interestet in. Therefore we need further processing as opening/closing, delation with respect to the original image (or another image with more details, but with less noise) to keep fine structures of the original image without loosing them. Below I will give some examples of an original image and of the stepps of processing the region of interetst.

#

(a) (b)

(c) (d)

Figure 3.1: Set of images to find region of interest



Figure 3.1 shows our images that we are using to define the region of interest. Image (a) is the original slice of the image-stack, what is given by the confocal microscope. Image (b) is the first step to find the region of interest, it is produced out of image (a) with a threshold to change the gray-scale image into black and white, and with opening, to get rid of the small white particles. Image (c) is also generated by thresholding and opening from image (a), but here the threshold- level and the opening were a bit less, to keep the fine structure of the cell. The final image (d) is produced out of image (b) with dilation with respect to image (c). That means, that we made the white areas of image (b) bigger, but just in that directions and regions where the image (b) already has white spots. Therefore it is not a problem that we have more details in image (c) than we are interested in, we will not thake them in account.

With this region of interest (image (b)) we can process the original slice to enhance the quality. For example we can apply smoothin, a filter, or deconvolution just in the roi and we will save a lot of processor power and time.

Thomas Bauer Data-Processing


4 Data-Processing

In this section I will give a detailed explanation of our programs. Some of them are not completely developed so far and we will have to make some changes with them, but to have an overview they will be sufficient.

4.1 Display images with Matlab

With Matlab we are only displaying and processing images in 2D and 2½D, because so fare the processing in three dimensions takes much to long. As we are not yet using the regions of interest to apply filters and deconvolution, depending on the power of the computer you have to wait up to some minutes to see the result. For usefull applications this is to long, so we only use the VTK libraries with the programming language TCL/TK to display images in 3D.

4.1.1 Display a single slice

To display just a single slice out of the image stack of the *.PIC file we wrote the Matlab m-file “openpic.m”. With this program you can open any slice within the whole image-stack if you call the file-name and the slice-number. In the command window of Matlab the use of the m-file looks like this:

>> slice = openpic('nuevo1.pic', 46);

This command displays the slice number “46” of the file “nuevo1.pic” and returns the image in the variable “slice” for further use. To call the file just by writing its filename, it is necessary that the file is located in the working directory of Matlab. Below you can find the result window of this command.

Data-Processing Thomas Bauer


Following I will give the source-code of the Matlab function. Note, I have deleded all the comments and the help text, to make it a bit more clear and easier to read, I will make some explanations in the end of each section or in each line with the “%”-sign to mark comments:

4.1.1.1 Open the file and read the data: function[pic] = openpic(acFilename,nNum); %definition of the function fid = fopen(acFilename,'rt','l'); %file identifier nPictureSize = fread(fid,[1,2],'int16'); nX = nPictureSize(1,1); %width in pixels nY = nPictureSize(1,2); %higth in pixels nNumberOfImages = fread(fid,1,'int16'); xRampAndNotes = fread(fid,6,'int16'); acNametemp = fread(fid,[1,32],'uchar'); %filename by microscope acName = char(acNametemp); xTemp = fread(fid,26,'int8'); %to “go” to end of header

Figure 4.1: Example for “openpic.m”



In this section comes first the definition of the function with its name, vriables and parameters. In the second line we open the file named by “acFilename” for read acsess only, the data format is “little endian”. That means if you have a word (two bytes) that the left byte is the higher part of the word and the right byte is the lower part of the word. In the “big endian” format the right byte is the higher part. The first two words are the dimensions of the imagenes folowed by the number of slices and some information about the color, the filename and some more information (see figure 2.3). In the following part of the program we make use of the dimensions of the images and the number of the slices, so we save them in a variable.

4.1.1.2 Display the choosen image

if nNum == 1 pic = fread(fid,[nX,nY],'uint8'); mmshow(uint8(pic));

elseif (1 < nNum)& (nNum <= nNumberOfImages) for I = 1:nNum pic = fread(fid,[nX,nY],'uint8'); end mmshow(uint8(pic));

else display ('wrong image number'); end

fclose(fid);

Here we work with an if- loop to realize the selection of the desired slice. If you want to display the first image of the stack, we directely read the first slice and display it. If you want to read another than the first slice, we need to read all the slices before first without displaying them, to go to that point of the file where we can find the desired image, now we read this one and display it. If the number you typed in the command window is not within the borders of the stack, a warning (wrong image number) is displayed on the screen.

4.1.2 Display multiple slices

To display a set of slices out of the image stack of the *.PIC file we wrote the Matlab m-file “picmontage.m”. With this program you can open any set of slices within the whole image-stack if you call the file-name, the number of the first slice to display and the number of the last slice you want to display. In the command window of Matlab the use of the m-file looks like this:



>> set = picmontage('nuevo1.pic',40 , 43);

This command displays the slice numbers 40 to 43 of the file “nuevo1.pic” all in the same figure and returns the images in the variable “set” for further use. To call the file just by writing its filename, it is necessary that the file is located in the working directory of Matlab. Below you can find the result window of this command.


4.1.2.1 Open the file and read the data:

This part of the function is identical with the part for reading the header of the function “openpic.m” so I will not explain it again (see 4.1.1.1). The only difference is the definition of the function as we have different parameters and variables, but they are used only in the second part of the program.

Figure 4.2: Example for “picmontage.m”



function[pic] = picmontage(acFilename,nNum1,nNum2); fid = fopen(acFilename,'rt','l'); %file identifier nPictureSize = fread(fid,[1,2],'int16'); nX = nPictureSize(1,1); %width in pixels nY = nPictureSize(1,2); %higth in pixels nNumberOfImages = fread(fid,1,'int16'); xRampAndNotes = fread(fid,6,'int16'); acNametemp = fread(fid,[1,32],'uchar'); %filename by microscope acName = char(acNametemp); xTemp = fread(fid,26,'int8'); %to “go” to end of header

4.1.2.2 Display the choosen images

if nNum2 < (nNumberOfImages+1) if nNum1 == 1 for j=1:nNum2 pic(:,:,1,j) = fread(fid,[nX,nY],'uint8'); end subplot(1,1,1); montage(pic, [0,255]); elseif (1 < nNum1)& (nNum1 <= nNumberOfImages) for j=1:(nNum1-1) pic = fread(fid,[nX,nY],'uint8'); end for j=1:(1+nNum2-nNum1) pic(:,:,1,j) = fread(fid,[nX,nY],'uint8'); end subplot(1,1,1); montage(pic, [0,255]); else display ('wrong image number'); end else display ('number of last image is too big!'); end fclose(fid);

This set of cascaded if- loop is just that big to prevent the case that wrong image numbers are entered in the command line while calling the function.



The first if- loop checkes that the number of the last image to display is not to big. When it is to big, we go to the “else” section and a warning is displayed (number of last image is too big!) and the porgram will be terminated.

If the number is within the borders of the image stack we enter the second loop. When the first image is desired to be slice number 1, we directely read the choosen slices and display them. Note, the subplot is just used to fit the images better in the figure, as you can see, it only has one element.

If you want to start with another slice than number 1 we enter the elseif- loop and again we have to read with the first for- loop the slices we don’t want to display, with the second for- loop we read the desired slices and later display them.

In the case that we didn’t read any slices because the number of the first image to display is not within the right borders, another warning is displayed (wrong image number) and the program will be terminated.

4.1.3 Converting slices into a movie

To convert a set of slices into a movie we wrote the Matlab m-file “picmovie.m”. With this program you can convert any set of slices within the whole image-stack into a movie. In the command window you call the function with file-name, the number of the first slice and the number of the last slice you want to convert. In the command window of Matlab the use of the m-file looks like this:

>> film = picmovie('nuevo1.pic',30 , 43);

This command plays a movie from the slice numbers 30 to 43 of the file “nuevo1.pic” and returns the movie matrix in the variable “film” for further use. To call the file just by writing its filename, it is necessary that the file is located in the working directory of Matlab.




4.1.3.1 Open the file and read the data:

This part of the function is identical with the part for reading the header of the function “openpic.m” to read the explanation again see 4.1.1.1. The only difference is the definition of the function as we have different parameters and variables, but they are used only in the second part of the program.

function[picmov] = picmovie(acFilename,nNum1,nNum2); fid = fopen(acFilename,'rt','l'); %file identifier nPictureSize = fread(fid,[1,2],'int16'); nX = nPictureSize(1,1); %width in pixels nY = nPictureSize(1,2); %higth in pixels nNumberOfImages = fread(fid,1,'int16'); xRampAndNotes = fread(fid,6,'int16'); acNametemp = fread(fid,[1,32],'uchar'); %filename by microscope acName = char(acNametemp); xTemp = fread(fid,26,'int8'); %to “go” to end of header

4.1.3.2 Play the movie

if nNum2 < (nNumberOfImages+1) if nNum1 == 1 for j=1:nNum2 pic(:,:,1,j) = fread(fid,[nX,nY],'uint8'); end picmov = immovie(pic,gray(256)); movie(picmov, 0, 5); elseif (1 < nNum1)& (nNum1 <= nNumberOfImages) for j=1:(nNum1-1) pic = fread(fid,[nX,nY],'uint8'); end for j=1:(1+nNum2-nNum1) pic(:,:,1,j) = fread(fid,[nX,nY],'uint8'); end picmov = immovie(pic,gray(256)); movie(picmov, 0, 5); else display ('wrong image number'); end else display ('number of last image is too big!'); end fclose(fid);



The first if- loop checkes that the number of the last image to convert is not to big. When it is to big, we go to the “else” section and a warning is displayed (number of last image is too big!) and the porgram will be terminated.

If the number is within the borders of the image stack we enter the second loop. When the first image is desired to be slice number 1, we directely read the choosen slices, convert them into a movie matrix and play them.

If you want to start with another slice than number 1 we enter the elseif- loop and again we have to read with the first for- loop the slices we don’t want to convert, with the second for-loop we read the desired slices, convert them into a movie matrix and later play them.

In the case that we didn’t read any slices because the number of the first image to convert is not within the right borders, another warning is displayed (wrong image number) and the program will be terminated.

4.2 Display a volume with VTK

To display and to process the image stacks in three dimensions, the libraries of VTK are much better than Matlab. We implement the VTK libraries in the programs written with TCL/TK. This is much more comfortable as you don’t need to build, link and compile the source code to have the executable programs as in C++ programs.

Following I will show the sourcecode of the program “OpenPic.tcl” and I will explain the steps in the visualisation pipeline. The lines starting with “#” are commands and not a part of the sourcecode:

#This two lines are the standart head of each TCL file,these commands #are to call the VTK libraries and the program to execute the written #sourcecode. catch {load vtktcl} source vtkInt.tcl #Here comes the first part of the pipeline, the reader for the data. We can #set the filename, the dimensions of the imagestack, the ROI and the size #of the header, to know where the raw image data beginns



vtkImageReader reader reader SetFileName "nuevo1.pic" reader SetFileDimensionality 3 reader SetDataExtent 0 511 0 511 0 75 reader SetDataVOI 0 511 0 511 0 75 reader SetDataOrigin 0 0 0 reader SetDataScalarTypeToUnsignedChar reader SetDataSpacing 1.0 1.0 1.0 reader SetHeaderSize 76 #The following part is to cut off the dark parts of the images. With this #we make the separation between background and foreground. All levels #of gray between 0 and 60 are cut off. The rest (60 – 256) is not changed # Create transfer mapping scalar value to opacity vtkPiecewiseFunction opacityTransferFunction opacityTransferFunction AddPoint 60 1.0 opacityTransferFunction AddPoint 255 1.0 opacityTransferFunction ClampingOff #With this transferfunction we change the colormap from grayscale to #RGB. The colors stay levels of gray but they are represented with the #three parts of the RGB colors. # Create transfer mapping scalar value to color vtkColorTransferFunction colorTransferFunction colorTransferFunction AddRGBPoint 0.0 0.0 0.0 0.0 colorTransferFunction AddRGBPoint 255.0 1.0 1.0 1.0 #This hands over the properties we just defined to the VolumeProperty #function to combine them for a later use. # The property describes how the data will look vtkVolumeProperty volumeProperty volumeProperty SetColor colorTransferFunction volumeProperty SetScalarOpacity opacityTransferFunction volumeProperty SetInterpolationTypeToLinear #This step defines the render method, we are using a volume renderer. #This changes the single slices to a 3D volume vtkVolumeRayCastCompositeFunction compositeFunction vtkVolumeRayCastMapper volumeMapper volumeMapper SetVolumeRayCastFunction compositeFunction volumeMapper SetInput [ reader GetOutput]



# The volume holds the mapper (renderer) and the properties we defined #before and can be used to display and position the 3D image vtkVolume volume volume SetMapper volumeMapper volume SetProperty volumeProperty #This will display the image vtkRenderer ren1 ren1 AddVolume volume #This opens a new window and displays the volume vtkRenderWindow renwin renwin AddRenderer ren1 #Here we can change the size, the lights and the angle of view vtkRenderWindowInteractor iren iren SetRenderWindow renwin ren1 SetBackground 1 1 1 renwin SetSize 512 512 iren Initialize iren AddObserver SetExitMethod { exit } #This terminates the sourcecode wm withdraw . vwait forever



The result in the window will look like the following figure 4.3. You can move and rotate the volume by clicking with the mouse. Depending the computer, it can be amazingly fast to visualize and to rotate the 3D representation.

4.3 Call the VTK program from Matlab

Another feature we implement is to execute the TCL written programs using the VTK lebraries from the Matlab environment. With this we want to have the possibillity to open files with different filenames and we want to choose only a part of the volume (just a fraction of the entire image stack) without changing the source code in the program. For this we call the Matlab function “vtkopenpic”, we can enter the filename and the region of the slices we are interested in, pass this variables to the TCL program and open the desired region of the image stack in 3D.

To pass the variables to the TCL program we save them in Matlab as text files in the working directory and open them with the TCL program to read the written values and to define the filename of the *.PIC file and the slices to open.

Figure 4.3: Example for “openpic.tcl”



The Matlab m-file looks like this:

function[] = vtkopenpic(acFilename, nNum1, nNum2);

%Create new file for read the number of the first slice fid = fopen('acVariableTCL1.txt','wt'); fwrite(fid, sprintf('%d',nNum1),'char'); fclose(fid); %Create new file for read the number of the last slice fid = fopen('acVariableTCL2.txt','wt'); fwrite(fid, sprintf('%d',nNum2),'char'); fclose(fid); %Create new file for read the name of the file *.PIC to open fid = fopen('acFilename.txt','wt'); fwrite(fid, sprintf('%c',acFilename),'char'); fclose(fid); %Calls the tcl program, to open the image .pic !openpic3d.tcl &

We create a new text file (e.g. “acVariableTCL1”) whith write access, write the handed over variable in this text file and close the file. This we do for all three variables we want to pass to the TCL program.

In the TCL program compared to the original program (see 4.2) the use of the variables as text files needs just some changes in the sourcecode. I will show just the changes below, the rest of the program looks like the original code:

catch {load vtktcl} source vtkInt.tcl set fd1 [open "acVariableTCL1.txt" "r"] set fd2 [read $fd1] set fd3 [open "acVariableTCL2.txt" "r"] set fd4 [read $fd3] set file1 [open "acFilename.txt" "r"] set file2 [read $file1]



vtkImageReader reader reader SetFileName "$file2" reader SetFileDimensionality 3 reader SetDataExtent 0 511 0 511 0 75 reader SetDataVOI 0 511 0 511 $fd2 $fd4 reader SetDataOrigin 0 0 0 reader SetDataScalarTypeToUnsignedChar … … …

In stead of writing the filename directly in the line: ‘reader SetFileName “nuevo1.pic”’ we write the pointer to the variable: ‘$file2’ above we have been readin the filename out of the text file we saved by the Matlab file. The same are we doing with the number of the first and last slice we want to open. Whith this variables we change the line: ‘reader SetDataVOI’. We replaced the original numbers by the pointer to the variable and so we handed over the tree parameters and can the program can be executed with variables and we don’t have to change the sourcecode to open different slices.

Thomas Bauer Future enhancements


5 Future enhancementes

- We have to implement the region of interest in a way that we are not processing the whole image, so fare we are always processing the whole image and we need a lot of time to apply for example the deconvolution.

- We need to find a kind of deconvolution that fits our problems and that enhances the quality of the slices and the whole volume.

- There is a problem because the images at the beginning and at the end of the image-stack only have very little information and with the same filters we use for the middle of the stack, where is a lot of information, we loose the little structures.

- We have to implement various filters also in the TCL programs, so fare we have a lot of nois within the 3D representations.

- With the use of the ROI we have to realize the visualisation in 3D of the vomule with Matlab. Without ROI, Matlab is defenitely to slow to apply the filters or the deconvolution.

processing and 3d visualisation of biological ......images recorded via optical sectioning...

Documents