Beyond RedndashGreenndashBlue (RGB) Spectrum-Based ColorImaging Technology1

Masahiro YamaguchiImaging Science and Engineering Laboratory Tokyo Institute of Technology 4259-R2-56 Nagatsuta

Midori-ku Yokohama 226-8503 JapanE-mail guchiisltitechacjp

Hideaki HaneishiResearch Center for Frontier Medical Engineering Chiba University 1-33 Yayoi-cho Inage-ku Chiba

263-8522 Japan

Nagaaki OhyamaImaging Science and Engineering Laboratory Tokyo Institute of Technology 4259-R2-56 Nagatsuta

Midori-ku Yokohama 226-8503 Japan

Abstract This article presents a comprehensive study on thespectrum-based color reproduction system called Natural Vision(NV) which aims to break through the limitation of redndashgreenndashblue(RGB) three-primary schemes After a basic discussion on the mo-tivation for color imaging technology beyond RGB the method forsystematizing the multispectral and multiprimary color imaging tech-nologies including image capture processing storage printing anddisplay is presented Then experimental multispectral systems forboth still image and video are introduced and the following featuresof spectrum-based scheme are revealed a) highly accurate colorreproduction is possible even under different illumination environ-ment b) an expanded color gamut can be reproduced bymultiprimary color displays c) the influence of observer metamerismcan be reduced by the spectral color reproduction and d) the quan-titative spectral attributes of an object useful for its analysis or rec-ognition can be captured and preserved Finally the effectivenessof the system is also demonstrated through experiments in fields ofapplication such as medicine digital archives color printing elec-tronic commerce and computer graphics copy 2008 Society for Imag-ing Science and TechnologyDOI 102352JImagingSciTechnol2008521010201

INTRODUCTIONDigital images are widely used at present in digital broad-casting digital still cameras (DSCs) and image delivery overthe broadband network and are being applied to extendedfields such as telemedicine electronic commerce and elec-tronic museum The natural color reproduction is one of thekey issues in these applications in addition to high-resolution or large-screen display technologies Howevermany of the conventional color imaging systems are de-signed for user preference and thereby it is difficult to re-produce the original color of an object Color managementtechnology has greatly progressed especially in color print-

ing but there still remains a limitation in color reproducibil-ity employing redndashgreenndashblue (RGB) three primary colorsystems

An approach to break through this limitation is goingbeyond RGB ie using a spectrum-based system instead ofRGB It has been reported that the use of multispectral im-aging significantly improves the color accuracy1ndash5 In the dis-play industry the multiprimary color approach becomes oneof the choices for expanding color gamut6ndash12 but it is diffi-cult to take full advantage of multiprimary color technologybecause a wide gamut image source is not available for suchdisplays A total system based on spectral information hasnot yet been established A few papers13ndash17 were devoted tomultispectral systems including both input and output forhardcopy applications141617 or still image display13ndash15 butno systematized implementation of the multispectral andmultiprimary color imaging technologies has been reported

This article presents the concepts for systematizationimplementation and applications of video and still imagesystems with spectrum-based color image reproduction in-cluding multispectral image capture processing compres-sion transmission storage printing and display The systempresented in this article enables exploitation of the advan-tages of multispectral and multiprimary technologies In ad-dition conventional imaging devices based on RGB can bealso employed in the proposed spectrum-based systemthough the color reproduction accuracy is not very highWith the integrated model system the effectiveness of thespectrum-based technology is also demonstrated in variousfields of application The results shown in this article areprimarily obtained from the project Natural Vision (referredto as NV hereafter) an industry-government-academic jointproject (1999ndash2006) aimed at the development of a visualtelecommunication system that enables high reality imagereproduction with natural color18ndash21 Although some resultswere previously published this article comprehensively sum-

1Presented in part at ISampTSIDrsquos Fourteenth Color Imaging Conference

November 6ndash10 2006 Scottsdale AZ USA

Received Jan 19 2007 published online Jan 15 2008

1062-3701200852101020115$2000

Journal of Imaging Science and Technologyreg 52(1) 010201ndash010201-15 2008copy Society for Imaging Science and Technology 2008

010201-1

marizes the study on the spectrum-based system for colorimage reproduction

SPECTRUM-BASED COLOR REPRODUCTIONSYSTEMWhy is a Spectrum-Based System NeededLet us briefly review the limitations of conventional RGB-based systems in this section to make clear the advantage ofspectrum-based color reproduction

(1) The RGB values obtained in conventional systemsoften have different meanings depending on thedevice characteristics or color processing For ex-ample many conventional color imaging systemsare designed for user preference and the RGB val-ues do not represent objective color informationEven if colorimetric color calibration is appliedmost color cameras do not satisfy the Luther con-dition ie the spectral sensitivity is different fromthat of human vision22 Thus the RGB signal doesnot have one-to-one correspondence to thetristimulus values perceived by human vision If thespectral sensitivity is closer to that of human visionthe color fidelity is improved but noise behaviormay become worse Spectral considerations arethus essential in the color acquisition process

(2) The RGB or any other three primary color signalsthat comply with the color space such as sRGB orYCbCr are defined under a white point such as CIED65 or D50 standard illuminant When the il-luminant of the observation environment is differ-ent from that of image capture the color under thedifferent illuminant should reproduce the color as ifthe object were placed at the site of the observerWhite balance adjustment in conventional colorimaging is performed in RGB space sometimes in-troducing a color appearance model But the colo-rimetric accuracy is not high because the spectralreflectance of the object and the spectral distribu-tion of illuminant are required in principle in orderto calculate the color under the different il-luminant When images are displayed on softcopymonitors it is possible to reproduce the color un-der different illuminant based on spectrum-basedcolor conversion18 However there is an additionalissue the standard color spaces are defined for acertain standard illuminant and do not support thecolor under an arbitrary illuminant In thehardcopy applications moreover spectral printingis required to solve this problem of illuminantmetamerism23

(3) In the color image display the color gamut doesnot cover all the existent colors and some high-saturation colors cannot be reproduced To enlargethe color gamut the saturation of primary colorscan be increased but the gamut is still limitedwithin a triangle The multiprimary color ap-proach ie using more than three primary colors

was proposed for a larger color gamut6ndash8 Thegamut then becomes a polygon where its verticescorrespond to the color coordinates of the prima-ries or a polyhedron in three-dimensional colorspace Even if the display device allows the displayof a wider color gamut conventional color signalssuch as sRGB or ITU-R BT709 do not support awider gamut color signal Wide gamut color spaceshave recently become available24 such asAdobeRGB and xvYCC but most of the color inputdevices cannot capture high-saturation colors cor-rectly for the reason explained previously

(4) In highly accurate color reproduction the effect ofobserver metamerism cannot be ignored which hasits origin in the difference of color matching func-tions among individuals When the color displayedon a monitor is compared with the real objectssuch as printed materials the observed colors maydisagree with each other due to the observermetamerism effect even if the colorimetric accu-racy is high The multispectral and multiprimaryapproach solves this problem by means of spectralcolor reproduction1825

(5) In the image archive database or analysis the uti-lization of color information is limited since theRGB signal depends on the devices illuminantsand preprocesses involved in the imaging systemsFor example in the image retrieval using color in-formation the target object cannot be found if theillumination condition is different In contrast tothis the original attribute of an object that gener-ates color ie spectral radiance reflectance ortransmittance information is captured and pre-served with multispectral imaging The quantitativecolor information is useful for the analysis or therecognition of the object in the image database ofthe digital archives2627 of cultural heritage art-works and clinical cases in medicine Moreover theexploration of invisible features becomes possiblefrom spectral images

The spectrum-based color reproduction provides the solu-tion to these problems as shown in Figure 1 but it is nec-essary to integrate the system so as to take advantage of it inimage and video communication systems

BackgroundIt is known that spectral measurement provides accuratecolorimetry and the spectral color acquisition is also utilizedin imaging applications In addition to accurate imagingcolorimetry multispectral imaging has been employed forcomputational color constancy428 and color imageanalysis26272930

Although there have been numerous reports on theoret-ical and experimental aspects of multispectral imaging fewworks can be found on the system architecture for spectralcolor imaging including input and output A significant con-tribution to the system architecture was made by Keusen13

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-2 J Imaging Sci Technol 521Jan-Feb 2008

and Hill14 In the proposed architecture the spectral stimu-lus which is an output of the spectral scanner is expandedby a set of basis functions and encoded as K-channel coef-ficients for the basis functions The first three components ofthe encoded data correspond to the tristimulus values undera standard illuminant and the remaining K-3 channels rep-resent invisible spectral components It was suggested to em-ploy such spectral data for multispectral display andprinting14

Rosen et al revealed the concept of spectral color man-agement including both input and output and a spectralprofile connection space (PCS) was introduced instead ofthe conventional PCS used in the color managementsystem1516 The architecture mainly deals with the spectralprinting though an example of a spectral image visualiza-tion tool was reported15 In addition to reduce the dimen-sionality of the data for spectral printing an interim connec-tion space (ICS) was introduced called LABPQR17 wherethe first three channels (LAB) corresponds the color spaceunder standard illuminant and the additional three channels(PQR) corresponds to the ldquoinvisiblerdquo components The con-cept is quite similar to the one of Keusen and Hill but thespectral PCS is a more flexible approach for the interchangeof spectral color information

The spectrum-based color reproduction system de-scribed in the following section was also presented in thiscontext from the NV project1821 The architecture is similarto the spectral PCS proposed by Rosen et al and it is testedin various application fields of multispectral imaging

namely the color reproduction in both still image and videosystems multiprimary or spectral displays printing andspectral image analysis The architecture enables retentionand utilization of the information captured by multispectralimaging in every class of applications The feasibility of thearchitecture has been confirmed through experiments invarious application fields as described next One of the spe-cific features of the work in the context of the NV project isthe multispectral video system in which real-time color con-version transmission and display is realized using the six-band camera and the six-primary display as explained nextAnother feature is the demonstration of multiprimary dis-plays which makes it possible to reproduce high-reality im-ages with natural color and wide color gamut

How Does a Spectrum-Based System WorkThe concept of the spectrum-based color reproduction sys-tem is depicted in Figure 2 To obtain sufficient informationa multispectral camera (MSC) is desired as an input deviceThe profile of the input device including spectral sensitivitytone curve and dark current level of the camera and thespectral energy distribution of the illuminant are attached tomultispectral image (MSI) data In image processing trans-mission and storage the image is accompanied by the pro-file metadata so that the spectral radiance reflectance ortransmittance can be retrieved In the image display the im-age of tristimulus values or spectral radiance is reproducedon three-primary or multiprimary color displays with devicecalibration The system can also provide data to a color

Figure 1 The role of spectrum-based color reproduction in the video and still-image systems a Reproducingthe color as if the observer were at the remote site b reproducing the color as if object were placed at thesite of observer c expanding color gamut of the display the hexagon denotes the gamut of six-primaryprojector and d enabling the storage and analysis of the image with quantitative spectral information

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-3

printing system in the form of CIE XYZ tristimulus valuesor the spectral reflectance under D50 standard illuminantThe architecture of our spectrum-based color reproductionsystem is similar to the ICC (International Color Consor-tium) color management system but the PCS is based on aphysical model ie spectrum-based PCS (SPCS) instead ofa color appearance model This can be any CIEXYZ underarbitrary illumination spectral radiance or spectral reflec-tance (or transmittance) The information required for theforward or inverse transform to SPCS is held in the profiledata The profile data format was defined as NV image dataformat182131 The XML version NV format was also devel-oped for easier handling32 Table I shows the summary of thedata items used in the NV profile data by XML format

Figure 3 shows the schematic presentation of theworkflow of the spectrum-based color reproduction systemOnce a MSI is captured by an input device raw image dataand the characteristics of the input device are obtained Thespectrum of the illuminant can be also attached to the dataApplying the spectral estimation technique described inldquoSpectral Estimationrdquo the matrix to estimate the spectralreflectance transmittance or radiance is derived and at-tached to the profile data Artificial spectral image is alsosupported where it is accompanied by the characteristics ofthe virtual input device3334 or the spectral reconstructionmatrix It is also possible to assign the matrix to estimate thetristimulus values under a certain illuminant for conve-nience To make use of the stored MSI data afterward it isoften important to know the profile of the input device Forthis purpose the profile data as shown in Table I are at-tached to the image data in the database or in the system forinterchanging images In image display or printing the de-vice profile of the output device as well as the illuminationspectrum of the assumed output environment is presentedto the color conversion system By this system the colorreproduction under an arbitrary illumination environmentas shown in Fig 1(b) is possible while the standard il-luminant (such as CIE D65 or D50) can be specified foroutput to the conventional color display or printing devices

Accordingly the system is capable of output of the imagedata compatible with the standard color space

In this system the number of channels in the imagecapture and the display are independent and input and out-put devices with arbitrary numbers of channels can be em-ployed Color conversion can be performed without loss ofinformation contained in the MSI Three-channel devicescan also be used in this system with proper device charac-terization although there arise restrictions in the color re-production capability or accuracy

MULTISPECTRAL AND MULTIPRIMARYTECHNOLOGYMultispectral Image CaptureFor high-fidelity color reproduction with the spectrum-based color reproduction system it is crucial to increase thenumber of bands in the image input device There have beenvarious reports on devices for the acquisition of mulispectralimages2ndash527ndash3034 for example a monochrome camera with arotating filter wheel245 a grism (grating-prism)35 or a liq-uid crystal tunable filter30 Capturing a multichannel imagewith modulating illumination light is also a useful techniqueto obtain the spectral reflectance of an object26 The use ofdichroic mirrors to separate the spectral components pro-vides highest signal-to-noise ratio though the optical systembecomes relatively complex36 In the experiment explainednext the MSCs for still image and video developed in theNV project are used181937

Figure 4(a) shows the 16-band MSC with a rotatingfilter wheel developed in the NV project18 The color repro-ducibility evaluated using a GretagMacbeth color checker isshown in Figure 4(b) The accuracy by MSC is high and theerror is smaller than the discriminable level of human visionwhile visually apparent error is observed in three-band DSC

Figure 5(a) shows a six-band HDTV camera for theacquisition of motion picture37 In the six-band camera thelight is divided into two optical paths by a half mirror andincident on two conventional threendashcharge coupled device(CCD) cameras after transmission through the special color

Figure 2 The scheme of the spectrum-based color reproduction system

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-4 J Imaging Sci Technol 521Jan-Feb 2008

filters inserted in each path Six-channel uncompressedHDTV video signal is captured and recorded on a magnetichard disk As a result of evaluating the color reproducibilityof this six-band HDTV camera using the GretagMacbethcolor checker the CIELAB color difference average andmaximum Eab

=14 and 42 respectively where Eab =41

and 82 in the case of the conventional HDTV camera The

same Wiener estimation technique mentioned in the nextsection was used in the evaluation of both devices

Figure 5(b) shows the estimation results of high satura-tion colors In this experiment the interference filters of the16-band MSC were illuminated by a light box (DNPHi-Vision Color Viewer) and employed as test colors Thecolors were captured with the six-band camera and the

Table I Summary of the sample profile data in XML version of NV format The mark hellip represents the omitted numerical data

XML Description

Nvision xmlnsldquohttp hellip NvXmlSchemaxsdrdquoNvisionImageImageInfo d2p1ID=ldquo1rdquo

ImageTypeRAWIMAGEImageTypeImageWidth1920ImageWidthImageHeight-1080ImageHeightImageBands6ImageBandsBitSizePerBand16BitSizePerBandDataTypeUINT16DataType

ImageInfoNvisionImage

Basic information of image data

NvisionInput d2p1InputDate=ldquo2006-12-31T0000000000000+0900rdquo Input profile tag

DarkCurrentData d2p1ID=ldquo1rdquoDarkCurrentValue584229 hellip 511390DarkCurrentValue

DarkCurrentData

N-Component bias digital counts

ToneCurvesData d2p1ID=ldquo2rdquoCurveValues0 0 hellip 65535CurveValues

ToneCurvesData

N-Component tone reproduction curves

ExposureTimeData d2p1ID=ldquo3rdquoExposureTimeValue199 hellip 0998ExposureTimeValue

ExposureTimeData

Coefficients of N-componentsensitivity level for the correction ofexposure time setting

SpecSensiData d2p1ID=ldquo4rdquoSpecSensiValue 0000000 hellip 4269989SpecSensiValue

SpecSensiData

Spectral Sensitivity of P-channel inputdevice

NoiseData d2p1ID=ldquo5rdquoNoiseValue0699997 hellip 0699997NoiseValue

NoiseData

N-channel noise levels of Inputdevice

InputIllu d2p1ID=ldquo6rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquo

InputSpecData 0007034 hellip 0167847InputSpecDataInputIllu

Spectral radiance of Q-Inputilluminants

AutoCorrelationMatrix d2p1ID=ldquo7rdquoAutoCorrelationEigenValue0000000 hellip 0256989 AutoCorrelationEigenValueAutoCorrelationMatrixValue0000000 hellip 0099686 AutoCorrelationMatrixValue

AutoCorrelationMatrixNvisionInput

Principal components of objectrsquosspectral reflectance

NvisionConversiond2p1ConvDate=ldquo2006-12-31T0000000000000+0900rdquoXYZConvData d2p1ID=ldquo8rdquo

XYZConvValues0208511 hellip -0004578XYZConvValuesXYZConvData

Relative colorimetric valuesestimation matrix

SpecReflectData d2p1ID=ldquo9rdquoSpecReflectValues0642792 hellip 0000366SpecReflectValues

SpecReflectData

Relative spectral reflectanceestimation matrix

SpecStimuliData d2p1ID=ldquo10rdquoSpecStimuliValues0068008 hellip 0000031SpecStimuliValues

SpecStimuliData

Relative spectral radianceestimation matrix

ColorMatchingFuncData d2p1ID=ldquo11rdquoColorMatchingFuncValues0001358 hellip 0000000ColorMatchingFuncValues

ColorMatchingFuncData

RenderingIllu d2p1ID=ldquo12rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquoRenderingSpecData0007019 hellip 0167862RenderingSpecData

RenderingIlluNvisionConversion

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-5

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

marizes the study on the spectrum-based system for colorimage reproduction

SPECTRUM-BASED COLOR REPRODUCTIONSYSTEMWhy is a Spectrum-Based System NeededLet us briefly review the limitations of conventional RGB-based systems in this section to make clear the advantage ofspectrum-based color reproduction

(1) The RGB values obtained in conventional systemsoften have different meanings depending on thedevice characteristics or color processing For ex-ample many conventional color imaging systemsare designed for user preference and the RGB val-ues do not represent objective color informationEven if colorimetric color calibration is appliedmost color cameras do not satisfy the Luther con-dition ie the spectral sensitivity is different fromthat of human vision22 Thus the RGB signal doesnot have one-to-one correspondence to thetristimulus values perceived by human vision If thespectral sensitivity is closer to that of human visionthe color fidelity is improved but noise behaviormay become worse Spectral considerations arethus essential in the color acquisition process

(2) The RGB or any other three primary color signalsthat comply with the color space such as sRGB orYCbCr are defined under a white point such as CIED65 or D50 standard illuminant When the il-luminant of the observation environment is differ-ent from that of image capture the color under thedifferent illuminant should reproduce the color as ifthe object were placed at the site of the observerWhite balance adjustment in conventional colorimaging is performed in RGB space sometimes in-troducing a color appearance model But the colo-rimetric accuracy is not high because the spectralreflectance of the object and the spectral distribu-tion of illuminant are required in principle in orderto calculate the color under the different il-luminant When images are displayed on softcopymonitors it is possible to reproduce the color un-der different illuminant based on spectrum-basedcolor conversion18 However there is an additionalissue the standard color spaces are defined for acertain standard illuminant and do not support thecolor under an arbitrary illuminant In thehardcopy applications moreover spectral printingis required to solve this problem of illuminantmetamerism23

(3) In the color image display the color gamut doesnot cover all the existent colors and some high-saturation colors cannot be reproduced To enlargethe color gamut the saturation of primary colorscan be increased but the gamut is still limitedwithin a triangle The multiprimary color ap-proach ie using more than three primary colors

was proposed for a larger color gamut6ndash8 Thegamut then becomes a polygon where its verticescorrespond to the color coordinates of the prima-ries or a polyhedron in three-dimensional colorspace Even if the display device allows the displayof a wider color gamut conventional color signalssuch as sRGB or ITU-R BT709 do not support awider gamut color signal Wide gamut color spaceshave recently become available24 such asAdobeRGB and xvYCC but most of the color inputdevices cannot capture high-saturation colors cor-rectly for the reason explained previously

(4) In highly accurate color reproduction the effect ofobserver metamerism cannot be ignored which hasits origin in the difference of color matching func-tions among individuals When the color displayedon a monitor is compared with the real objectssuch as printed materials the observed colors maydisagree with each other due to the observermetamerism effect even if the colorimetric accu-racy is high The multispectral and multiprimaryapproach solves this problem by means of spectralcolor reproduction1825

(5) In the image archive database or analysis the uti-lization of color information is limited since theRGB signal depends on the devices illuminantsand preprocesses involved in the imaging systemsFor example in the image retrieval using color in-formation the target object cannot be found if theillumination condition is different In contrast tothis the original attribute of an object that gener-ates color ie spectral radiance reflectance ortransmittance information is captured and pre-served with multispectral imaging The quantitativecolor information is useful for the analysis or therecognition of the object in the image database ofthe digital archives2627 of cultural heritage art-works and clinical cases in medicine Moreover theexploration of invisible features becomes possiblefrom spectral images

The spectrum-based color reproduction provides the solu-tion to these problems as shown in Figure 1 but it is nec-essary to integrate the system so as to take advantage of it inimage and video communication systems

BackgroundIt is known that spectral measurement provides accuratecolorimetry and the spectral color acquisition is also utilizedin imaging applications In addition to accurate imagingcolorimetry multispectral imaging has been employed forcomputational color constancy428 and color imageanalysis26272930

Although there have been numerous reports on theoret-ical and experimental aspects of multispectral imaging fewworks can be found on the system architecture for spectralcolor imaging including input and output A significant con-tribution to the system architecture was made by Keusen13

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-2 J Imaging Sci Technol 521Jan-Feb 2008

and Hill14 In the proposed architecture the spectral stimu-lus which is an output of the spectral scanner is expandedby a set of basis functions and encoded as K-channel coef-ficients for the basis functions The first three components ofthe encoded data correspond to the tristimulus values undera standard illuminant and the remaining K-3 channels rep-resent invisible spectral components It was suggested to em-ploy such spectral data for multispectral display andprinting14

Rosen et al revealed the concept of spectral color man-agement including both input and output and a spectralprofile connection space (PCS) was introduced instead ofthe conventional PCS used in the color managementsystem1516 The architecture mainly deals with the spectralprinting though an example of a spectral image visualiza-tion tool was reported15 In addition to reduce the dimen-sionality of the data for spectral printing an interim connec-tion space (ICS) was introduced called LABPQR17 wherethe first three channels (LAB) corresponds the color spaceunder standard illuminant and the additional three channels(PQR) corresponds to the ldquoinvisiblerdquo components The con-cept is quite similar to the one of Keusen and Hill but thespectral PCS is a more flexible approach for the interchangeof spectral color information

The spectrum-based color reproduction system de-scribed in the following section was also presented in thiscontext from the NV project1821 The architecture is similarto the spectral PCS proposed by Rosen et al and it is testedin various application fields of multispectral imaging

namely the color reproduction in both still image and videosystems multiprimary or spectral displays printing andspectral image analysis The architecture enables retentionand utilization of the information captured by multispectralimaging in every class of applications The feasibility of thearchitecture has been confirmed through experiments invarious application fields as described next One of the spe-cific features of the work in the context of the NV project isthe multispectral video system in which real-time color con-version transmission and display is realized using the six-band camera and the six-primary display as explained nextAnother feature is the demonstration of multiprimary dis-plays which makes it possible to reproduce high-reality im-ages with natural color and wide color gamut

How Does a Spectrum-Based System WorkThe concept of the spectrum-based color reproduction sys-tem is depicted in Figure 2 To obtain sufficient informationa multispectral camera (MSC) is desired as an input deviceThe profile of the input device including spectral sensitivitytone curve and dark current level of the camera and thespectral energy distribution of the illuminant are attached tomultispectral image (MSI) data In image processing trans-mission and storage the image is accompanied by the pro-file metadata so that the spectral radiance reflectance ortransmittance can be retrieved In the image display the im-age of tristimulus values or spectral radiance is reproducedon three-primary or multiprimary color displays with devicecalibration The system can also provide data to a color

Figure 1 The role of spectrum-based color reproduction in the video and still-image systems a Reproducingthe color as if the observer were at the remote site b reproducing the color as if object were placed at thesite of observer c expanding color gamut of the display the hexagon denotes the gamut of six-primaryprojector and d enabling the storage and analysis of the image with quantitative spectral information

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-3

printing system in the form of CIE XYZ tristimulus valuesor the spectral reflectance under D50 standard illuminantThe architecture of our spectrum-based color reproductionsystem is similar to the ICC (International Color Consor-tium) color management system but the PCS is based on aphysical model ie spectrum-based PCS (SPCS) instead ofa color appearance model This can be any CIEXYZ underarbitrary illumination spectral radiance or spectral reflec-tance (or transmittance) The information required for theforward or inverse transform to SPCS is held in the profiledata The profile data format was defined as NV image dataformat182131 The XML version NV format was also devel-oped for easier handling32 Table I shows the summary of thedata items used in the NV profile data by XML format

Figure 3 shows the schematic presentation of theworkflow of the spectrum-based color reproduction systemOnce a MSI is captured by an input device raw image dataand the characteristics of the input device are obtained Thespectrum of the illuminant can be also attached to the dataApplying the spectral estimation technique described inldquoSpectral Estimationrdquo the matrix to estimate the spectralreflectance transmittance or radiance is derived and at-tached to the profile data Artificial spectral image is alsosupported where it is accompanied by the characteristics ofthe virtual input device3334 or the spectral reconstructionmatrix It is also possible to assign the matrix to estimate thetristimulus values under a certain illuminant for conve-nience To make use of the stored MSI data afterward it isoften important to know the profile of the input device Forthis purpose the profile data as shown in Table I are at-tached to the image data in the database or in the system forinterchanging images In image display or printing the de-vice profile of the output device as well as the illuminationspectrum of the assumed output environment is presentedto the color conversion system By this system the colorreproduction under an arbitrary illumination environmentas shown in Fig 1(b) is possible while the standard il-luminant (such as CIE D65 or D50) can be specified foroutput to the conventional color display or printing devices

Accordingly the system is capable of output of the imagedata compatible with the standard color space

In this system the number of channels in the imagecapture and the display are independent and input and out-put devices with arbitrary numbers of channels can be em-ployed Color conversion can be performed without loss ofinformation contained in the MSI Three-channel devicescan also be used in this system with proper device charac-terization although there arise restrictions in the color re-production capability or accuracy

MULTISPECTRAL AND MULTIPRIMARYTECHNOLOGYMultispectral Image CaptureFor high-fidelity color reproduction with the spectrum-based color reproduction system it is crucial to increase thenumber of bands in the image input device There have beenvarious reports on devices for the acquisition of mulispectralimages2ndash527ndash3034 for example a monochrome camera with arotating filter wheel245 a grism (grating-prism)35 or a liq-uid crystal tunable filter30 Capturing a multichannel imagewith modulating illumination light is also a useful techniqueto obtain the spectral reflectance of an object26 The use ofdichroic mirrors to separate the spectral components pro-vides highest signal-to-noise ratio though the optical systembecomes relatively complex36 In the experiment explainednext the MSCs for still image and video developed in theNV project are used181937

Figure 4(a) shows the 16-band MSC with a rotatingfilter wheel developed in the NV project18 The color repro-ducibility evaluated using a GretagMacbeth color checker isshown in Figure 4(b) The accuracy by MSC is high and theerror is smaller than the discriminable level of human visionwhile visually apparent error is observed in three-band DSC

Figure 5(a) shows a six-band HDTV camera for theacquisition of motion picture37 In the six-band camera thelight is divided into two optical paths by a half mirror andincident on two conventional threendashcharge coupled device(CCD) cameras after transmission through the special color

Figure 2 The scheme of the spectrum-based color reproduction system

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-4 J Imaging Sci Technol 521Jan-Feb 2008

filters inserted in each path Six-channel uncompressedHDTV video signal is captured and recorded on a magnetichard disk As a result of evaluating the color reproducibilityof this six-band HDTV camera using the GretagMacbethcolor checker the CIELAB color difference average andmaximum Eab

=14 and 42 respectively where Eab =41

and 82 in the case of the conventional HDTV camera The

same Wiener estimation technique mentioned in the nextsection was used in the evaluation of both devices

Figure 5(b) shows the estimation results of high satura-tion colors In this experiment the interference filters of the16-band MSC were illuminated by a light box (DNPHi-Vision Color Viewer) and employed as test colors Thecolors were captured with the six-band camera and the

Table I Summary of the sample profile data in XML version of NV format The mark hellip represents the omitted numerical data

XML Description

Nvision xmlnsldquohttp hellip NvXmlSchemaxsdrdquoNvisionImageImageInfo d2p1ID=ldquo1rdquo

ImageTypeRAWIMAGEImageTypeImageWidth1920ImageWidthImageHeight-1080ImageHeightImageBands6ImageBandsBitSizePerBand16BitSizePerBandDataTypeUINT16DataType

ImageInfoNvisionImage

Basic information of image data

NvisionInput d2p1InputDate=ldquo2006-12-31T0000000000000+0900rdquo Input profile tag

DarkCurrentData d2p1ID=ldquo1rdquoDarkCurrentValue584229 hellip 511390DarkCurrentValue

DarkCurrentData

N-Component bias digital counts

ToneCurvesData d2p1ID=ldquo2rdquoCurveValues0 0 hellip 65535CurveValues

ToneCurvesData

N-Component tone reproduction curves

ExposureTimeData d2p1ID=ldquo3rdquoExposureTimeValue199 hellip 0998ExposureTimeValue

ExposureTimeData

Coefficients of N-componentsensitivity level for the correction ofexposure time setting

SpecSensiData d2p1ID=ldquo4rdquoSpecSensiValue 0000000 hellip 4269989SpecSensiValue

SpecSensiData

Spectral Sensitivity of P-channel inputdevice

NoiseData d2p1ID=ldquo5rdquoNoiseValue0699997 hellip 0699997NoiseValue

NoiseData

N-channel noise levels of Inputdevice

InputIllu d2p1ID=ldquo6rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquo

InputSpecData 0007034 hellip 0167847InputSpecDataInputIllu

Spectral radiance of Q-Inputilluminants

AutoCorrelationMatrix d2p1ID=ldquo7rdquoAutoCorrelationEigenValue0000000 hellip 0256989 AutoCorrelationEigenValueAutoCorrelationMatrixValue0000000 hellip 0099686 AutoCorrelationMatrixValue

AutoCorrelationMatrixNvisionInput

Principal components of objectrsquosspectral reflectance

NvisionConversiond2p1ConvDate=ldquo2006-12-31T0000000000000+0900rdquoXYZConvData d2p1ID=ldquo8rdquo

XYZConvValues0208511 hellip -0004578XYZConvValuesXYZConvData

Relative colorimetric valuesestimation matrix

SpecReflectData d2p1ID=ldquo9rdquoSpecReflectValues0642792 hellip 0000366SpecReflectValues

SpecReflectData

Relative spectral reflectanceestimation matrix

SpecStimuliData d2p1ID=ldquo10rdquoSpecStimuliValues0068008 hellip 0000031SpecStimuliValues

SpecStimuliData

Relative spectral radianceestimation matrix

ColorMatchingFuncData d2p1ID=ldquo11rdquoColorMatchingFuncValues0001358 hellip 0000000ColorMatchingFuncValues

ColorMatchingFuncData

RenderingIllu d2p1ID=ldquo12rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquoRenderingSpecData0007019 hellip 0167862RenderingSpecData

RenderingIlluNvisionConversion

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-5

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

and Hill14 In the proposed architecture the spectral stimu-lus which is an output of the spectral scanner is expandedby a set of basis functions and encoded as K-channel coef-ficients for the basis functions The first three components ofthe encoded data correspond to the tristimulus values undera standard illuminant and the remaining K-3 channels rep-resent invisible spectral components It was suggested to em-ploy such spectral data for multispectral display andprinting14

Rosen et al revealed the concept of spectral color man-agement including both input and output and a spectralprofile connection space (PCS) was introduced instead ofthe conventional PCS used in the color managementsystem1516 The architecture mainly deals with the spectralprinting though an example of a spectral image visualiza-tion tool was reported15 In addition to reduce the dimen-sionality of the data for spectral printing an interim connec-tion space (ICS) was introduced called LABPQR17 wherethe first three channels (LAB) corresponds the color spaceunder standard illuminant and the additional three channels(PQR) corresponds to the ldquoinvisiblerdquo components The con-cept is quite similar to the one of Keusen and Hill but thespectral PCS is a more flexible approach for the interchangeof spectral color information

The spectrum-based color reproduction system de-scribed in the following section was also presented in thiscontext from the NV project1821 The architecture is similarto the spectral PCS proposed by Rosen et al and it is testedin various application fields of multispectral imaging

namely the color reproduction in both still image and videosystems multiprimary or spectral displays printing andspectral image analysis The architecture enables retentionand utilization of the information captured by multispectralimaging in every class of applications The feasibility of thearchitecture has been confirmed through experiments invarious application fields as described next One of the spe-cific features of the work in the context of the NV project isthe multispectral video system in which real-time color con-version transmission and display is realized using the six-band camera and the six-primary display as explained nextAnother feature is the demonstration of multiprimary dis-plays which makes it possible to reproduce high-reality im-ages with natural color and wide color gamut

How Does a Spectrum-Based System WorkThe concept of the spectrum-based color reproduction sys-tem is depicted in Figure 2 To obtain sufficient informationa multispectral camera (MSC) is desired as an input deviceThe profile of the input device including spectral sensitivitytone curve and dark current level of the camera and thespectral energy distribution of the illuminant are attached tomultispectral image (MSI) data In image processing trans-mission and storage the image is accompanied by the pro-file metadata so that the spectral radiance reflectance ortransmittance can be retrieved In the image display the im-age of tristimulus values or spectral radiance is reproducedon three-primary or multiprimary color displays with devicecalibration The system can also provide data to a color

Figure 1 The role of spectrum-based color reproduction in the video and still-image systems a Reproducingthe color as if the observer were at the remote site b reproducing the color as if object were placed at thesite of observer c expanding color gamut of the display the hexagon denotes the gamut of six-primaryprojector and d enabling the storage and analysis of the image with quantitative spectral information

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-3

printing system in the form of CIE XYZ tristimulus valuesor the spectral reflectance under D50 standard illuminantThe architecture of our spectrum-based color reproductionsystem is similar to the ICC (International Color Consor-tium) color management system but the PCS is based on aphysical model ie spectrum-based PCS (SPCS) instead ofa color appearance model This can be any CIEXYZ underarbitrary illumination spectral radiance or spectral reflec-tance (or transmittance) The information required for theforward or inverse transform to SPCS is held in the profiledata The profile data format was defined as NV image dataformat182131 The XML version NV format was also devel-oped for easier handling32 Table I shows the summary of thedata items used in the NV profile data by XML format

Figure 3 shows the schematic presentation of theworkflow of the spectrum-based color reproduction systemOnce a MSI is captured by an input device raw image dataand the characteristics of the input device are obtained Thespectrum of the illuminant can be also attached to the dataApplying the spectral estimation technique described inldquoSpectral Estimationrdquo the matrix to estimate the spectralreflectance transmittance or radiance is derived and at-tached to the profile data Artificial spectral image is alsosupported where it is accompanied by the characteristics ofthe virtual input device3334 or the spectral reconstructionmatrix It is also possible to assign the matrix to estimate thetristimulus values under a certain illuminant for conve-nience To make use of the stored MSI data afterward it isoften important to know the profile of the input device Forthis purpose the profile data as shown in Table I are at-tached to the image data in the database or in the system forinterchanging images In image display or printing the de-vice profile of the output device as well as the illuminationspectrum of the assumed output environment is presentedto the color conversion system By this system the colorreproduction under an arbitrary illumination environmentas shown in Fig 1(b) is possible while the standard il-luminant (such as CIE D65 or D50) can be specified foroutput to the conventional color display or printing devices

Accordingly the system is capable of output of the imagedata compatible with the standard color space

In this system the number of channels in the imagecapture and the display are independent and input and out-put devices with arbitrary numbers of channels can be em-ployed Color conversion can be performed without loss ofinformation contained in the MSI Three-channel devicescan also be used in this system with proper device charac-terization although there arise restrictions in the color re-production capability or accuracy

MULTISPECTRAL AND MULTIPRIMARYTECHNOLOGYMultispectral Image CaptureFor high-fidelity color reproduction with the spectrum-based color reproduction system it is crucial to increase thenumber of bands in the image input device There have beenvarious reports on devices for the acquisition of mulispectralimages2ndash527ndash3034 for example a monochrome camera with arotating filter wheel245 a grism (grating-prism)35 or a liq-uid crystal tunable filter30 Capturing a multichannel imagewith modulating illumination light is also a useful techniqueto obtain the spectral reflectance of an object26 The use ofdichroic mirrors to separate the spectral components pro-vides highest signal-to-noise ratio though the optical systembecomes relatively complex36 In the experiment explainednext the MSCs for still image and video developed in theNV project are used181937

Figure 4(a) shows the 16-band MSC with a rotatingfilter wheel developed in the NV project18 The color repro-ducibility evaluated using a GretagMacbeth color checker isshown in Figure 4(b) The accuracy by MSC is high and theerror is smaller than the discriminable level of human visionwhile visually apparent error is observed in three-band DSC

Figure 5(a) shows a six-band HDTV camera for theacquisition of motion picture37 In the six-band camera thelight is divided into two optical paths by a half mirror andincident on two conventional threendashcharge coupled device(CCD) cameras after transmission through the special color

Figure 2 The scheme of the spectrum-based color reproduction system

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-4 J Imaging Sci Technol 521Jan-Feb 2008

filters inserted in each path Six-channel uncompressedHDTV video signal is captured and recorded on a magnetichard disk As a result of evaluating the color reproducibilityof this six-band HDTV camera using the GretagMacbethcolor checker the CIELAB color difference average andmaximum Eab

=14 and 42 respectively where Eab =41

and 82 in the case of the conventional HDTV camera The

same Wiener estimation technique mentioned in the nextsection was used in the evaluation of both devices

Figure 5(b) shows the estimation results of high satura-tion colors In this experiment the interference filters of the16-band MSC were illuminated by a light box (DNPHi-Vision Color Viewer) and employed as test colors Thecolors were captured with the six-band camera and the

Table I Summary of the sample profile data in XML version of NV format The mark hellip represents the omitted numerical data

XML Description

Nvision xmlnsldquohttp hellip NvXmlSchemaxsdrdquoNvisionImageImageInfo d2p1ID=ldquo1rdquo

ImageTypeRAWIMAGEImageTypeImageWidth1920ImageWidthImageHeight-1080ImageHeightImageBands6ImageBandsBitSizePerBand16BitSizePerBandDataTypeUINT16DataType

ImageInfoNvisionImage

Basic information of image data

NvisionInput d2p1InputDate=ldquo2006-12-31T0000000000000+0900rdquo Input profile tag

DarkCurrentData d2p1ID=ldquo1rdquoDarkCurrentValue584229 hellip 511390DarkCurrentValue

DarkCurrentData

N-Component bias digital counts

ToneCurvesData d2p1ID=ldquo2rdquoCurveValues0 0 hellip 65535CurveValues

ToneCurvesData

N-Component tone reproduction curves

ExposureTimeData d2p1ID=ldquo3rdquoExposureTimeValue199 hellip 0998ExposureTimeValue

ExposureTimeData

Coefficients of N-componentsensitivity level for the correction ofexposure time setting

SpecSensiData d2p1ID=ldquo4rdquoSpecSensiValue 0000000 hellip 4269989SpecSensiValue

SpecSensiData

Spectral Sensitivity of P-channel inputdevice

NoiseData d2p1ID=ldquo5rdquoNoiseValue0699997 hellip 0699997NoiseValue

NoiseData

N-channel noise levels of Inputdevice

InputIllu d2p1ID=ldquo6rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquo

InputSpecData 0007034 hellip 0167847InputSpecDataInputIllu

Spectral radiance of Q-Inputilluminants

AutoCorrelationMatrix d2p1ID=ldquo7rdquoAutoCorrelationEigenValue0000000 hellip 0256989 AutoCorrelationEigenValueAutoCorrelationMatrixValue0000000 hellip 0099686 AutoCorrelationMatrixValue

AutoCorrelationMatrixNvisionInput

Principal components of objectrsquosspectral reflectance

NvisionConversiond2p1ConvDate=ldquo2006-12-31T0000000000000+0900rdquoXYZConvData d2p1ID=ldquo8rdquo

XYZConvValues0208511 hellip -0004578XYZConvValuesXYZConvData

Relative colorimetric valuesestimation matrix

SpecReflectData d2p1ID=ldquo9rdquoSpecReflectValues0642792 hellip 0000366SpecReflectValues

SpecReflectData

Relative spectral reflectanceestimation matrix

SpecStimuliData d2p1ID=ldquo10rdquoSpecStimuliValues0068008 hellip 0000031SpecStimuliValues

SpecStimuliData

Relative spectral radianceestimation matrix

ColorMatchingFuncData d2p1ID=ldquo11rdquoColorMatchingFuncValues0001358 hellip 0000000ColorMatchingFuncValues

ColorMatchingFuncData

RenderingIllu d2p1ID=ldquo12rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquoRenderingSpecData0007019 hellip 0167862RenderingSpecData

RenderingIlluNvisionConversion

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-5

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

printing system in the form of CIE XYZ tristimulus valuesor the spectral reflectance under D50 standard illuminantThe architecture of our spectrum-based color reproductionsystem is similar to the ICC (International Color Consor-tium) color management system but the PCS is based on aphysical model ie spectrum-based PCS (SPCS) instead ofa color appearance model This can be any CIEXYZ underarbitrary illumination spectral radiance or spectral reflec-tance (or transmittance) The information required for theforward or inverse transform to SPCS is held in the profiledata The profile data format was defined as NV image dataformat182131 The XML version NV format was also devel-oped for easier handling32 Table I shows the summary of thedata items used in the NV profile data by XML format

Figure 3 shows the schematic presentation of theworkflow of the spectrum-based color reproduction systemOnce a MSI is captured by an input device raw image dataand the characteristics of the input device are obtained Thespectrum of the illuminant can be also attached to the dataApplying the spectral estimation technique described inldquoSpectral Estimationrdquo the matrix to estimate the spectralreflectance transmittance or radiance is derived and at-tached to the profile data Artificial spectral image is alsosupported where it is accompanied by the characteristics ofthe virtual input device3334 or the spectral reconstructionmatrix It is also possible to assign the matrix to estimate thetristimulus values under a certain illuminant for conve-nience To make use of the stored MSI data afterward it isoften important to know the profile of the input device Forthis purpose the profile data as shown in Table I are at-tached to the image data in the database or in the system forinterchanging images In image display or printing the de-vice profile of the output device as well as the illuminationspectrum of the assumed output environment is presentedto the color conversion system By this system the colorreproduction under an arbitrary illumination environmentas shown in Fig 1(b) is possible while the standard il-luminant (such as CIE D65 or D50) can be specified foroutput to the conventional color display or printing devices

Accordingly the system is capable of output of the imagedata compatible with the standard color space

In this system the number of channels in the imagecapture and the display are independent and input and out-put devices with arbitrary numbers of channels can be em-ployed Color conversion can be performed without loss ofinformation contained in the MSI Three-channel devicescan also be used in this system with proper device charac-terization although there arise restrictions in the color re-production capability or accuracy

MULTISPECTRAL AND MULTIPRIMARYTECHNOLOGYMultispectral Image CaptureFor high-fidelity color reproduction with the spectrum-based color reproduction system it is crucial to increase thenumber of bands in the image input device There have beenvarious reports on devices for the acquisition of mulispectralimages2ndash527ndash3034 for example a monochrome camera with arotating filter wheel245 a grism (grating-prism)35 or a liq-uid crystal tunable filter30 Capturing a multichannel imagewith modulating illumination light is also a useful techniqueto obtain the spectral reflectance of an object26 The use ofdichroic mirrors to separate the spectral components pro-vides highest signal-to-noise ratio though the optical systembecomes relatively complex36 In the experiment explainednext the MSCs for still image and video developed in theNV project are used181937

Figure 4(a) shows the 16-band MSC with a rotatingfilter wheel developed in the NV project18 The color repro-ducibility evaluated using a GretagMacbeth color checker isshown in Figure 4(b) The accuracy by MSC is high and theerror is smaller than the discriminable level of human visionwhile visually apparent error is observed in three-band DSC

Figure 5(a) shows a six-band HDTV camera for theacquisition of motion picture37 In the six-band camera thelight is divided into two optical paths by a half mirror andincident on two conventional threendashcharge coupled device(CCD) cameras after transmission through the special color

Figure 2 The scheme of the spectrum-based color reproduction system

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-4 J Imaging Sci Technol 521Jan-Feb 2008

filters inserted in each path Six-channel uncompressedHDTV video signal is captured and recorded on a magnetichard disk As a result of evaluating the color reproducibilityof this six-band HDTV camera using the GretagMacbethcolor checker the CIELAB color difference average andmaximum Eab

=14 and 42 respectively where Eab =41

and 82 in the case of the conventional HDTV camera The

same Wiener estimation technique mentioned in the nextsection was used in the evaluation of both devices

Figure 5(b) shows the estimation results of high satura-tion colors In this experiment the interference filters of the16-band MSC were illuminated by a light box (DNPHi-Vision Color Viewer) and employed as test colors Thecolors were captured with the six-band camera and the

Table I Summary of the sample profile data in XML version of NV format The mark hellip represents the omitted numerical data

XML Description

Nvision xmlnsldquohttp hellip NvXmlSchemaxsdrdquoNvisionImageImageInfo d2p1ID=ldquo1rdquo

ImageTypeRAWIMAGEImageTypeImageWidth1920ImageWidthImageHeight-1080ImageHeightImageBands6ImageBandsBitSizePerBand16BitSizePerBandDataTypeUINT16DataType

ImageInfoNvisionImage

Basic information of image data

NvisionInput d2p1InputDate=ldquo2006-12-31T0000000000000+0900rdquo Input profile tag

DarkCurrentData d2p1ID=ldquo1rdquoDarkCurrentValue584229 hellip 511390DarkCurrentValue

DarkCurrentData

N-Component bias digital counts

ToneCurvesData d2p1ID=ldquo2rdquoCurveValues0 0 hellip 65535CurveValues

ToneCurvesData

N-Component tone reproduction curves

ExposureTimeData d2p1ID=ldquo3rdquoExposureTimeValue199 hellip 0998ExposureTimeValue

ExposureTimeData

Coefficients of N-componentsensitivity level for the correction ofexposure time setting

SpecSensiData d2p1ID=ldquo4rdquoSpecSensiValue 0000000 hellip 4269989SpecSensiValue

SpecSensiData

Spectral Sensitivity of P-channel inputdevice

NoiseData d2p1ID=ldquo5rdquoNoiseValue0699997 hellip 0699997NoiseValue

NoiseData

N-channel noise levels of Inputdevice

InputIllu d2p1ID=ldquo6rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquo

InputSpecData 0007034 hellip 0167847InputSpecDataInputIllu

Spectral radiance of Q-Inputilluminants

AutoCorrelationMatrix d2p1ID=ldquo7rdquoAutoCorrelationEigenValue0000000 hellip 0256989 AutoCorrelationEigenValueAutoCorrelationMatrixValue0000000 hellip 0099686 AutoCorrelationMatrixValue

AutoCorrelationMatrixNvisionInput

Principal components of objectrsquosspectral reflectance

NvisionConversiond2p1ConvDate=ldquo2006-12-31T0000000000000+0900rdquoXYZConvData d2p1ID=ldquo8rdquo

XYZConvValues0208511 hellip -0004578XYZConvValuesXYZConvData

Relative colorimetric valuesestimation matrix

SpecReflectData d2p1ID=ldquo9rdquoSpecReflectValues0642792 hellip 0000366SpecReflectValues

SpecReflectData

Relative spectral reflectanceestimation matrix

SpecStimuliData d2p1ID=ldquo10rdquoSpecStimuliValues0068008 hellip 0000031SpecStimuliValues

SpecStimuliData

Relative spectral radianceestimation matrix

ColorMatchingFuncData d2p1ID=ldquo11rdquoColorMatchingFuncValues0001358 hellip 0000000ColorMatchingFuncValues

ColorMatchingFuncData

RenderingIllu d2p1ID=ldquo12rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquoRenderingSpecData0007019 hellip 0167862RenderingSpecData

RenderingIlluNvisionConversion

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-5

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

filters inserted in each path Six-channel uncompressedHDTV video signal is captured and recorded on a magnetichard disk As a result of evaluating the color reproducibilityof this six-band HDTV camera using the GretagMacbethcolor checker the CIELAB color difference average andmaximum Eab

=14 and 42 respectively where Eab =41

and 82 in the case of the conventional HDTV camera The

same Wiener estimation technique mentioned in the nextsection was used in the evaluation of both devices

Figure 5(b) shows the estimation results of high satura-tion colors In this experiment the interference filters of the16-band MSC were illuminated by a light box (DNPHi-Vision Color Viewer) and employed as test colors Thecolors were captured with the six-band camera and the

Table I Summary of the sample profile data in XML version of NV format The mark hellip represents the omitted numerical data

XML Description

Nvision xmlnsldquohttp hellip NvXmlSchemaxsdrdquoNvisionImageImageInfo d2p1ID=ldquo1rdquo

ImageTypeRAWIMAGEImageTypeImageWidth1920ImageWidthImageHeight-1080ImageHeightImageBands6ImageBandsBitSizePerBand16BitSizePerBandDataTypeUINT16DataType

ImageInfoNvisionImage

Basic information of image data

NvisionInput d2p1InputDate=ldquo2006-12-31T0000000000000+0900rdquo Input profile tag

DarkCurrentData d2p1ID=ldquo1rdquoDarkCurrentValue584229 hellip 511390DarkCurrentValue

DarkCurrentData

N-Component bias digital counts

ToneCurvesData d2p1ID=ldquo2rdquoCurveValues0 0 hellip 65535CurveValues

ToneCurvesData

N-Component tone reproduction curves

ExposureTimeData d2p1ID=ldquo3rdquoExposureTimeValue199 hellip 0998ExposureTimeValue

ExposureTimeData

Coefficients of N-componentsensitivity level for the correction ofexposure time setting

SpecSensiData d2p1ID=ldquo4rdquoSpecSensiValue 0000000 hellip 4269989SpecSensiValue

SpecSensiData

Spectral Sensitivity of P-channel inputdevice

NoiseData d2p1ID=ldquo5rdquoNoiseValue0699997 hellip 0699997NoiseValue

NoiseData

N-channel noise levels of Inputdevice

InputIllu d2p1ID=ldquo6rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquo

InputSpecData 0007034 hellip 0167847InputSpecDataInputIllu

Spectral radiance of Q-Inputilluminants

AutoCorrelationMatrix d2p1ID=ldquo7rdquoAutoCorrelationEigenValue0000000 hellip 0256989 AutoCorrelationEigenValueAutoCorrelationMatrixValue0000000 hellip 0099686 AutoCorrelationMatrixValue

AutoCorrelationMatrixNvisionInput

Principal components of objectrsquosspectral reflectance

NvisionConversiond2p1ConvDate=ldquo2006-12-31T0000000000000+0900rdquoXYZConvData d2p1ID=ldquo8rdquo

XYZConvValues0208511 hellip -0004578XYZConvValuesXYZConvData

Relative colorimetric valuesestimation matrix

SpecReflectData d2p1ID=ldquo9rdquoSpecReflectValues0642792 hellip 0000366SpecReflectValues

SpecReflectData

Relative spectral reflectanceestimation matrix

SpecStimuliData d2p1ID=ldquo10rdquoSpecStimuliValues0068008 hellip 0000031SpecStimuliValues

SpecStimuliData

Relative spectral radianceestimation matrix

ColorMatchingFuncData d2p1ID=ldquo11rdquoColorMatchingFuncValues0001358 hellip 0000000ColorMatchingFuncValues

ColorMatchingFuncData

RenderingIllu d2p1ID=ldquo12rdquo d2p1ShortWaveLength=ldquo380000000rdquod2p1DataNumber=ldquo401rdquo d2p1WaveInterval=ldquo1000000rdquoRenderingSpecData0007019 hellip 0167862RenderingSpecData

RenderingIlluNvisionConversion

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-5

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

tristimulus values were calculated by the spectrum-basedmethod explained next The results are compared with thecolors measured by a spectroradiometer in Fig 5(b) Thesimulated results with a three-band camera are also shownin the Fig 5(b) It can be seen that the colors estimated fromthe three-band camera clusters around RGB pure primariesthat is the vertices of the camera analysis gamut The six-

band camera offers considerably better color tones in thesehigh-saturation colors in comparison to the three-bandcamera The color differences are relatively large for greenobjects but they are out-of-gamut of the output deviceseven in the case of wide-gamut displays The results clearlyshow the advantage of multispectral imaging especially inthe case of high-saturation colors The camera was employed

Figure 3 The schematic presentation of the workflow of multispectral images The spectrum-based colorreproduction system supports the image management with the metadata profile data

Figure 4 a 16-band MSC 4M pixelsband b The color estimation accuracy of the 16-band MSC andthe 3-band DSC under daylight DL CIE A and F2 illuminants where DL was used in the image capturingGray and white bars denote the average and maximum CIELAB color differences respectively

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-6 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

for the experiment of real-time video reproduction de-scribed next

In image capture the spectral distribution of the illumi-nation is also measured to estimate the spectral reflectanceor the transmittance of the object For example the reflectedlight from a standard white plate is measured by aspectroradiometer or the MSC or the ambient illuminationis directly measured by a compact fiber-optic spectropho-tometer For correction of the spatial nonuniformity of illu-mination a white plate (in the case of reflective object) or atransparent glass plate (in the case of transmissive object) iscaptured by the MSC For practical use it is expected that

more easily handled devices will become available for themeasurement of the illumination spectrum

Spectrum EstimationIn the spectrum-based color reproduction system the spec-tral radiance reflectance or transmittance is estimated fromthe MSI data An arbitrary estimation method can be usedin this system as the original MSI data can be preservedwith the spectral sensitivity of MSC Among various tech-niques for spectral estimation the method based on Wienerestimation is practical and accurate If the covariance func-tion of spectral reflectance of the target object is available itconsiderably improves the accuracy However it is not nec-essarily possible to obtain the covariance function and wecan alternatively use an approximation in which the covari-ance is modeled as a first-order Markov process1 Thismodel works fairly well for most natural objects because thespectral reflectance of natural objects is for the most partsmooth except in special cases

Even though spectrum estimation may be used in thecolor conversion where the spectrum space requires highdimensionality the implementation of the spectrum-basedcolor conversion is not so difficult in many cases insofar asthe spectral estimation may be be formulated by a linearmodel38 For example to display the color under an il-luminant different from the image capturing environmentthe image spectral reflectance is estimated from the multi-spectral image and the illuminant spectrum of the imagecapturing environment Then the tristimulus values are cal-culated by multiplying the illuminant spectrum of imagedisplay environment and the color matching function Thenthe device RGB signal is obtained by multiplying the matrixfor color conversion After the nonlinearity of the imagecapturing device is corrected the spectrum-based color con-version from the multispectral image to the device RGB sig-nal can be realized with the matrix operation of N3 andfollowing tone curve correction which can be implementedby one-dimensional look-up tables Therefore computationin the spectral space is needed only when changing the colorconversion matrix

In color reproduction for video images the gain param-eter noticeably impacts the color accuracy and the imagequality The gain parameter should be set such that the six-band signals from a reference white object become constantin order to set the signal levels within the dynamic range ofevery band Then the gain parameters are required in thecolor estimation step to correct the magnitudes of the out-put signals as well as to set the noise variance in the deri-vation of a Wiener estimation matrix since the noise levelsof the camera output depend on the gain setting

Multiprimary Display SystemThere were few previous works on multiprimary color dis-plays for larger color gamut NHK (Japan BroadcastingCorp) demonstrated a four-primary projection display toevaluate the wider color space in 19946 The present authorsalso reported a seven-primary display using a holographicoptical element7 However system development and evalua-

Figure 5 a 6-band HDTV camera b Color estimation results of thecolors of interference filters by the six-band camera in CIE xy chroma-ticity diagram The ideal color coordinates obtained by spectral measure-ments and the simulated results obtained by the three-band camera are also shown Black and gray arrows denote the color shifts fromthe original colors for six-band and three-band cases respectivelyDashed lines represent the locus of the colors obtained from the six-bandcamera

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-7

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

tion of the muliprimary display had been originally startedin the context of the NV project In the multiprimary pro-jection display developed in the NV project the images fromtwo projectors [liquid crystal display (LCD) or digital lightprocessing (DLP)] each of which is adapted to produce adifferent set of three primary colors are overlaid on thescreen as shown in Figure 689 As a flat-panel typemultiprimary display a four-primary flat-panel LCD withlight emitting diode (LED) backlight was implemented bymeans of time sequential display10

Figure 7 shows the color gamut of a six-primary DLPdisplay9 As for the gamut of natural objects Pointer gamut39

and standard object color spectra (SOCS) database (exclud-ing fluorescent objects)40 are combined Pointer+SOCSbecause some objects in SOCS are out of the Pointer gamutSix-primary DLP display almost covers the gamut of naturalobjects and the gamut volume in CIELAB space is about 18times larger than that of the normal RGB DLP projectorThe gamut of the six-primary display is enlarged in the darkred cyan purple and bright orange regions as comparedwith the conventional RGB display

Spectral Color DisplayThere exists variability in color matching functions of hu-man observers originating from macular pigments lens ab-sorption and cone sensitivity Due to the individual differ-ence of color matching functions a color difference mayappear to a certain observer when two color stimuli areshown even if perceived as the same color by another ob-server or the CIE standard observer This phenomenon iscalled observer metamerism It causes the color mismatchbetween different media such as color printed materials anddisplays even though the colorimetric match is achieved forCIE 1931 or 1964 color matching functions It has beenshown that the influence of observer metamerism due to thevariation of color matching functions in color reproductionis not negligible4142

Based on the multispectral and multiprimary technol-ogy spectral color display becomes possible and the colormismatch between the display and the real object can disap-pear for different observers The advantage of spectral colordisplay was experimentally demonstrated using the six-

Figure 6 The six-primary projection display using two modified projectors The spectral intensities of eachprojector are also shown

Figure 7 Color gamut of six-primary DLP display conventional three-primary DLP display and natural objectin a-b planes of L=10 50 and 90

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-8 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

primary display25 with the signal processing method brieflyexplained in the next subsection

Multiprimary Color ConversionThe multiprimary color signal is generated from the imageof tristimulus values or multispectral data calledmultiprimary color conversion similar to the color decom-position for multiprimary color printers For colorimetriccolor reproduction three-dimensional tristimulus values aretransformed to M-dimensional multiprimary color values Itinvolves a degree of freedom plural combinations ofmultiprimary color values can reproduce a certain color

The following methods have been developed as themultiprimary color conversion for colorimetric color repro-duction

(1) The matrix switching method43 in which the poly-hedral color gamut spanned by multiprimary colorsis divided into pyramids and selecting a 33 ma-trix depending on the pyramid we perform a linearcolor conversion in each pyramid

(2) Linear interpolation in equiluminance planemethod44 where the multiprimary signal values areinterpolated within the equiluminance plane in thesolid color

(3) Metameric black method45 whereby multiprimarysignal values are decomposed into visible and invis-ible (metameric black) components The visiblecomponents are uniquely solved where themetameric black components are determined suchthat the multiprimary signal values changessmoothly if the change of tristimulus values issmooth

(4) The spherical average method46 in which the con-version is based on the analytical solution such thatthe multiprimary signal values are continuous forthe smooth change of tristimulus values

The computation speed ease of hardware implementa-tion and image quality depend on the conversion methodWhen an image with smooth color tonal change is repro-duced on a multiprimary color display a contourlike patternsometimes appears The observer dependence of colormatching functions and the device characterization error arethe sources of the artifact and proper selection of themultiprimary color conversion process suppresses suchartifacts47 Color tone reproduction is one of the importantissues for image quality in multiprimary displays

For the spectral color reproduction explained in theprevious subsection it becomes N to M conversion where Nis the number of channels of MSI In the proposed N to Mconversion method called the ldquospectral approximationmethodrdquo the spectral error is minimized under the con-straint that the colorimetric match is attained for the stan-dard observer The effectiveness of this approach was con-firmed experimentally2548

Multispectral Image Transmission CompressionThe system architecture of image transmission and signalprocessing in the spectrum-based color reproduction systemis shown in Figure 8 The signal from MSC (multispectralinput signal) is converted to a signal for transmission andconverted again to the display signal for multiprimary orRGB display The same scheme can be used in the storage ofimage data There Three cases to implement the image

Figure 8 The system architecture of multispectral video transmission Case 1 Multispectral input signal istransmitted and the color under the arbitrary illuminant can be computed at the receiver site This scheme issuitable for image archive and distribution purposes Case 2 The spectrum of observation illuminant is passedto the sender in advance and the color under the observation illuminant is transmitted Suitable for one-to-onecommunication or natural color reproduction with fixed observation illuminant Case 3 The spectrum ofobservation illuminant and the display profile are passed to the sender in advance and the display signal istransmitted This seems suitable for client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-9

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

transmission system can be considered and the features ofthese implementations are given in Table II

The compression and encoding are also issues for MSItransmission As an MSI compression technique consideringthe colorimetric accuracy a modified Karhunen-Lueve (KL)transform called weighted KL transform was proposed andcombined with the JPEG2000 scheme49 For video compres-sion three methods that support multichannel imagesMPEG4 studio profile H264AVC and Motion JPEG2000were tested The test included a coding method in which aMSI signal was converted into visible and invisible compo-nents with the subsampling of chrominance signals As aresult it was shown that intraplane codecs and the full-resolution formats without subsampling give better qualityat higher bit rate while a subsampled format surpasses othermethods at lower bit rate50 JPEG2000 is one of the suitableformats for high-quality encoding of both video and stillMSIs The profile data for NV format described previouslycan be easily implemented in JPEG2000 as an extended ICCprofile or metadata using XML

Real-Time Video Reproduction and Transmission SystemA prototype of the multispectral video transmission shownin Figure 9 was also developed51 The six-channel video sig-nal from six-band high definition television (HDTV) camerais converted in a six-to-three color converter to the colori-metric tristimulus values by using a spectrum-based colorreproduction technique In the spectrum-based color repro-duction after correction of the tone reproduction curve ofCCD the spectral reflectance is calculated for every pixel by

Wiener estimation with the spectral sensitivity of the cameraand illuminant spectrum of the image-capturing environ-ment measured in advance The colorimetric tristimulus val-ues are then generated with using the illuminant spectrumof the observation environment All of these processes canbe done by 63 matrix multiplication in real-time

For the image display both three-primary andmultiprimary displays can be employed The color display ischaracterized in advance and the parameters are set into thereal-time color converter When using the six-primary colordisplay a three-to-six color converter consisting of six 3DLUTs is used to generate a six-primary color video signalFigure 10 is the photograph of the demonstration of six-band real-time reproduction along with comparison withthe real objects and the three-band system The advantage ofthe spectrum-based system can be easily observed from thedemonstration in Figure 10

In real-time reproduction direct conversion from threeto six channels is difficult because of the hardware complex-ity Thus colorimetric color reproduction is implementedwith the six-to-three color converter In addition it is pos-sible to apply a limited version of the spectral approximationtechnique the spectral radiance of the object is approxi-mated from the colorimetric tristimulus signal using thespectral estimation technique with the illuminant spectrumof observation environment The object-dependent basisfunctions can be employed to reconstruct the spectra fromthe tristimulus values if available In this case the spectraldifference between the real object and the displayed image

Figure 9 The experimental prototype system for multispectral and multiprimary video reproduction

Table II Transmission of video signal in natural vision

Case Transmissionsignal Description Examples of application

1 Multispectral input signal The color under the arbitrary illuminant can be computed at the receiver site Image archiveImage distribution

2 Colorimetric signal The spectrum of observation illuminant sent to the sender in advance and the colorunder the observation illuminant is transmitted

One-to-one communicationWide-gamut color reproductionwith fixed observation illuminant

3 Multiprimary display signal The spectrum of observation illuminant and the display profile sent to the sender inadvance and the display signal transmitted

Client-server image transmission

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-10 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

becomes smaller if the spectral shape of the reflectance issmooth which is assumed in the spectral estimationThrough the visual observation in an informal manner theeffectiveness of the spectral approximation is found by directcomparison of the object and the display for example in thereproduction of skin colors

The captured image is recorded on the hard disk drive(HDD) video recorder without any compression The datavolume is about 35 Gbps In an experiment on real-timereproduction at a remote site the parallel MPEG-2 encoderdecoder for transmitting video signal through the TCPIPnetwork was used When the network bandwidth was80 Mbps the CIELAB color difference due to the codec andthe network was smaller than unity Eab

Multispectral Image Editing Analysis and DatabaseA multispectral video editor is required to create video con-tents In the multispectral video editor it is expected tohandle image data of an arbitrary number of spectral bandsIt is possible to edit movies from MSI sequences of the vari-ous numbers of bands using a sort of script but it becomesrather burdensome to handle different format images Amethod was proposed to simplify the multispectral videoeditor in which the image data of various numbers of bandsare converted to a single format with the fixed number ofbands where a virtual multispectral camera is assumed todefine the fixed format33 A MSI can be considered as avalue-added color image because every pixel has quantitativespectral data Thus it is possible to make use of MSI datawith the image database and analysis The techniques forspectrum-based image processing analysis and retrieval ex-plained in the next section were developed for specific ap-plications but they can be exploited in other applications aswell

APPLICATIONSMedical ApplicationThe use of digital color images is extremely valuable forteleconference teleconsultation education training imageanalysis and reference database in pathology endoscopyand dermatology In these applications it may also be re-

quired to reproduce the complexion of a patient through thevisual communication system for telediagnosis and homecare If the reproduced color is not accurate it may causemisdiagnosis thus the reliability of reproduced color is criti-cal

In the pathology application the spectrum-based sys-tem enables not only the accurate color reproduction ofstained tissue samples but also the correction of the colorvariation caused by the staining procedure By use of thespectral information in the image the amount of dye in eachpixel can be estimated based on the LambertndashBeer Law andan image can be reconstructed by digital adjustment of thedensity and balance of the staining52 In addition the spec-tral information is valuable for the feature extraction or im-age segmentation30 To show the results of the image analy-ses in a form more familiar to medical doctors a methodcalled digital stain was proposed53 A colorization matrixwas applied to the results of pixel classification and an im-age equivalent to a physically stained specimen was gener-ated through computer processing from the MSI data Itbecomes easier to evaluate the result of image analysis usingthis technique in the pathology field

In dermatology it has been shown that MSI is useful forthe diagnosis of melanoma29 The experiments in the NVproject dealt mainly with inflammatory skin lesions inwhich a faint color difference is important for diagnosis TheMSI systems of both still image and video were tested andthe color reproducibility of the multispectral system wasshown to be sufficiently high from the visual evaluation bydermatologists5455 It was also shown that spectral informa-tion can be utilized for image analysis that supports diagno-sis in both pathology and dermatology for example thegrading of disorders or the quantitative evaluation oftreatments54

Digital Archive and Electronic MuseumThe color of archived images depends on the imaging deviceand illumination condition if we use conventional imagingdevices Many reports are found on the multispectral imag-ing for digital archives262756 In the NV project the digitalarchives created both by multispectral still imaging andvideo are investigated

For the still image digital archive the woodprint byShiko Munakata a famous Japanese printmaker was cap-tured by the 16-band MSC displayed on a screen and trans-ferred to the press in collaboration with Aomori Digital Ar-chives Association Moreover with Biblioteca Nacional deAntropologia y Historia (BNAH) and Instituto Nacional deAstrofisica Optica y Electronica (INAOE) Mexico 16-bandMSIs of ldquocodicesrdquo pictorial documents of the Aztecan erawere captured The result of color reproduction displayedusing a 16-band multispectral image was found to be satis-factory for the art management staffs57 In the capturing ofhistorical works it is often necessary to minimize light irra-diation so the flash lamp whose spectral energy distributionhad been measured in advance was synchronized with 16exposures in order to minimize exposure For large size ob-

Figure 10 Experimental live reproduction using a six-band HDTV cam-era six-primary display and real-time color conversion comparing withconventional HDTV

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-11

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

jects a high-resolution image is similarly generated by tilingsmaller images

The image data can also be employed for high-fidelitycolor prints as explained in the next subsection Further-more the spectral information in the image will be usefulfor the analysis of pigments or dyes or the selection of ma-terials for restoration

High-fidelity natural color reproduction will be valuablefor educational video and multimedia contents promotionsof a regional nature and virtual travel simulation Multi-spectral movies were created to demonstrate the feasibility ofthe multispectral video system using the video editing tech-nique explained previously In the introduction of the tech-nology developed in the NV project as an example of scien-tific video the effectiveness of multispectral andmultiprimary technologies were demonstrated using a wide-gamut multiprimary display The content is composed ofboth the computer graphic (CG) images and the video cap-tured by the six-band camera An introduction of Okinawanregional nature and culture related to color included theaerial view video captured by the six-band video camerafrom a helicopter This material successfully demonstratedthe feasibility of the basic technology of the video acquisi-tion and production though improvement in usability isstrongly recommended

In addition the natural color reproduction of the au-rora was tried experimentally in cooperation with the Uni-versity of Alaska at Fairbanks Since the luminance of theaurora is very low multispectral image capture is difficultInstead high-sensitivity RGB cameras (NAC Image Technol-ogy Inc with Texas Instruments Impactron CCD sensors)are employed with spectral characterization The spectralcharacteristics of the aurora are caused by the oxygen andnitrogen ions which can be measured by spectroradiometerand these data are used in the spectral estimation fromthree-band images The images 30 fps video images fromthree cameras 640480 pixels for each camera are tiled toobtain a larger field of view In the display neutral densityfilters are attached to the multiprimary projectors to repro-duce the very low luminance of real aurora By visual evalu-ation the reproduced colors including faint green dark redand light pink appear faithful to reality

Printing ApplicationMultispectral imaging greatly improves the fidelity of colorprints As a test for the printing of catalogs for productpromotion assuming a normal photographic situation forthe commercial products a 16-band MSC and professionalRGB DSCs were compared in joint work with DNP MediaCreate Co Ltd58 In the current catalog printing process thecolor adjustment by in-house color proof is essential becausethe color reproduction capability of conventional DSCs isnot enough If the in-house color proofing and adjustmentcan be omitted the simplification of the printing workflowbecomes possible along with print quality improvement Inthe experiment the object and camera setting were preparedassuming normal conditions for photography of commercialproducts by a professional photographer Nine different ar-

rangements were tried A 16-band MSC and RGB DSCs(Sinarback 23 by Sinar and D1X by Nikon) were used forcomparison purposes In the multispectral image capturethree flash lamps were synchronized with shutters for 16exposure times At first the images obtained from the RGBDSCs were printed by an ink jet printer which is used fornormal color proofing A professional print director putsinstructions for color corrections on the printed samplesColor correction instructions were given at four to fivepoints on average for every picture

To process the image captured by 16-band MSC thespectral distribution of the flash lamp was measured in ad-vance and then the spectrum-based color reproduction ap-plied to obtain the image in CIEXYZ color space under stan-dard D50 illuminant Then it was converted to 16-bitCIELAB digital color the unsharp mask filter was appliedto adjust the sharpness to the DSC images and RGB datawere generated by Apple ColorSync 30 using an ICC profileprepared for this purpose The printed results were evaluatedagain by the same print director and no instruction for colorcorrection was given for any of the resultant samples Theprint director also commented that the printed results from16-band MSI was even better than the proof for presentationproduced from RGB DSC image after color correction

The development of spectral printing techniques whichis expected to reduce the illumination metamerism has beenreported Although it is not directly dealt with in the NVproject the spectrum-based color reproduction scheme ex-plained in ldquoSpectrum-Based Color Reproduction Systemrdquosupports the use of MSIS for spectral printing For examplespectral color management was proposed using the interimconnection space called LabPQR17 It can be employed withthe system described in the section Spectrum-Based ColorReproduction System above by incorporating the conver-sion from the SPCS to LABPQR

Furthermore spectral color display can be useful forcolor proofing applications using softcopy monitors sincethe color matching between display and print is considerablyimproved thanks to spectral approximation

Electronic Commerce ApplicationIn electronic commerce (EC) such as online shopping usingcolor images color differences may lead to the return ofpurchased items Furthermore the color matching ofsamples between designers factories buyers and sales isneeded and quantitative color information is quite benefi-cial A high-fidelity natural color reproduction systemshould contribute to the further evolution of EC for vehiclesapparel cosmetics toys and interior furnishings A web-based server-client prototype of the electronic catalog systemshown in Figure 11 was developed in the NV project Besidescolor reproduction under an arbitrary observation il-luminant the features offered by the prototype system in-clude automatic selection of the color patch index corre-sponding to the specified image and the retrieval of a fabricimage based on the color histogram or spectral reflectance

The display of a wider color gamut is required in textileapplications From the experiment using 464 colors from

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-12 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

SCOTDIC(R) color book (Kensaikan International Ltd) forcotton it was confirmed that 100 of the colors are coveredby the six-primary display while 6 are out of the sRGBcolor gamut The advantage of the six-primary displays be-comes clearer in consideration of the character of ambientlight According to the evaluation by color experts in theapparel industry the color difference between the real fabricand the reproduced image is almost negligible though theimpression of surface texture is somewhat different betweenthe display screen and the real fabrics The use of motionpictures improves such impression better reproducing theangular dependency of reflection in textiles as well as inmetallic or pearl automotive finishes

Wide Gamut CG SystemThe expanded color gamut of the multiprimary display pro-vides a new tool or new colors for graphics expression59

Through the experiences of graphic expression with thehigh-chroma colors enabled by six-primary display the sig-nificance of wide-gamut graphics was observed This signifi-cance includes (a) rendering of the reality of actual objectssuch as flowers butterflies marine blue water verdant greenleaves dresses and accessories in vivid or splendid colors(b) the enhanced reality based on ldquomemory colorrdquo whichoften shifts to higher saturation with time (c) strong impactor fantastic impressions resulting from color combinationsscarcely encountered in the real world and (d) rich colortones thanks to the expansion of the color range includingdepth impression by the subtle color change in these darkcolors or the expression of gloss metallicity or emissivecolors

Digital PrototypingmdashSpectral BRDF Measurement andRenderingDigital prototyping is an important technology for the expe-dition and efficiency of product development and CG tech-nologies enable the realistic rendering of virtual productsusing the bidirectional reflectance distribution function(BRDF) or Bidirectional Texture Function (BTF) but thecolor often disagrees with that of the real products Themultispectral BRDFBTF measurement system and multi-spectral rendering technique were developed for high-fidelitycolor digital prototyping The system is based on a device forthe BRDF measurement (OGM-3 Digital Fashion Ltd) andXenon light sources to which filter wheels are attached tocapture 16-band MSI The spectral BRDF rendering is real-ized using the multispectral image the image of each band is

independently calculated The result is reproduced on thesix-primary display and compared with the real object toconfirm high color reproducibility34

ConclusionsThe concept technology and applications of a spectrum-based color reproduction system have been introduced inthis article Spectrum-based color reproduction enables notonly high-fidelity color reproduction but also the applicationof image analysis based on quantitative spectral informationMoreover multispectral information will also be of greatutility in image editing for preferred color and other variousimage processing applications such as object extraction orimage synthesis though those were not the main topics ofthe NV project which basically targets natural color repro-duction Wide gamut display is one of the most recent topicsin the display industry and presentations of multiprimarydisplays can also be found from other groups But conven-tional color camera devices and image color spaces such assRGB do not support wider gamut image data at present

Going beyond RGB significant benefits emerge in ad-vanced imaging applications Multiprimary printing is al-ready available in commerce but to make better use of de-vices that support innovative color reproducibility aplatform for the spectrum-based system is expected iemultispectral and wide-gamut video content creation man-agement distribution and utilization

AcknowledgmentsThe authors acknowledge all the former research team mem-bers Hiroyuki Fukuda Norihito Fujikawa Junko KishimotoHiroshi Kanazawa Masaru Tsuchida Ryo Iwama KenroOhsawa Toshio Uchiyama Hideto Motomura SatoshiNambu Tatsuki Inuzuka Yuri Murakami the research fel-lows joint researchers of NV project and the participatingcompanies NTT NTT Data NTT CommunicationsOlympus NHK Hitachi JVC Dainippon Printing ToppanPrinting Panasonic Keisoku Giken and Digital FashionSpecial thanks are given to the following cooperating insti-tutes and companies Kagawa University Hospital YokohamaCity University Hospital University of Pittsburgh MedicalCenter BNAH INAOE the University of Alaska atFairbanks Aomori Digital Archives Association and DNPMedia Create Co Ltd This work is supported by theNational Institute of Information and Communications

Figure 11 An electronic commerce prototype system Image data can be retrieved from the database and thecolor under the observation illuminant can be reproduced through a web-based user interface The illuminationenvironment and display profile are sent to the server and the image for display is generated in the server andsent back to the client

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-13

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

Technology the Ministry of Internal Affairs and Communi-cation and the Natural Vision Promotion Council of Japan

REFERENCES1 W K Pratt and C E Mancill ldquoSpectral estimation techniques for the

spectral calibration of a color image scannerrdquo Appl Opt 15 73ndash75(1976)

2 B Hill and F W Vorhagen ldquoMultispectral image pick-up systemrdquo USPatent 5319472 (1994)

3 P D Burns and R S Berns ldquoAnalysis of multispectral image capturerdquoProc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA1996) pp 19ndash22

4 S Tominaga ldquoMultichannel vision system for estimating surface andillumination functionsrdquo J Opt Soc Am A 13 2163ndash2173 (1996)

5 M Yamaguchi R Iwama Y Ohya T Obi N Ohyama and Y Komiyaand T Wada ldquoNatural color reproduction in the television system fortelemedicinerdquo Proc SPIE 3031 482ndash489 (1997)

6 R Yajima M Kanazawa and S Sakaida ldquoWide-color-gamut systemrdquoITE lsquo94 Annual Convention (Institute of Television Engineers of JapanSaga Japan 1994) pp 355ndash356 (in Japanese)

7 M Yamaguchi T Ajito and N Ohyama ldquoMultiprimary color displayusing holographic optical elementrdquo Proc SPIE 3293 70ndash77 (1998)

8 T Ajito T Obi M Yamaguchi and N Ohyama ldquoExpanded color gamutreproduced by six-primary projection displayrdquo Proc SPIE 3954130ndash137 (2000)

9 H Motomura N Ohyama M Yamaguchi H Haneishi K Kanamori SSannohe ldquoDevelopment of six-primary HDTV display systemrdquo ProcInt Display Research Conference (SID San Jose CA 2002) pp 563ndash566

10 S Komura I Hiyama and N Ohyama ldquoFour-primary-color LCD fornatural visionrdquo Inf Disp 8(3) 18ndash21 (2003)

11 S Roth I BenndashDavid M BenndashChorin D Eliav and O BbenndashDavidldquoWide gamut high-brightness multiple primaries single panel projectiondisplaysrdquo SID Int Symp Digest Tech Papers (SID San Jose CA 2003)pp 118ndash121

12 H Sugiura H Kaneko S Kagawa J Someya and H Tanizoe ldquoSix-primary-color LCD monitor using six-color LEDs with an accuratecalibration systemrdquo Proc SPIE 6058 60580H1ndash60580H8 (2005)

13 T Keusen and W Praefcke ldquoMultispectral color system with anencoding format compatible with the conventional tristimulus modelrdquoProc ISampTSID 3rd Color Imaging Conference (ISampT Springfield VA1995) pp 112ndash114

14 B Hill ldquoMultispectral color technology a way towards high definitioncolor image scanning and encodingrdquo Proc SPIE 3409 2ndash13 (1998)

15 M Rosen M D Fairchild G M Johnson and D R Wyble ldquoColormanagement within a spectral visualization toolrdquo Proc ISampTSID 8thColor Imaging Conference (ISampT Springfield VA 2000) pp 75ndash80

16 M Rosen F Imai X Jiang and N Ohta ldquoSpectral reproduction fromscene to hardcopy II Image processingrdquo Proc SPIE 4300 33ndash41 (2001)

17 M Derhak and M Rosen ldquoSpectral colorimetry using LabPQRmdashaninterim connection spacerdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 246ndash250

18 M Yamaguchi T Teraji K Ohsawa T Uchiyama H Motomura YMurakami and N Ohyama ldquoColor image reproduction based on themultispectral and multiprimary imaging Experimental evaluationrdquoProc SPIE 4663 15ndash26 (2002)

19 M Yamaguchi H Haneishi H Fukuda J Kishimoto H Kanazawa MTsuchida R Iwama and N Ohyama ldquoHigh-fidelity video and still-image communication based on spectral information Natural visionsystem and its applicationsrdquo Proc SPIE 6062 60620G1ndash60620G12(2006)

20 M Yamaguchi ldquoBeyond RGB Spectrum-based color imagingtechnologyrdquo Proc ISampTrsquos NIP 22 (ISampT Springfield VA 2006) pp326ndash330

21 K Ohsawa H Fukuda Y Komiya M Yamaguchi and N OhyamaldquoSpectrum-based color reproduction system for motion picturerdquo ProcSPIE 5008 229ndash236 (2003)

22 P-C Hung ldquoColorimetric calibration for scanners and mediardquo ProcSPIE 1448 164ndash174 (1991)

23 L A Taplin and R S Berns ldquoSpectral color reproduction based on asix-color inkjet output systemrdquo Proc ISampTSID 9th Color ImagingConference (ISampT Springfield VA 2001) pp 209ndash213

24 IEC61966-2-4 Multimedia systems and equipmentmdashColourmeasurement and managementmdashPart 2ndash4 Colour managementmdashExtended-gamut YCC colour space for video applicationsmdashxvYCC(2006)

25 Y Murakami J Ishii T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multi-primary display for spectral colorreproductionrdquo J Electron Imaging 13 701ndash708 (2004)

26 K Martinez J Cupitt D Saunders and R Pillay ldquoTen years of artimaging researchrdquo Proc IEEE 90 28ndash41 (2002)

27 H Maitre F Schmitt J Crettez Y Wu and J Y HardebergldquoSpectrophotometric image analysis of fine art paintingsrdquo Proc ISampTSID 4th Color Imaging Conference (ISampT Springfield VA 1996) pp50ndash53

28 L T Maloney and B A Wandell ldquoColor constancy a method forrecovering surface spectral reflectancerdquo J Opt Soc Am A 3 29ndash33(1986)

29 B Farina C Bartoli A Bono A Colombo M Lualdi G Tragni RMarchesini B Farina C Bartoli A Bono A Colombo M Lualdi andG Tragni ldquoMultispectral imaging approach in the diagnosis ofcutaneous melanoma Potentiality and limitsrdquo Phys Med Biol 451243ndash1254 (2000)

30 R Levenson P J Cronin and K K Pankratov ldquoSpectral imaging forbrightfield microscopyrdquo Proc SPIE 4959 27ndash33 (2003)

31 H Motomura K Ohsawa N Ohyama M Yamaguchi and H HaneishildquoNatural Vision image data file format for spectral-based colorreproduction systemrdquo Proc ISampTrsquos PICs Conference (ISampT SpringfieldVA 2003) pp 278ndash284

32 H Kanazawa M Yamaguchi H Haneishi and N Ohyama ldquoNaturalVision XML Databaserdquo Proc AIC Colour 05ndash10th Congress of theInternational Colour Association (ICA Granada Spain 2005) pp1661ndash1664

33 T Uchiyama M Yamaguchi H Haneishi N Ohyama and S NambuldquoA method for the unified representation of multispectral images withdifferent number of bandsrdquo J Imaging Sci Technol 48 120ndash124 (2004)

34 M Tsuchida Y Sakaguchi H Arai M Nishiko N Fujikawa MYamaguchi H Haneishi and N Ohyama ldquoHigh-fidelity colorreproduction based on multi-channel BTFBRDF acquisition renderingand displayrdquo ACM SIGGRAPH 2005 Sketches (ACM Los Angeles 2005)p 42

35 T Hyvarinen E Herrala and A DallAva ldquoDirect sight imagingspectrograph A unique add-on component brings spectral imaging toindustrial applicationsrdquo Proc SPIE 3302 165ndash175 (1998)

36 A Hirai M Hashimoto K Itoh and Y Ichioka ldquoMultichannel spectralimaging system for measurements with the highest signal-to-noiseratiordquo Opt Rev 4 334ndash341 (1997)

37 K Ohsawa T Ajito H Fukuda Y Komiya H Haneishi M Yamaguchiand N Ohyama ldquoSix-band HDTV camera system for spectrum-basedcolor reproductionrdquo J Imaging Sci Technol 48 85ndash92 (2004)

38 D H Marimont and B A Wandell ldquoLinear models of surface andilluminant spectrardquo J Opt Soc Am A 9 1905ndash1913 (1992)

39 M R Pointer ldquoThe gamut of real surface coloursrdquo Color Res Appl 5145ndash155 (1980)

40 ldquoStandard Object Colour Spectra Database for Colour ReproductionEvaluation (SOCS)rdquo TR X001298 Japanese Standards Association(1998)

41 R L Alfvin and M D Fairchild ldquoObserver variability in metamericcolor matches using color reproduction mediardquo Color Res Appl 22174ndash188 (1997)

42 K Ohsawa F Konig M Yamaguchi and N Ohyama ldquoColor matchingexperiment using 6-primary displayrdquo 3rd International Conference onMultispectral Color Science (Univ Joensuu Joensuu Finland 2001) pp85ndash88

43 T Ajito K Ohsawa T Obi M Yamaguchi and N Ohyama ldquoColorconversion method for multiprimary display using matrix switchingrdquoOpt Rev 8 191ndash197 (2001)

44 H Motomura ldquoColor conversion for a multi-primary display usinglinear interpolation on equi-luminance plane method (LIQUID)rdquo J IntSID 11 371ndash378 (2003)

45 F Konig K Ohsawa M Yamaguchi N Ohayama and B Hill ldquoAmultiprimary display Optimized control values for displayingtristimulus valuesrdquo Proc ISampTrsquos PICS Conference (ISampT Springfield VA2002) pp 215ndash220

46 H Kanazawa ldquoColor conversion for multiprimary displays using aspherical average methodrdquo Proc ISampTSID 12th Color ImagingConference (ISampT Springfield VA 2004) pp 65ndash69

47 J Takiue S Sugino Y Murakami T Obi M Yamaguchi and NOhyama ldquoEvaluation of smoothness in color gradation on multiprimarydisplayrdquo Proc ISampTSID 12th Color Imaging Conference (ISampTSpringfield VA 2004) pp 257ndash264

48 T Uchiyama M Yamaguchi H Haneishi and N Ohyama ldquoA visual

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

010201-14 J Imaging Sci Technol 521Jan-Feb 2008

evaluation of images reproduced on a multiprimary display by colordecomposition based on spectral approximationrdquo J Imaging SciTechnol 49 410ndash417 (2005)

49 S Yu Y Murakami T Obi M Yamaguchi and N OhyamaldquoMultispectral image compression for high fidelity colorimetric andspectral reproductionrdquo J Imaging Sci Technol 50 64ndash72 (2006)

50 S Takamura and Y Yashima ldquoMultiband Video Signal Coding withH264AVC MPEG-4 Studio Profile and Motion JPEG 2000 andAssociated Signal Formatrdquo J Inst Image Electronics Engineers Japan 33792ndash801 (2004)

51 R Iwama M Mitsui M Yamaguchi H Haneishi and N OhyamaldquoReal-time multispectral and multiprimary video systemrdquo Proc AICColour 05ndash10th Congress of the International Colour Association (ICAGranada Spain 2005) pp 137ndash140

52 T Abe Y Murakami M Yamaguchi N Ohyama and Y Yagi ldquoColorcorrection of pathological images based on dye amount quantificationrdquoOpt Rev 12 293ndash300 (2005)

53 P Bautista T Abe M Yamaguchi Y Yagi and N Ohyama ldquoDigitalstaining for multispectral images of pathological tissue specimens basedon combined classification of spectral transmittancerdquo Comput MedImaging Graph 29 649ndash657 (2005)

54 M Yamaguchi M Mitsui Y Murakami H Fukuda N Ohyama and Y

Kubota ldquoMultispectral color imaging for dermatology Application ininflammatory and immunologic diseasesrdquo Proc ISampTSID 13th ColorImaging Conference (ISampT Springfield VA 2005) pp 52ndash58

55 M Yamaguchi R Iwama H Kanazawa N Fujikawa H Fukuda HHaneishi N Ohyama H Wada T Kambara M Aihara Y YamakawaA Nemoto M Furukawa and Z Ikezawa ldquoColor reproducibility of skinlesions in multispectral video Experimental evaluationrdquo Proc ISampTSID14th Color Imaging Conference (ISampT Springfield VA 2006) pp 8ndash13

56 H Haneishi T Hasegawa A Hosoi Y Yokoyama N Tsumura and YMiyake ldquoSystem design for accurately estimating spectral reflectance ofart paintingsrdquo Appl Opt 39 6621ndash6632 (2000)

57 H Fukuda T Uchiyama H Haneishi M Yamaguchi and NagaakiOhyama ldquoDevelopment of 16-bands multispectral image archivingsystemrdquo Proc SPIE 5667 136ndash145 (2005)

58 A Uchida K Ohsawa Y Komiya M Yamaguchi H Haneishi and NOhyama ldquoProduct photography for catalogs with multispectral camerardquoProc Spring Conference of the Japanese Society of Printing Science andTechnology (JSPST Chiba Japan 2003) pp 131ndash134 (in Japanese)

59 J Kishimoto M Yamaguchi H Haneishi and N OhyamaldquoIRODORImdashA Color-rich Palette Based on Natural Vision TechnologyrdquoACM SIGGRAPH 2004 Emerging Technologies (ACM Los Angeles 2004)p 13

Yamaguchi Haneishi and Ohyama Beyond RGB Spectrum-based color imaging technology

J Imaging Sci Technol 521Jan-Feb 2008 010201-15