Broadcast Technology No.55, Winter 2014 ● C NHK STRL 5

High-speed cameras are capable of reproducing slow- motion video of momentary phenomena that are invisible to the human eye and to ordinary television cameras. They are used in all sorts of programs, including sports and science programs, and to produce dramatic slow- motion effects in TV dramas. At NHK STRL, we are working on the development of an ultra-high-speed CCD (charge coupled device) capable of capturing a million frames per second - far beyond the capabilities of conventional high-speed cameras - and a compact color camera based on this device. We have successfully captured footage of ultra-high-speed phenomena that have hitherto been difficult to capture, and we have worked towards making our ultra-high-speed cameras ready for program production. This article discusses the R&D trends in high-speed imaging devices, presents an overview of the research and development of our new ultra-high-speed imaging device, and introduces some applications of this device.

1. IntroductionHigh-speed cameras can produce slow-motion video

of fast-moving phenomena that cannot be seen clearly by the naked eye. They are used in the fields of science and measurement and are widely used in broadcasting. Most ordinary high- speed broadcast cameras use CMOS (complementary metal oxide semiconductor) imaging devices that are capable of capturing images at high speed.1) These can capture images at approximately 1,000 frames per second, which allows high-quality slow motion video playback to be shown in programs like sports broadcasts that feature human movements. However, in ordinary imaging devices like CMOS imaging devices, the signal charges formed in each pixel have to be completely extracted from the imaging device before the device can begin reading the next signal charges. Therefore, as the imaging speed increases, the number of pixels that can be read out becomes smaller, resulting in a loss of image resolution. Furthermore, a faster imaging speed means that there is less time to form the constituent pixels of each frame, so the amount of light incident on the imaging device is smaller and the S/N is worse. Therefore, in the production of science programs and the like, where a very fast capture rate is required, there has been a need for an ultra-high-speed camera that can overcome these issues.

At STRL, we have been researching and developing an ultra-high-speed CCD in which each pixel has its own memory and a larger photodiode area and cameras based on this device. So far, we have developed an ultra-high-speed CCD with 300,000 pixels that can capture video at a rate of 1 million frames per second and a compact single-chip color camera that uses this CCD. We have applied this camera to program production in

a variety of different genres, including sports programs and natural science programs.2) To further increase the performance of the ultra-high-speed camera, we have also been researching and developing an ultra-high-speed CCD with an improved dynamic range in high-speed imaging that is capable of capturing two million frames per second3) and an ultra-high-speed back-side-illuminated CCD with dramatically improved sensitivity and imaging speed.4) In this article, we first discuss the research and development of high-speed imaging devices; then we present the R&D done here at STRL and introduce examples of the applications in which we are using this technology.

2. R&D on high-speed camerasHigh-speed cameras can capture and record video at

higher speeds (frame rates) than ordinary TV cameras. By playing this video back at a normal frame rate, they are able to reproduce slow-motion video with smooth movements (Figure 1). Table 1 presents a classification of high-speed and ultra-high-speed cameras according to the method used to increase the speed of the imaging device. In this table, a high-speed imaging device is defined as an imaging device that obtains continuous video by outputting the captured signals at high speed to an external memory in the camera, and an ultra-high-speed imaging device is defined as an imaging device provided with its own internal memory that temporarily stores the captured signals before they are output from the device. In a high-speed imaging device, the signal charges for a single frame generated by the photodiodes must all be output during the next frame period. Therefore, the imaging speed depends on the number of pixels and on the operating speed of the imaging device’s output circuit. To increase the speed at which data is read out from the imaging device, a parallel configuration of multiple output circuits is normally used (Figure 2). Currently, an imaging device with 128 parallel outputs has been developed,5) and a camera capable of capturing 2,000 frames per second at full HDTV resolution (about 2 megapixels) is commercially available. This camera can capture video at even higher speeds by reducing the number of output pixels (decreasing the resolution) (Figure 3). For example, by reducing the sensor resolution to 80,000 pixels, it is possible to achieve an image capture rate of 20,000 frames per second. Thus, in ordinary high- speed cameras, there is a trade-off relationship between the image capture rate and resolution.

On the other hand, the ultra-high-speed imaging device has a built-in memory that temporarily accumulates the signal charges from the photodiodes. Once shooting has finished, the signal charges are output from the imaging device, so it is possible to perform ultra-high-speed image capture independently of the operating speed of the

R&D on ultra-high-speed imaging devices

Broadcast Technology No.55, Winter 2014 ● C NHK STRL

output circuitry. However, since this approach requires that the device includes enough memory to store the number of frames being captured, its capacity is limited by the area of the device. We are currently developing ultra-high-speed imaging devices with resolutions of

80,000 pixels to 310,000 pixels and sufficient memory for 100-150 frames. Ultra-high-speed imaging devices can be categorized into pixel-internal memory devices,6) where the memory that accumulates the signal charges is situated inside each pixel, and pixel external devices,7)

6

Table 1: Classification of high-speed and ultra-high-speed imaging devices

Method used to increase speed

Device

Parallel output and restriction of output pixel area

CMOS CMOS

Memory deployed inside the device

CCD

High-speed imaging device Ultra-high-speed imaging device

Pixel internal memory

Pixel external memory

Figure 3: Device operated at higher speed by restricting the number of outputting pixels (Approx. 80,000 pixels, 20,000

frames per second)

320

240

8 outputs

8 outputs

Horizontal scan circuit

Ver

tica

l sca

n c

ircu

it

Horizontal scan circuit

Ver

tica

l sca

n c

ircu

it

Figure 2: High-speed imaging device with 128 parallel outputs (Approx. 2 megapixels, 2,000 frames per second)

64 outputs

64 outputs

1,920

1,080

Ver

tica

l sca

n c

ircu

it

Ver

tica

l sca

n c

ircu

it

Horizontal scan circuit

Horizontal scan circuit

Figure 1: The principle of slow-motion video capture

Play back at normal speed (30 frames per second)

Capture instantaneous phenomena at high speed(20,000 frames per second)

Ultra-high-speed camera

Capture instantaneous phenomena at high speed(20,000 frames per second)

Broadcast Technology No.55, Winter 2014 ● C NHK STRL

Feature

7

where the memory is situated outside of the pixels. In a pixel-internal memory device, the CCD transfer path that sends the signal charges from the photodiodes to the output circuitry is used as the memory (Figure 4). At STRL, in order to produce an ultra-high-speed camera for broadcast use, we are working to increase the resolution and sensitivity of ultra-high-speed imaging devices with pixel internal memories.

A pixel-external memory device is configured with each pixel’s photodiode connected to a memory for the number of frames to be recorded (Figure 5). A characteristic of CMOS imaging devices is their low power consumption. It is currently possible to purchase a monochrome camera with a resolution of 100,000 pixels that can capture video at up to ten million frames per second.

3. R&D on ultra-high-speed imaging devices at STRLThe production of sport programs and dramas has

hitherto involved using cameras with a capture rate of roughly 180-300 frames per second.8) However, such

rates are too slow to capture clear slow-motion video of near-instantaneous events such as a ball bouncing off a bat in a baseball game or the impact of a club on a golf ball. For such purposes, an even faster camera is needed. STRL’s research and development on ultra-high-speed cameras began in 2003. Initially, we built a prototype three-chip ultra-high-speed color camera9) incorporating CCDs that had a peak capture rate of one million frames per second.10) This camera was designed for scientific research applications but was also tried out at a night-time baseball match in Tokyo Dome. The illumination of the subject position was only about 1,300 lx, which is quite dark for high- speed shooting, but we were still able to obtain clear footage of the instant the ball was struck by the bat by operating the camera at 1,000 frames per second (Figure 6). We also used this camera to capture video at 4,000 frames per second for a live golf broadcast. Despite the cloudy sky, the images clearly showed the deformation and subsequent movement of a golf ball struck by a club (Figure 7). However, the video was of insufficient quality for broadcast because

Figure 4: Schematic configuration of an ultra-high-speed imaging device with internal memory

Figure 5: Schematic configuration of an ultra-high-speed imaging device with external memory

Photodiode

Memory CCD

Output from CCD to external circuitry

Drain

Photodiode

Memory

Pixel unit

External memory unit

Broadcast Technology No.55, Winter 2014 ● C NHK STRL8

it only had the CCDs of the camera had only 80,000 pixels. We subsequently came up with a new CCD with twice the resolution (150,000 pixels)11) and not long after developed a technology for bonding two CCD elements together with high accuracy. This development yielded a prototype 300,000-pixel ultra-high-speed CCD with excellent resolution characteristics.12) We mounted a color filter on this CCD and used it to make a compact single-chip ultra-high-speed camera.13) The camera was subsequently used in various genres, including sports and science programs.

The repeated trials of ultra-high-speed cameras in a diverse range of programs have shown that they are powerful tools, but there is a need for even greater image capture rates and resolutions. Therefore, we have continued with our research by developing an ultra-high-speed CCD with improved dynamic range during high-speed image capture and a maximum image capture rate of two million frames per second and a back-side-illuminated ultra-high-speed CCD with significantly improved sensitivity and image capture rate.

4. Applications of ultra-high-speed camerasLightning strikes appear as momentary flashes to the

human eye, but in fact start out with a weak discharge generated from the thundercloud. This discharge (called a stepped leader) intermittently advances while branching outwards, until a large amount of charge is injected into one of its branches as lightning.14) This stepped leader advances at a very high average speed of 150 km/s, making it very hard to capture on video.

With help from the ultra-high voltage research center of the Nippon Institute of Technology and the Central Research Institute of Electric Power Industry (CRIEPI), NHK used an ultra-high-speed single-chip camera developed at STRL to capture a stepped leader discharge on video for the first time (Figure 8). This camera was only able to capture 144 frames, so the experiment needed a special trigger device that operated on a wireless signal sent at a timing controllable to the microsecond. Furthermore, as the light from the stepped leader is very weak and the sensitivity of the ultra-high-speed camera is insufficient to pick it up, the camera was equipped with an image intensifier in order to amplify the faint light from the discharge (Figure 9). As shown in Figure

Figure 6: Video captured during a live baseball broadcast (capture rate: 1,000 fps)

Figure 7: Video captured during a live golf broadcast (capture rate: 4,000 fps)

Figure 8: Camera setup for recording discharge phenomena

Figure 9: Ultra-high-speed camera fitted with an image intensifier

Image intensifier

Figure 10: Growth of stepped discharge(Image capture rate: 1 million frames/second)

① ② ③ ④

Broadcast Technology No.55, Winter 2014 ● C NHK STRL

10, we succeeded in capturing clear slow-motion video of a stepped leader (one million frames per second). This video was broadcast during the program “The eyes of Einstein”.

5. ConclusionsWe discussed recent R&D on high-speed imaging

devices, with a focus on ultra-high-speed imaging and its applications.

The ultra-high-speed single-chip color cameras developed at STRL can capture images at a million frames per second without loss of resolution. They have been used in scientific programs and in live sports broadcasting to create slow-motion video of unprecedented clarity. The applications of these cameras extend to other fields besides broadcasting, such as science and medicine, where there is a need to visualize ultra-high-speed phenomena. Our latest ultra-high-speed single-chip color imaging devices have 300,000 pixels and frame rates of two million per second. These ultra-high-speed cameras will surely make great contributions to any field in which it is used.

We gratefully acknowledge the contributions of Kinki University, Hitachi Kokusai Electric Inc., and Teledyne DALSA Inc. to the development of ultra-high-speed CCDs and cameras described here.

References

1) A. Krymski, N. Bock, D. Van Blerkorn, N. Tu, and E.R. Fossum: “A High Speed, 240 Frames/s 4 Megapixel CMOS Sensor,” Proc. IEEE Workshop on CCD and AIS (2001)

2) K. Tomiyama, K. Hisatomi, M. Katayama, Y. Iwadate, T. Hayashida, H. Maruyama, T.G. Etoh, and T. Yoshida: “Advanced Video Image Technologies for Sports TV Productions,” NAB Proc., pp. 193-198 (2008)

3) T.Arai, K.Kitamura, J.Yonai, H.Ohtake, T.Hayashida, H.Maruyama, H.Van Kuijk, and T.G.Eto: “2-Mfps 300-kpixel Ultrahigh-speed Single-chip Color Camera”, Institute of Electronics, Information and Communication Engineers C Vol. J94 -C, No. 9, pp. 252-260 (2011)

4) T.Arai, J.Yonai, T.Hayashida, H.Ohtake, H.Van Kuijk, and T.G.Eto: “Development of a 16.7 Mfps, 312 kPixel Back-side- Illuminated Ultrahigh-Speed CCD Image Sensor with a Sensitivity 12.7 Times Higher than That with Front- Side Illuminated Structure,” J. IEICE, Vol. J96-C, No. 7, pp. 180-190 (2013)

5) B. Cremers, M. Agarwal, T. Walschap, R. Singh, and T. Geurts: “A High Speed Pipelined Snapshot CMOS Image Sensor with 6.4 Gpixel/s Data Rate,” Proc. International Image Sensor Workshop, p. 9 (2009)

6) W.F. Kosonocky, G. Yang, R.K. Kabra, C. Ye, Z. Pektas, J.L. Lowrance, V.J. Mastrocora, F.V. Shallcross, and V. Patel: “360×360 Element Three-phase Very High Frame Rate Burst Image Sensor: Design, Operation, and Performance,” IEEE Transactions on Electron

Devices, Vol. 44, No. 10, pp. 1617-1624 (1997)7) T.Ogasawara, M.Yamauchi, Y.Tomura, J.Yamazaki,

M.Gotoh, Y.Hashimoto, H.Cho, E.Kochi, and S.Kanayama: “A 300- fps Progressive Scan HDTV High Speed Camera,” J. Institute of Image Information and Television Engineers, Vol. 60, No. 3, pp. 358-365 (2006)

8) Y. Tochigi, K. Hanzawa, Y. Kato, R. Kuroda, H. Mutoh, R. Hirose, H. Tominaga, K. Takubo, Y. Kondo, and S. Sugawa: “A Global-shutter CMOS Image Sensor with Readout Speed of 1 Tpixels/s Burst and 780 Mpixel/s Continuous,” Proc. IEEE International Solid State Circuits Conference, pp. 382-383 (2012)

9) H. Maruyama, H. Ohtake, T. Hayashida, M. Yamada, K. Kitamura, T. Arai, K. Tanioka, T.G. Etoh, J. Namiki, T. Yoshida, H. Maruno, Y. Kondo, T. Ozaki, and S. Kanayama: “Color Video Camera Capable of 1,000,000 fps with Triple Ultrahigh-speed Image Sensors,” SPIE 26th International Congress on High- speed Photography and Photonics, p. 5580 (2004)

10)T.G. Etoh, D. Poggemann, A. Ruckelshausen, A. Theuwissen, G. Kreider, H.O. Folkerts, H. Mutoh, Y. Kondo, H. Maruno, K. Takubo, H. Soya, K. Takehara, T. Okinaka, Y. Takano, T. Reisinger, and C. Lohmann: “A CCD Image Sensor of 1 Mframes/s for Continuous Image Caputring of 103 Frames,” Proc. IEEE International Solid State Circuits Conference, pp. 46-47 (2002)

11) H. Ohtake, T. Hayashida, K. Kitamura, T. Arai, J. Yonai, H. Maruyama, K. Tanioka, T.G. Etoh, D. Poggemann, A. Ruckelshausen, H. van Kuijk, and J.T. Bosiers: “Design of a 148,680-pixel Ultrahigh-speed High-sensitivity CCD,” Proc. IEEE Workshop on CCD and AIS (2005)

12) T. Hayashida, K. Kitamura, T. Arai, J. Yonai, H. Ohtake, H. Maruyama, and T.G. Etoh: “300,000-pixel ultrahigh-speed, high-sensitivity CCD,” Institute of Image Information and Television Engineers Annual Conference, 23-1 (2005)

13) H. Ohtake, T. Hayashida, K. Kitamura, T. Arai, J. Yonai, K. Tanioka, H. Maruyama, and T.G. Etoh: “300,000-pixel ultrahigh-speed, high-sensitivity CCD, and a single-chip color camera that uses it,” NHK STRL R&D, No. 97, pp. 38-47 (2006)

14) Z.Kawasaki: “Structures and Dynamics of Lightning Discharges in Atmosphere and Ionosphere,” J. Plasma Fusion Res., Vol. 84, No. 7, pp. 405-409 (2008)

9