proposal of fingering detection method for touch typing

Bulletin of the JSME

Journal of Advanced Mechanical Design, Systems, and ManufacturingVol.15, No.2, 2021

© 2021 The Japan Society of Mechanical Engineers[DOI: 10.1299/jamdsm.2021jamdsm0021]Paper No.20-00322

Proposal of fingering detection method for touch typing learning support system

*Department of Information and Computer Science, Doshisha University

1-3 Tatara-Miyakodani, Kyotanabe, Kyoto 610-0321, Japan

E-mail: [email protected]

1. Introduction

Among the data utilization skills nowadays—actions used for utilizing data, scientifically understanding information,

and partaking in an information-based society—typing skills are considered to be one of the most fundamental skills and are given much importance (MEXT) (Matsuyama, et al., 2002). Surveys among university students, however, show that some students’ typing skills are so low as to pose a problem once they join the workforce. Despite this, the majority of students considered to be of the “Internet generation” are falsely thought of as proficient in ICT (Kojima, 2016). One method for improving typing skills is touch typing. Touch typing is also referred to as “the touch method,” and it involves entering keys on the keyboard without looking at one’s hands and only looking at the screen. By learning touch typing, one can increase their input speed and reduce fatigue caused by moving the eyes (Tamura and Takaoka, 2012). However, according to a 2018 survey conducted among 384 male and female university students about to enter the job market, 60.7% answered that they were not capable of touch typing (New recruit White paper, 2017).

Daichi KANO* and Masashi OKUBO*

Received: 28 June 2020; Revised: 14 September 2020; Accepted: 20 November 2020

Abstract Recently, typing skills especially touch typing is given much importance. However, the typing skill of entrant students tends to be poor, because of the use of smart phone. Although some students try to improve their typing skill using typing software, pre-existing one similarly ignores which fingers are used, and instead emphasizes hitting the correct keys as fast as possible. The goal of this study is to develop a learning support system for touch typing based on fingering detected when typing, and to evaluate the possibility of teaching touch typing to non-touch typists through practicing with this system. In this study, we propose a method to determine which fingers are used to press which keys by using video of fingering made when typing and data of the pressed keys. Furthermore, we conduct evaluation experiments with the system using this method and evaluate the effectiveness of the system. According to the experiment results, estimation accuracy tended to decrease as CPM values increase. We learned that there were cases in which incorrect estimations were made, and we classified the causes into two categories: errors caused by occlusion and errors due to the image sampling being unable to keep up with the fingering. It is evident from the experimental results and discussion that when this system is used for non-touch typists to learn touch typing, it gives correct estimation results and is effective. For further research, we suggest using this system to facilitate the learning of touch typing for non-touch typists. More specifically, learners should enter a sequence of letters shown on-screen. Messages should be displayed on-screen when an incorrect key is pressed or when the recommended finger is not used. We believe that touch typing skills can be easily acquired by making users aware of what fingers they use to press the keys.

Keywords : Touch typing, Learning support system, Fingering recognition, Keyboard, IT skill, Image processing and Serious game

1

2© 2021 The Japan Society of Mechanical Engineers[DOI: 10.1299/jamdsm.2021jamdsm0021]

Kano and Okubo, Journal of Advanced Mechanical Design, Systems, and Manufacturing, Vol.15, No.2 (2021)

There are several proposed methods of discerning whether people have acquired touch typing skills. Matsunaga proposes a method that involves using their own developed typing test software and blank keyboard to determine if subjects can touch type correctly (Matsunaga, 2007). In addition, Ando and associates tracked the eye movements of subjects and found that those who did not look at their hands while typing had better typing skills, stating that beginners must learn touch typing and train their fingers to make quick movements in order to improve their typing skills (Ando and Ito, 2010). However, these methods do not focus on which fingers were used to press the keys. Pre-existing typing software, such as e-typing, similarly ignores which fingers are used, and instead emphasizes hitting the correct keys as fast as possible (E-typing CO., Ltd.).

The goal of this study is to construct a system to facilitate the learning of correct fingering for those who have not acquired touch typing skills. Previously proposed methods have used blank keyboards and eye tracking to measure typing skills. In this study, however, we propose a method to determine which fingers are used to press which keys by using video of fingering made when typing and data of the pressed keys in order to develop a learning support system for touch typing. Another research developed the typing skill learning environment using web camera and markers on the finger tips (Soga, 2013). As a result of experiment, the proposed environment is useful for typing skill learning. On the other hand, our proposed system utilizes the key position which obtained the color markers on the keys using same web camera, once before the exercise. This proposed method doesn’t need time and effort for learners.

2. Typing skills 2.1 Touch typing

Figure 1 shows the key locations and finger home positions, in addition to the finger recommended to use for each key. “Home position” refers to the primary area in which to place the fingers.

Fig. 1 Home positions and key assignments.

2.2 Typing speed In order to identify the differences in typing speeds between touch typists and non-touch typists, we conducted a

survey regarding typing speed (CPM). CPM stands for “characters per minute” and refers to the number of keystrokes made per minute, an indicator of typing skills. This survey was conducted on 16 graduate students from Doshisha University and asked participants whether they considered themselves touch typists, to which they answered “yes” or “no.” Participants were then made to type, and their typing speed was measured. For measuring typing speed, the practice mode of a typing software named e-typing was used (E-typing CO., Ltd.). Figure 2 shows a histogram of the typing speeds of the self-declared touch typists (a) and non-touch typists (b). Self-declared touch typists (Fig. 2 (a)) had CPMs ranging from 150 to 400, and those who did not consider themselves touch typists (Fig. 2 (b)) had CPMs ranging from 100 to 250. Judging from these results, the system developed for detecting the fingering of non-touch typists must be suited for CPM speeds up to 300 CPM.

2



Fig. 2 Histogram of typing speed of (a) touch typists and (b) non-touch typists.

2.3 Research objectives

The goal of this study is to develop a learning support system for touch typing based on fingering detected when

typing, and to evaluate the possibility of teaching touch typing to non-touch typists through practicing with this developed system.

In this study, we propose a method to determine which fingers are used to press which keys by using video of fingering made when typing and data of the pressed keys. We then conducted evaluation experiments with the system using this method, and evaluated the effectiveness of the system by calculating estimation accuracy. 3. Detection of fingering made when typing 3.1 System overview

Figure 3 shows an overview of the system. Video recorded from a web camera installed directly above the keyboard is sent to the computer, which then calculates the coordinates of all keys and all fingers. The proposed system uses a normal keyboard associated with PC. The size of key area is W280 x D95mm, and key pitch is about 19mm. And the web camera (Sanwa Supply CMS-V32BK) resolution is 1280 x 720pixels, aperture value is F2.8, sampling rate is 30fps. The distance from keyboard to camera is about 300mm that the camera can capture the all keys used in the system. When a key is pressed, the name of the pressed key is sent to the computer, and the system detects the coordinates of the key from the coordinates of all keys calculated previously based on the pressed key’s coordinates and the coordinates of all fingers. The system then estimates the finger used to type.

3



Fig. 3 System configuration.

3.2 Detection process for fingers used to press key 3.2.1 Detection process for finger coordinates

In order to detect the fingers’ coordinates, we mounted colored markers on each finger. Figure 4 shows the flow of

the detection process using colored markers with the example of green markers. The video obtained via web camera is divided frame by frame and then processed as still images. In order to increase detection accuracy when evaluating the colored regions during the process, we placed a black piece of fabric underneath the black keyboard. First, each region of the colored markers is detected using the images taken from the web camera. Next, the centers of these detected colored marker regions are calculated (indicated with red dots in the figure). The coordinates in the images of the calculated centers are set as the coordinates for the color markers.

Fig. 4 Procedure for detecting green colored markers.

Fingers mounted with colored markers in this system are the four fingers on each hand used most for typing—the index fingers through the little fingers—for eight fingers in total. Four colors were used for the main colored markers and two colors were used for the sub-colored markers. Four colors were used in order to improve the detection capabilities for the markers. In addition, two colors were used for the sub-markers because the system cannot differentiate between the left and right hands in detecting the coordinates of the four main color markers when typing with only one hand or typing while the hands cross over each other. The diameter of color markers is about 7mm. the main marker’s position is the center of fingernail and sub marker’s position is between fingernail and first knuckle joint. The positions of the markers influence both position precision of the finger-tip and outbreak of the occlusion. The positions are determined from viewpoint of system reliability.

In order to detect the region for each colored marker, we applied a hue filter to the video obtained from the web camera and set a hue (H) from the HSV color range as the threshold. Table 1 shows which colored markers are paired with which fingers. Table 2 shows the filter values for each color. Each filter value has been adjusted so as not to falsely detect other colored markers, the black and gray areas around the keyboard, or the skin color on the back of the hands.

4



In this experimental condition, the misdetection based on the hue color range wasn’t seen. The coordinates of each colored marker are obtained through this colored maker detection process, and then the coordinates of each finger are estimated using the main and sub-colored marker pairs. For example, when two green main markers are detected, the one with a vermilion sub-marker next to it is used to estimate the right-hand index finger coordinates, and the one without a vermilion sub-marker next to it is used to estimate the left-hand little finger coordinates. This method is used to estimate the coordinates of all eight fingers.

Table 1 Combination of each finger and color marker.

Table 2 Hue filter values for each colored marker.

3.2.2 Detection process for key coordinates Figure 5 (a) shows the alphabet keys divided into upper, middle, and lower rows. Figure 5 (b) shows the colored

marker acrylic tags attached to the keyboard. Table 3 shows the hue filter values for each color. A hue filter is applied, similar to the method for detecting the colored marker region for the fingers explained previously. Once the detection process for each colored marker is complete, the coordinates for each alphabet key are estimated by comparing the x-axis (horizontal direction) values in the image for the coordinates of the markers obtained with the same row.

Fig. 5 (a) Markers on keyboard for (b) key position detection.

Hand Finger Color (main) Color (sub)Left Pinky Green None

Ring Blue NoneMiddle Indigo NoneIndex Purple None

Right Index Green VermilionMiddle Blue VermilionRing Indigo WakatakePinky Purple Wakatake

Color Hue maximum value

Hue minimum value

Green 160 120Blue 200 162Indigo 230 202Purple 320 280

Vermillion 35810

3500

Wakatake 104 60

5



Table 3 Hue filter values of HSV per color. (key colored markers).

3.2.3 Estimation process for finger used to type the key

Figure 6 shows a visualization for the estimation process for which finger was used when the F key is pressed. The

coordinates for the eight fingers when the F key is pressed and the coordinates previously obtained for the F key are compared. The Euclidean distances between the F key coordinates and each of the eight fingers’ coordinates are calculated, and the finger with the smallest distance is estimated to be the finger used to type the key. However, estimates cannot be made when distances are 40 cells or greater. This estimation process for the fingers is repeated every time one of the alphabet keys is entered.

Fig. 6 Finger estimating process when “F” is inputted.

4. System evaluation experiment 4.1 Method for system evaluation experiment

Figure 7 shows the experimental method for the system evaluation. When a string of letters is typed, the system

estimates which finger was used to press which key. Afterward, the experimenter visually checks the video of the fingering when the string of letters was typed and determines which finger actually pressed which key. The estimated results generated from the system and the visually confirmed results are then compared, and the estimation accuracy of this proposed method is calculated.

Figure 8 shows the format of the keys to be entered. In this experiment, the experimenter enters the string of letters. The letter strings are formed from 26 letters from A through Z. There are two patterns of letter string combinations: a random combination (“p f e s r f q v f z x l q o m x i k r u”), and a combination that is recognizable (“o h a y o u a r i g a t o u o h a y o u”). This recognizable combination contains familiar words [to Japanese people] like “ohayo” (good morning) and “arigato” (thank you). These 20-letter strings are each repeated five times, meaning that a total of 200 keys are estimated. Note that the typing speed is changed for each of the five times the strings are typed due to differences in estimation accuracy among different speeds.

Rows Color Hue maximum value

Hue minimum value

Upper Purple 320 248Middle Blue 220 176Lower Green 174 104

6



Fig. 7 Experimental procedure.

Fig. 8 (a) Random and (b) recognizable character strings.

4.2 Results

Equation (1) shows how estimation accuracy is calculated. Figure 9 shows experimental results by the two letter

string combinations. The x-axis represents typing speed (CPM), and the y-axis represents estimation accuracy, displaying the estimation accuracies at different speeds used to press the keys. According to the experiment results, estimation accuracy tends to decrease as CPM values increase.

estimation accuracy 100 (1)

E: the number of all estimated characters S: the number of characters the results generated from the system and visually confirmed are the same

Fig. 9 Estimation accuracies for typing speed for (a) random and (b) recognizable patterns.

A random combination A recognizable combination

Enter 20 letters Enter 20 letters

p f e s r f q v f z xl q o m x i k r u

o h a y o u a r i g a t o u o h a y o u

(a) (b)

7



4.3 Discussion 4.3.1 Cases of incorrect estimates

After evaluating incorrect estimates based on the video of the typing and data of the estimated finger coordinates, we

were able to classify the causes into two categories. Figure 10 shows an example of the first cause for the incorrect estimates: occlusion. In this phenomenon, the colored

marker(s) of the finger used to press the key is hidden by another finger or another finger’s colored marker(s), resulting in an incorrect estimate. This was observed often when typing A and S. In the recognizable string of letters, A was typed often, resulting in a larger number of incorrect estimates due to this first cause.

Figure 11 shows an example of the second cause for the incorrect estimates: that the image sampling (30 FPS) could not keep up with the fingering. In this phenomenon, colored markers in the video obtained from the web camera could not be distinguished, causing the incorrect estimates.

Figure 12 shows the number of times incorrect estimates occurred for different typing speeds (CPM) separated into the first cause and the second cause.

Fig. 10 Example of incorrect estimate cause by occlusion.

Fig. 11 Example of incorrect estimate due to image sampling unable to keep up with fingering.

8



Fig. 12 Number of incorrect estimates for each cause categorized by typing speed.

4.3.2 System effectiveness From the experimental results and discussion, the system is considered to generate accurate estimates when the CPM

is 300 or less, excluding instances when incorrect estimates are caused by occlusion. The subjects of the system used in this study were non-touch typists, and according to the survey results detailed in 2.2, the CPM scores of these subjects are highly unlikely to exceed 300. In other words, when this system is used for non-touch typists to learn touch typing, its methods are likely to give correct estimation results, with the exception of incorrect estimates due to occlusion. Furthermore, we are thinking it to be able to evade the problem because the learning support system shows an appropriate reply to the user when the system cannot extract it.

5. Conclusion and further research

The ultimate goal of this study was to develop a learning support system for touch typing based on fingering detected

when typing, and to evaluate the possibility of teaching touch typing to non-touch typists through practicing with this system. In this study, we proposed a method to determine which fingers are used to press which keys by using video of fingering made when typing and data of the pressed keys. Furthermore, we conducted evaluation experiments with the system using this method and evaluated the effectiveness of the system by calculating estimation accuracy. According to the experiment results, estimation accuracy tended to decrease as CPM values increase. We learned that there were cases in which incorrect estimations were made, and we classified the causes into two categories: errors caused by occlusion and errors due to the image sampling being unable to keep up with the fingering. It is evident from the experimental results and discussion that when this system is used for non-touch typists to learn touch typing, it gives correct estimation results (with the exception of incorrect estimates due to occlusion) and is effective. As the next step, we will develop the learning support system to learn touch typing based on this proposed method. After that, it is needed to perform the experiments for some participants who aren’t used to touch typing by using the system, and to compare the skill up of the touch typing with other general systems. In this study, we used the general keyboard and web camera, and didn’t try different type of keyboard and web camera. In this experiment, the reliability of the system is satisfied. However, in the case of different type of instruments, for example, small keyboard with small key pitch, web camera with low resolution, and so on, the reliability is not guaranteed.

For further research, we suggest using this system to facilitate the learning of touch typing for non-touch typists. More specifically, learners should be made to enter a sequence of letters shown on-screen. Messages should be displayed on-screen when an incorrect key is pressed or when the recommended finger is not used. We believe that touch typing skills can be easily acquired by making users aware of what fingers they use to press the keys when typing. Acknowledgements

Some part of this study was funded by KAKENHI (18K11414).

9



References Ando, A. and Ito, T., Consideration related with typing skills and eye-movements, Research Bulletin of Information

Processing Center of Miyagi University of Education, No.17 (2010), pp.43-46 (in Japanese). E-typing CO., Ltd., e-typing (online), available from <http://www.e0typing.ne.jp/>, (accessed on 7 July, 2018) (in

Japanese). Kojima, K., On the information literacy of the net generation in the BYOD era, Journal of Nagoya Gakuin University;

Social Science, Vol.52, No.3 (2016), pp.45-57 (in Japanese). Matsunaga, Y., Development and practice of typing software for examinations, Research Report of Aichi University of

Education, No.56 (2007), pp.211-215 (in Japanese). Matsuyama, C., Nakajima T. and Ishii N., The effect of touch-typing practice in information processing classes, The

Transactions of the Institute of Electrical Engineers of Japan. C, Vol.122-C, No.12 (2002), pp.2189-2190 (in Japanese).

Ministry of Education, Culture, Sports, Science and Technology (MEXT), About information literacy (online), available from <http://www.mext.go.jp/a_menu/shotou/zyouhou/detail/__icsFiles/afieldfile/2017/01/17/1381046_01_1_1.pdf>, (accessed on 7 July, 2018) (in Japanese).

New recruit white paper 2017 (in Japanese), Research of Mynavi Corporation (Feb.2017). Soga, M.,Tamura,T. and Taki,H., Development of a Typing Skill Learning Environment with Diagnosis and Advice on

Fingering Errors, Procedia Computer Science, Vol.22 (2013), pp.737-744. Tamura, K. and Takaoka, U. Analysis of performance characteristics of typing, Information Processing Society of Japan-

International Symposium on Stabilization, Safety, and Security of Distributed Systems (SSS 2012), Vol.2012, No.4 (2012), pp.155-160 (in Japanese).

10

proposal of fingering detection method for touch typing

Documents