ece 539 final project - university of...
TRANSCRIPT
1.Introduction
1.1.Problem Statement
Design of a speech recognition Artificial Neural Network that is able to distinguish between the figures 0 to 9 and the words “yes” and “no”.
1.2.Motivation
1.2.1.Applications
This system can be very useful for cellular phones so that the user only has to say the digits to dial the desired telephone number.
In addition, it can be used in elevators to get to the desired floors without having to push any buttons.
It can be used almost everywhere that choices require number selections
1.2.2.Comparison to huge systems
Conventional speech recognition systems contain huge libraries of all the stored patterns of the words to be recognized. This requires, of course, a large memory to store the data. In addition these systems need a fast processor responsible for the non-linear time adjustment of target patterns to the respective words to be recognized and their comparison to the patterns. After the comparison an error is calculated which represents the distance between a pattern and a word. The pattern with the smallest error will be the solution for the recognition.These things are too expensive for such small application.That is why I tried to train a neural network that can do the same.
1.3.Pre-observation of the Artificial Neural Network
I designed a Multilayer Perceptron (MLP) Network using the Back Propagation Algorithm of Professor Hu to solve this classification problem. Corresponding to the number of words I want to distinguish at the end, this is a question of 12 different classes. This limits the number of outputs of the network to 12.The input data are extracted feature values that represent the signal. Here I always store the first two formants of the spectrum of a signal’s part corresponding to the first two maxima of the spectrum. That is why the process is also called “Two-Formants-Recognition-System”.
I implemented the whole algorithm using Matlab.
2.Work Performed
2.1.Listing of Steps
2.1.1.Raw data collection
Recording of 50 samples for every word using “Cool Edit”o Sampling frequency : 44100 kHzo Channels : monoo Resolution : 32 bit (quantization of the amplitude values)
Storage of each sample in its own text file (stored amplitude values for all time values) Summary of all text file names in a vector “text_files” saved in “textf.mat” using the
function “textfiles.m”
2.1.2.Extraction of input data
Implementation of the text file vector to the function “spectrum_complete.m”o Adjustment of all signals to uniform signal length of 500 mso If real signal length smaller than 500 ms missing values filled with zeros
No time stretching or shrinking No change of the feature data
Windowing of the signal every 10 ms to parts of length of 20 ms using the Hann window (multiplication of signal with the window)
=> 49 separate sections for each sample
Calculation of the spectrum of these sections using Discrete Fourier Transformation
(whole spectrum is periodical with 22050 Hz, but only the first 4 kHz are of interest)
Implementation of the Cepstral Algorithm for the calculation of the spectrum’s contour (the spectrum becomes smoothed)
o Calculation of the Cepstral Coefficients by applyingthe Inverse Discrete Fourier Transformation after using the natural logarithm on the spectral coefficients => change to “quefrency” domain (kind of time)
o Front values contain information of the high frequencies, back values contain the information of the low frequencies
o Short pass liftering of the Cepstral Coefficients(taking only the first 147 Cepstral Coefficients)
o Calculation of the smoothed spectrum by applying the Discrete Fourier Transformation to the Cepstral Coefficients
Storage of the first two frequencies for the two first formants in a vector=> in this example these two values are 441 and 804
o The vector for every sample becomes 98 by 1o Connection of this vector with the desired values corresponding to the
respective class and storage in a matrixo This matrix consists of 50 of these vectors for every word so that its dimension
is 600 by 110.o Storage of these “input_data” the file “input.mat”
2.1.3.Input data for the MLP-Network
Scaling of the input data to [-1,1] using “scale_data.m”
Random division of the “input_scaled.mat” data into the files “training.mat” and “testing.mat” in a ratio of 9 to 1 using the file “rand_files.m”(integrated in the Back Propagation Algorithm files “bp_all.m” and “bp_ex.m”)
Implementation of these input data to the MLP-Network
Sizing and optimisation of the network
2.2. Used algorithms and formulas
2.2.1. Discrete Fourier Transformation (DFT)
Reference : “Signal Analysis and Recognition” Rüdiger Hoffmann
Actually I used Short Time Discrete Fourier Transformation (DFTF), which corresponds to the convolution of the spectra of the signal and the window function.
where h(k) represents the k-th value of the window function.The same solution occurs when multiplying the signal with the window function and applying the normal DFT to the product.
DFT (FFT)
2.2.2. Cepstral Algorithm
Reference : “Signal Analysis and Recognition” Rüdiger Hoffmann
Human speech consists of two different parts. At first the vocal chords produce an origin signal that has a special frequency. This frequency is about 150 Hz for men and about 300 Hz for women. After the production of the origin signal, the air is modulated by the vocal tract that consists of the pharyngeal, the oral and nasal cavities. This modulation causes the high frequencies in the voice. The maxima of these frequencies represent the different tones.The Cepstral Algorithm is used to separate these two signal parts.The modulation of the signal corresponds to a convolution of the signal with the system’s function of the vocal tract. This equals to a multiplication of both spectra.
# corresponds to the convolution operator F represents the Fourier Transformation
Applying the Inverse Fourier Transformation after using the logarithm of that expression, separates the two parts of the speech, because they concentrate on different ranges in the time domain.
The production of a real signal enables the use of the “Real Cepstrum” which is defined in that way:
if x is an energy signal
The calculation of the Cepstral Coefficients causes a change into the “quefrency” domain, a kind of logarithmic time domain. The information of the articulation are encoded in the smaller coefficients. That is why they are applied to a short pass “lifterer” with the border coefficient “c_border”. The corresponding coefficient to the respective frequency can be calculated in the following way:
c ≈ sample frequency / frequency
border frequency = 300 Hz => c_border ≈ 44100 Hz / 300 Hz =147
So the first 147 coefficients are taken and the DFT applied to them. After that the logarithm is cancelled by using the exponential function and calculating the root of the expression to get the spectral coefficients of the smoothed spectrum (contour).With this contour it is possible to determine the maxima - the formants - of which the first two are the representative features I used.
2.3. Optimisation of MLP
I chose a Multilayer Perceptron-Network and used the Back Propagation Algorithm of Professor Yu Hen Hu to solve this classification problem.The network consists of 98 inputs which result of 49*2 formants for each sample and 12 outputs corresponding to the number of different words.
At first I tried to design a two-word recognition system to get an idea of the size of the net. This resulted in a net that had 3 hidden layers each with 12 hidden neurons to let the algorithm converge with a classification rate of 100%.These values I also used for the classifier for the 12 words in the beginning, and after many tries and tests I established that these dimensions cause the best results for the classification.Then I tried to optimise parameter after parameter to get a better result.I began with the number of samples per epoch and fixed it to 8.Afterwards I changed the number of epochs and the number of epochs before convergence test to 200000 and 150.After that, I experimented with the learning rate and the momentum and realized that it is much better to shrink both. So I finally used 0.01 for the learning rate and 0 for the momentum. This seems to be similar to the problem of choosing a smaller step size for the LMS-Algorithm to shrink the misadjustment – actually the Error Correcting Learning is a kind of LMS-Algorithm.Subsequently, I tried to find a better solution by changing the scaling range of my input data. I established that it is very useful to take a range of [-1,1] instead of [-5,5].With these changes the classification rate reached 87% for training and only 65% after testing.That was a large progress, but a testing rate of 65% isn’t useful for a speech recognition system, so I had to try to improve upon it.The distinction between training and testing seemed to be caused by the similarities of the words zero and seven, which the network had problems deciding between. Also the word four made small problems. I tried to use a tuning set of 10% or 30% size of the input data, but that didn’t make any improvements.That’s why I thought about designing a neighbour network - an expert network - which is responsible for the distinction between the words zero, four, and seven if the main network gets a maximum output value for one of these words. That caused a large increase of the classification rate for testing after I had connected both networks together.
2.4.Program files
My programs can be tested in setting the path of “Matlab” to the respective folders and in starting the demo files.(Any overwritten necessary mat-files can be restored out of the folder “Original_data”)
Folder 1_Data_Collection
o Shows the data collection in running “data_collection_demo.m” Folder Speech_Samples (contains the text files of the samples) data_collection_demo.m getdata.m maximum.m scale_data.m spectrum_complete.m textfiles.m window_functions.m
Folder 2_Trainingo Folder Main_network
Shows the training process of the main network in running “training_main_demo.m” actfun.m actfunp.m bp_all.m bpconfig_all.m bpdisplay.m bptest.m cvgtest.m rand_files.m randomize.m rsample.m scale.m training_main_demo.m input_scale.mat
o Folder Expert_network Shows the training process of the expert network in running training_expert_demo.m”
actfun.m actfunp.m bp_ex.m bpconfig_ex.m bpdisplay.m bptest.m cvgtest.m data_choice.m rand_files.m randomize.m rsample.m scale.m training_expert_demo.m input_scale.mat
Folder 3_Recognitiono Shows the recognition of a word in running “recognition_demo.m”
actfun.m getdata.m maximum.m recognition_demo.m scale_data_2.m spectrum_single.m window_functions.m weights_all.mat weights_ex.mat final_test_classification_rate.txt (shows the recognition results of
all 600 used samples) test.txt (appropriate sample “eight”)
3. Final Results
3.1.Final MLP
logical unit which combines outputs
final output
main network expert network
......... ............
...98 inputs
12 outputs12 hidden neurons12 hidden neurons12 hidden neurons
3 outputs
3.2.Parameter and Results
3.2.1.Main network
(distinguishes between all words)
inputs : 98 outputs : 12 hidden layer : 3 hidden neurons per hidden layer : 12 learning rate : 0.01 momentum : 0 samples per epoch : 8 number of epochs before convergence test : 150 number epochs until best solution : 200000
classification rate after training : 87.6% classification rate after testing : 71.7%
3.2.2.Expert network
(distinguishes between the words 0, 4 and 7)
inputs : 98 outputs : 3 hidden layer : 3 hidden neurons per hidden layer : 12
learning rate : 0.01 momentum : 0 samples per epoch : 8 number of epochs before convergence test : 150 number epochs until convergence : 38000
classification rate after training : 100.0% classification rate after testing : 86.7%
3.2.2.Combined network
The function “recognition_network” uses the best calculated weights of the training.The logical unit combines the outputs of both networks. If the output of the main network is maximum for 0, 4 or 7 then the expert network is asked; this increases the total classification rate of the main network by about 20%.
That means the final classification rate after testing equals to 90.83%.
You are able to apply the final function “recognition_demo “ in using any appropriate text file. The name of the text file has to be changed in the function.
4.Discussion
I only got a classification rate of 70% by using the main network, although I tried to optimise all parameters, the scaling and so on, that is not enough for a speech recognition system.I think this is because of the input data: there are vectors that are too similar to each other causing overlapping of classes. So I decided myself for combining the main network with an expert network which increased the classification rate to 90.83%.
The similarities of the vectors appear because I don’t use any time adjusting algorithms like non-linear mapping of all samples to a standard length. I didn’t want the calculation expenditure to become too large. These are only short words - not longer than one half of a second - so that the shifting between the tones could be kept rather small.I trained the whole network only in using my own speech samples. So the results for samples of other voices can differ much if the person speaks to fast or too slowly. The worst results are achieved if a female voice is tested, because the frequencies lie in a higher range in comparison to a male voice. I tested my algorithm on some persons and the results oscillate quite much.The results that I got could be improved on of course by implementing another expert network (e.g. for the words 1, 9 and no), but instead of 20% improvement, of the classification rate in using this one expert network it would cause only about 1.5 % improvement that would be not very effective in comparison to increasing the network size. But that depends on the size of the network that may be used.In addition I realized that a single recognition of one word often takes too long. That’s probably because I used many loops in my program to implement my algorithm, but this is a point that was difficult for me to improve.
Altogether I can be quite satisfied with these results, but single aspects of it might be improved to better its performance.
Appendix:
References:
“Signal Analysis and Recognition” Rüdiger Hoffmann
“Technical Speech Communication” University of Technology Dresden (Lecture notes) (Institute for Acoustics and Speech Communication)