Neural Network Engine (NNE)
• NN is dominated by memory accesses not MAC operations
• 8-bit wordlength has little impact on model accuracy
Using the three optimization techniques in NNE reduced:
oMemory accesses by 5.6x compared to baseline xDSP
oEnergy costs by 5.5x compared to baseline xDSP
oMemory required for parameters by 3x
• NNE always performs in a deterministic number of cycles for any arbitrary network
Conclusion
Results
Deterministic number of cycles:
3. Two-step scaling• Solves inefficient individual scaling of activations
3.1 Within a vector (when writing results)
3.2 Across a layer (when reading results in a next layer)
L1 L2 L3 L4
*minimum number of required accesses. This number can grow depending on how many times previously computed values must be retrieved and shifted additionally
**this number does not contain energy required for fetching and decoding the instructions, which would increase the energy costs even more
Table 2: 24-bit DSP vs 8-bit NNE. Energy cost is estimated on a 45nm process [4]
NNE
• Inference = processing of a one-second
audio file
• Required MAC operations: 6,600/inference
• Total cycles: 7,295/inference
• Memory not accessed in only 45 cycles
(0.62% of all cycles)
• Accuracy: 80.28% (4,890 audio files)
L1 L2 L3 L4
GroupNumber of
shifts/groupAdditional
shifts
Green 2 ?
Blue 1 ?
Orange 3 ?
GroupNumber of
shifts/groupAdditiona
l shifts
Green 2 3 - 2 = 1
Blue 1 3 - 1 = 2
Orange 3 0
Benchmarking and improving NN execution on DSP vs.
custom accelerator for hearing instrumentsZuzana Jelčicová*, Adrian Mardari*, Oskar Andersson*, Evangelia Kasapaki*, Jens Sparsø✺
Demant A/S, Smørum (Denmark)*; Technical University of Denmark, Lyngby (Denmark)✺
N = number of layers excluding an input layer
O = the number of output neurons in the current layer
A = number of inputs/activations to the layer
NNE includes three optimizations:
1. Reduced wordlength • 24 to 8-bit parameters (inputs, weights, biases)
• On-the-fly symmetric quantization (each layer individually)
• Insignificant loss in accuracy
2. Several MACs in parallel•Input & output stationary processing
•Considerations: 96-bit memory interface and 8-bit data
Table 1: Number of read operations (inputs and weights) needed for inference using 8-bit parameters.
Biases are excluded as they are negligible.
DSPs in general
• Datapaths with 16 and 24-bit elements
Neural network (NN) inference
• 8 bits are usually sufficient
o More frequent overflows → additional memory accesses,
increased energy costs, and unpredictable timing behavior.
xDSP vs NN processing engine (NNE)
xDSP → Oticon’s DSP-based platform used for obtaining baseline
results
NNE achieves further power optimizations by exploiting three mutually
dependent techniques:
1. Reduced wordlength – 24 to 8 bits with insignificant loss in
accuracy
2. Several MACs in parallel – reduced wordlength → processing
of more data at once
3. Two-step scaling
• eliminates the need to reload and scale already computed
outputs to maintain the ratio
• makes our NNE always execute in a deterministic number
of cycles
Introduction
• Feedforward fully connected DNN model (250x144x144x144x12) [1]
trained on 12 words (first two categories represent silence and
unknown words): "yes", "no", "up", "down", "left", "right", "on", "off",
"stop", "go“
• ReLU and softmax (output layer) activation functions
• Dataset: 65,000 one-second utterances of 30 words [2]
• Input to DNN is the flattened feature matrix
• Input signal (length L) is framed into overlapping frames (length
l=40ms) with a stride (s=40ms), giving 25 frames as input/inference
→
• 10 frequency bins → 250 inputs
Keyword Spotting Application (KWS)
Figure 1: KWS system consisting of a feature extractor and a NN based classifier
• Generic audio DSP→ baseline for comparisons
• 24 bits for representing data (Q5.19)
• SIMD4 (96-bit memory interface)
Digital Signal Processor (xDSP)
[1] https://github.com/ARM-software/ML-KWS-for-MCU
[2] https://ai.googleblog.com/2017/08/launching-speech-commands-
dataset.html
[3] Quantization algorithms. https://nervanasystems.github.io/
distiller/algo_quantization.html
[4] Horowitz, M. 1.1 computing’s energy problem (and what we can do
about it). In 2014 IEEE International Solid-State Circuits Conference
Digest of Technical Papers (ISSCC) (Feb 2014), pp. 10–14
References
Figure 2: System
overview diagram
for the NNE
[3]
