Benchmarking Discussion Group

Telluride Cognitive Neuromorphic Engineering Workshop

2014

Major outcomes

• We need NE-specific benchmarking to:– Improve the performance of NE systems with apples-to-apples comparison– Convince investors and industry that our systems have high performance

• Benchmarks should be multi-variate, i.e. not just measuring accuracy but also any or all of the following:

– Biological realism – Necessity for tuning – Power consumption or performance per

operation or task– Latency / speed – Noise robustness – Area / technology node / resources

– Learning speed – Adaptability – Robustness in real-world problems – Multi-sensory problems – Usability

• Existing BenchmarksThere are a number of NE datasets already, such as MNISTDVSSpikes.There are a number of good datasets from cognate fields e.g. robot tasks, and spike sorting.

• New Benchmarks– Potential for NE-specific benchmarks encompassing all variables of interest– Potential for benchmarks with predetermine sample statistics, characteristics etc.– Annual (at Telluride) hardware benchmarking with common input (scenes, audio etc.)

• Hosting of Datasets– Giacomo is keen to host on INE website, with mirror sites at e.g. other Universities. UWS

is available to host immediately.– Data will be available under licence – we suggest Open Data Commons Attribution ODC-

BY 1.0.– Datasets should include readme files or ascii headers, and any code necessary to

translate e.g. jAER to MATLAB.

• Dissemination– Giacomo has suggested the possibility of two papers in FNE - a general overview of the

problem, and a specific paper on benchmarking of spatio-temporal pattern recognition systems (both hardware and software). These may form part of a special issue under Michael’s leadership.

– Anyone who wants to participate in these papers, please email Jon Tapson, jtapson@gmail.com

– Thanks to everyone who participated, esp. Danny who wrote up the meeting notes.

Hard Problems in Neuromorphic Engineering

• We should come up with a list of Neuromorphic Challenges that we think only neuromorphic engineering can solve

• These challenges can raise awareness and drive the field, as the Hilbert Problems did a century ago

• There should be an annual meeting to update and evaluate the latest progress

• This list may be taken as a definition of Neuromorphic Engineering, so we should be careful how we construct it

• Example challenges: – Face recognition within energy and time constraints – Speaker recognition within noise, energy, and time constraints – Adaptive Motor Control – Operant classical conditioning – Limited memory / Limited time response