Introduction to Embedded Systems Research:Weight precision alternatives

Robert Dick

dickrp@umich.eduDepartment of Electrical Engineering and Computer Science

University of Michigan

1.1040

2.1041

385

2.1039

704

1.1039

36

2.1040

1734

0.1039

1

3.1040

4

4.1039

642

3.1039

4.1040

409

3.1041

396665

5.1042

108612774

5.1040

337

4.1041

10644

6.1044

117

6.1039

609

6.1045

164

6.1040

841434 723938

6.1042

209

5.1041

4 12 164

5.1045

529417

5.1044

140105 88

5.1039

154 10551 90677

7.1039

1248

7.1047

2106

7.1040

29773362 241966 22903106

6.1041

3119 1936 40

6.1047

4128 253

8.1050

4

8.1042

9.1050

24784

8.1039

89632

8.1044

2840

8.1040

1088

9.1039

10.1040

4

11.1039

957

10.1050

144

11.1040

156

10.1041

32 16

10.1047

87780

10.1045

2152

10.1039

1165

12.1040

145

13.1052

2404

12.1042

24

12.1045

33008

12.1044

8217

12.1041

8

12.1039

135427

14.1049

113

14.1040

229

13.1050

132

13.1042

74433

13.1041

17187

13.1040

2715

13.1039

170059 90

15.1040

16

15.1050

242 1225

14.1050

237

14.1042

6200

14.1041

4

14.1044

720

14.1052

20

14.1039

84 36939

16.1040

88

15.1049

6

15.1039

56

17.1054

2919

16.1050

129

16.1041

36

16.1047

49154

16.1045

2632

16.1049

27

16.1052

16

16.1039

222734

18.1048

133

18.1039

3439

18.1040

241

17.1050

16

18.1049

36 339224

17.1042

172832

17.1041

49620 448

17.1040

1376493 2883

17.1045

72

17.1044

1073

17.1049

55441826

17.1048

124 3648290

17.1052

3547

17.1039

2484110477445

17.1047

24

19.1040

72

19.1039

6 88 60631617

18.1041

28 76

18.1047

3305

18.1054

13744

20.1040

109

20.1039

269

19.1052

8

19.1047

11712

19.1049

10

19.1054

7520

21.1039

82

20.1049

5

20.1047

4896

20.1054

864

22.1040

4

22.1050

23.1040

4

22.1039

144

24.1058

3389

23.1050

76

24.1040

4

23.1042

17528

23.1041

4

23.1054

24

23.1044

6234

23.1058

261

23.1049

4

23.1052

2944

23.1039

3069658

25.1040

80

24.1050

4

24.1039

58

25.1039

26.1040

4

27.1039

489

26.1050

4

27.1040

4

26.1047

3808

26.1045

2248

26.1058

113 80

26.1049

11

26.1039

66

26.1054

840

28.1055

84266 1542

27.1042

1229

27.1041

29619

27.1058

12

27.1049

3984

27.1048

35337

29.1040

262

29.1056

164

29.1039

742

28.1050

4 2464

28.1042

2137912

29.1055

1128

28.1041

4 2633 4

28.1040

2716132 691

28.1058

24 84 32

28.1049

36176 3

28.1048

1192 48

28.1039

365 110957 24475

29.105029.104229.104129.104729.104529.105829.104929.105229.1054

2

3

4

5

6

7

8

0 1 2 3 4 5 6 7 8

Pow

er

(mW

)Time (s)

35 40 45 50 55 60 65 70 75 80 85 90

-8 -6 -4 -2 0 2 4 6 8

-8

-6

-4

-2

0

2

4

6

8

35 40 45 50 55 60 65 70 75 80 85 90

Temperature (°C)

Position (mm)

Temperature (°C)

Glia

Remember when neurons were the only nervous system cells to signal?

Astrocytes also signal.

May be the proximal cause of fMRI blood flow changes.

2 R. Dick EECS 598-13

Digital biological weight representations

R. Wessel, C. Koch, and F. Gabbiani, “Coding of time-varying electric fieldamplitude modulations in a wave-type electric fish,” J. Neurophysiology,vol. 75, no. 6, June 1996.

3 R. Dick EECS 598-13

Analog biological weight representations

D. Debanne, A. Bialowas, and S. Rama, “What are the mechanisms foranalogue and digital signalling in the brain?” Nature Reviews Neuroscience,vol. 14, pp. 63–69, Jan. 2013.

4 R. Dick EECS 598-13

Conventional machine learning weight representation

sign · significand · 2exponent

Single-precision: 24-bit significand, 8-bit exponent.

Double-precision: 53-bit significand, 11-bit exponent.

(Unnecessarily) large dynamic range and precision.

5 R. Dick EECS 598-13

Reduced-precision floating point

16-bit (half-precision) common.

11-bit significand, 5-bit exponent.

Often reduces accuracy by a few percent or less.

6 R. Dick EECS 598-13

Fixed point

Integer, with an implied decimal position maintained by programmer orcompiler.

Higher potential efficiency than floating point.

Harder to deal with in practice for programmer or compiler.

Must determine maximum value at each stage in a computation DAG anduse appropriate implied scale.

Integer is a degenerate case of fixed point.

7 R. Dick EECS 598-13

Logarithmic

V. Sze, Y.-H. Chen, T.-J. Yang, and J. Emer, “Efficient processing of deepneural networks: A tutorial and survey,” Proc. IEEE, vol. 105, no. 12, Dec.2017.

8 R. Dick EECS 598-13

Binary

Degenerate case of integer.

Generally results in multi-digit accuracy reduction.

The fact that it works reasonably may be surprising.

Can use structural members to represent non-binary numbers.

This is inefficient compared to conventional number representations.

E.g., 1 + 1 + 1 vs. 1 · 20 + 1 · 21 + 1 · 22.

9 R. Dick EECS 598-13

Other encodings

Hinted at by weight compression research.

E.g., use indexed table of most common weights.

Other encodings possible.

10 R. Dick EECS 598-13