The Effect of Coding on Computation

Adina Lederhendler

Topics in Biological Physics 23/12/08

Shannon: A universal Turing machine with two internal states. Automata Studies, 1956

Miller: The magical number seven, plus or minus two: some limits on our capacity for processing information, Psychological Review, 1956

Outline

What is a code?

Abstract computation

Different codes for universal Turing machines

Real-world computation

The use of coding in working memory

What is a code?

Code – Set of rules for translation of input into output.

Input 1

Input 2

Input 3

Input NI

Output 1

Output 2

Output 3

Output NO

What is a code?

Examples of codes:

ASCII code

English alphabet

Binary digits

Genetic code

Nucleotide sequences

Amino acids

Turing machine

Information encoded as symbols on tape.

Configuration: Symbol +

Internal state.

X Y

Si

Z

Reading/ writing head

Turing machine

X Y

Si

Z

symbol + state symbol + shift + state

complexity of machine (size of code)

nm

CodeConfiguration Step

Universal Turing Machine

X Y

Si

Z

Input tape and code of a Turing machine T

UTMSame output as T.

X Y Z

UTM not related to any specific computation.

Universal Turing machine

Shannon, Automata Studies, 1956

Shannon, 1956 – what are the limitations on the code of a UTM?

Minimal number of states for a UTM?

Minimal number of symbols for a UTM?

No 1-symbol UTM

Cannot carry any information

No 1-state UTM

Each step depends only on the letter currently being read.

Cannot carry enough information

How do alternative codes affect the computation?

Universal Turing machine

Universal Turing machine ASymbols

miAi ,,1 m

States

njS j ,,1 n

Symbols

Universal Turing machine BStates

, 2

? ?Symbols

Universal Turing machine CStates

1,0 2

? ?

Two-state UTM

Symbols

iB m

yxjiB ,,,

mi ,,1

RLy

x

nj

mi

,

,

,,1

,,1

mn4

Symbols that represent

intermediate states during computation

Symbols that represent

input/output

Universal Turing machine B

States , 2

Information about states of A carried by symbols of B (symbol-state tradeoff)

Two-symbol UTM

Universal Turing machine C

States

sxxxi

i

T

T

,,, 21

Information about symbols of A carried by states of C (symbol-state tradeoff)

1,,1

1,0

,,1

s

x

ni

j

Symbols

1,0 2

s

s

xxxi

xxxi

R

L

,,,

,,,

21

21

si

si

V

U

,

, 12 n

12 n

222 n

4,3,2,1,0iAA tape:

3

0

1

2

3

4

0 00

0 0 1

0 01

0 11

0 01

1,0iCC tape:

Example – Machine A: m = 5, n

Two-symbol UTM

iS jS

2 1

iT0Ct

00 1

Two-symbol UTM

0 0

000

Machine A

Machine C

iS jS

2 1

00 1

0,iT 1,0,iT2Ct

Two-symbol UTM

0 0

000

Use information about machine A

iS jS

2 1

0,0,jL

10 1

3Ct

Two-symbol UTM

0 0

000

iS jS

2 1

0,jL

10 0

4Ct

Two-symbol UTM

0 0

000

iS jS

2 1

1,jV

10 0

Move to beginning of the next input

2,jV6Ct

Two-symbol UTM

0 0

000

iS jS

2 1

10 0

jT

Ready to begin next step.

7Ct

Two-symbol UTM

0 0

000

Symbol-state product

ABC

Symbols

States

Product

m

n

mn

2

mmn 4

mn8~

2

mnmn 86

mnmn 1612~

Shannon: What is the minimum symbol-state product required to construct a universal Turing machine?

Minimal Turing machines

Minsky, 1962

7-state, 4-symbol UTM.

Later efforts

Find additional minimal (size) UTMs.

Find more efficient minimal UTMs.

Woods, Theoretical Computer Science, 2008

Real world systems

Best code for specific task

Real-world considerations such as:

Efficiency:

How much time, what resources can we devote to the computation?

Accuracy:

How much noise is present in the system? How accurate does the

computation need to be?

Evolution/adaptation to specific task

Biological systems

Working memory

AMOUNT OF INFORMATION PER ITEMMiller, The Psychological Review, 1956

Miller, 1956

Attempt to quantify working memory capacity

How many items can we recall immediately after being presented with a sequence?

Retrieval

Adaptation through learning

Sequences of items

Recoding into “chunks” and storage in short-term memory

Working memory

New codes = More kinds of “chunks”

Working memory

Baddeley & Hitch, 1974

PTVOBK

All of B is included in A Some of C is included in B

What relation must there be

between A and C?

Summary

• A code is a set of rules for carrying out a computation.

• Different codes can be used to perform the same computation.

• The choice of code may affect the efficiency and accuracy of a computation.

• Biological systems can adapt codes to specific tasks.