February 1, 2011 1

February 1, 2011 2http://www.2wheelbikes.com/sitebuilder/images/cable-lock-comb-bike-accessories-469x345.jpg

How many bit strings of length n?

Adding one bit doubles the number

February 1, 2011 3

length Bit strings of that length Count

1 0, 1 2 = 21

2 00, 0110, 11

4 = 22

3 000, 001, 010, 011,100, 101, 110, 111

8 = 23

n 0 followed by strings of length n-1,1 followed by strings of length n-1

2n-1 + 2n-1 = 2 x 2n-1=2n

Text8 bits per character

“A” = 01000001

“(” = 00101000

How many combinations of 8 bits?

2· 2· 2· 2· 2· 2· 2· 2 = 28 = 256

February 1, 2011 4

Hexadecimal Digits

February 1, 2011 5

0000 0001 0010 0011 0100 0101 0110 0111

0 1 2 3 4 5 6 7

1000 1001 1010 1011 1100 1101 1110 1111

8 9 A B C D E F

ASCIIAmerican Standard Code for Information Interchange

Character represented by Hex xy, e.g. 4B is “K”

February 1, 2011 6

xy

0 1 2 3 4 5 6 7 8 9 A B C D E F

0

1

2 sp ! " # $ % & ' ( ) * + , - . /

3 0 1 2 3 4 5 6 7 8 9 : ; < = > ?

4 @ A B C D E F G H I J K L M N O

5 P Q R S T U V W X Y Z [ \ ] ^ _

6 ` a b c d e f g h i j k l m n o

7 p q r s t u v w x y z { | } ~ del

ASCII UnderneathEmails

Web pages

February 1, 2011 7

February 1, 2011 8http://farm4.static.flickr.com/3021/2494096946_2bf86f8571.jpg?v=0

What if you need more than 256 characters?

Unicode

32 bits per character

February 1, 2011 9

February 1, 2011 10

February 1, 2011 11

February 1, 2011 12

February 1, 2011 13

How many Unicode characters?

32 bits each, so there are in all

February 1, 2011 14

†

2324,294,967,296

February 1, 2011 15

Binary counting1+1=10, or 1+1=0 and carry 1

February 1, 2011 16

+1 +1 +1

0 1 1 0 0 1 1 1

+ 1

0 1 1 0 1 0 0 0

Positive and Negative Numbers

Signed and unsigned numbersUnsigned: 28=256 bit patterns represent 0 … 255Signed: 28 bit patterns represent -128 … +127Leftmost bit = sign bit: 0 => 0 or pos, 1 => negLargest 8-bit positive number = 01111111 = 1270 = 00000000Most negative negative number =

10000000 = -128

February 1, 2011 17

Negative numbers

February 1, 2011 18

+1 +1 +1 +1 +1 +1 +1

1 1 1 1 1 1 1 1

+ 1

0 0 0 0 0 0 0 0

-1 = 11111111so addition works the same for positive and negative numbers

Biggest NumbersBiggest positive number = 01111111 (like

999999 on a car odometer)

Most negative negative number = 10000000

February 1, 2011 19

Biggest Positive Number + 1 “=”

Most Negative Negative Number

February 1, 2011 20

+1 +1 +1 +1 +1 +1 +1

0 1 1 1 1 1 1 1

+ 1

1 0 0 0 0 0 0 0

OVERFLOW!

The Comair Christmas “Glitch”

16 bits for monthly count of crew changes

Biggest positive 16-bit number =32,767

December was a bad month, lots of snowstorms, lots of flights rescheduled

As Christmas approached the count went from 32,767 to -32,768 by adding 1

February 1, 2011 21

The Y2010 “Glitch”Binary representation of decimal 10 =

00001010

Binary Coded Decimal = write decimal 10 as sequence of 4-bit binary codes for digits Decimal 10 = BCD 0001 0000

What if you write decimal 10 in BCD but some other program reads it as decimal? Binary 0001 0000 = decimal 16

February 1, 2011 22

Bytes 1 byte = 8 bits = 2 hex digits = 1 character

210 =1024 bytes = 1 kilobyte = 1KB

220 =1,048,576 bytes = 1 megabyte = 1MB

230 bytes = 1 gigabyte = 1GB = “a billion”

240 bytes = 1 terabyte = 1TB = “a trillion”

250 bytes = 1 petabyte = 1PB = “a quadrillion”

260 bytes = 1 exabyte = 1EB = a quintillion bytes

270 = zetta

280 = yotta

February 1, 2011 23

KAll this terminology based on the accident that

Which is 1K?

There are new standard names: 1 kibibyte = 1000 bytes vs. 1 kilobyte = 1024 bytes

But almost no one uses “kibi-”, “mebi-”, etc.

February 1, 2011 24

†

10242101031000

February 1, 2011 25

February 1, 2011 26

Moore’s Law (1965)

The number of transistors on a silicon chip doubles every 18 [or 12, or 24] months

1965: 64 = 26

2008 = 2 billion ~ 231

25 doublings in 43 years = one doubling every 20+ months

February 1, 2011 27

Example of linear increase

February 1, 2011 28

Example of exponential increase

Now for the y axis use instead lg(y) = the exponent e such that 2e=y

February 1, 2011 29

Same plot, using lg(y) instead of y

February 1, 2011 30

One of the Greatest Engineering Achievements

An increase by a factor of 225 is about 30 millionfold

If human speed had increased that much over the past 43 years, we would now be traveling faster than the speed of light

February 1, 2011 31

ProbabilitiesFair coin: P(heads) = 1/2

Fair die: P(rolling 3) = 1/6

Fair card deck: P(hearts) = 1/4

P(ace) = 1/13

February 1, 2011 32

Probabilities of Independent Events Multiply

P(heads and then heads) = 1/2 · 1/2 = 1/4

P(3 and then 4) = 1/6 · 1/6 = 1/36

P(ace and ace) = 1/13·1/13 = 1/169 ≈ .0059 but only if the first card drawn is replaced and the deck is completely reshuffled, otherwise the events are not independent

P(ace and ace without reshuffling) = 1/13 · 3/51 ≈ .0045

February 1, 2011 33

Unlikely Events

How likely are 100 heads in a row?

(1/2)100 ≈ 10-32 =

.00000000000000000000000000000001

February 1, 2011 34

How Small is 2-100 ≈ 10-

30?Age of universe ≈ 1018 sec = 1027

nanoseconds (1 nanosecond = 1 ns = 1 billionth of a second = 10-9 sec)

If you do all 100 coin flips in a billionth of a second, you will get the 100-heads event about once every thousand lifetimes of the universe

1030 = 103 ·1027

This is “never” for all practical purposes

February 1, 2011 35

Morse’s telegraph1844 1848

February 1, 2011 36

Morse Code (1838)A B C D E F G H I J K L M

.- -… -.-. -.. . ..-. --. …. .. .--- -.- .-.. --

N O P Q R S T U V W X Y Z

-. --- .--. --.- .-. … - ..- …- .-- -..- -.-- --..

February 1, 2011 37

Morse Code (1838)A

.08B

.01C

.03D

.04E

.12F

.02G

.02H

.06I

.07J

.00K

.01L

.04M

.02

.- -… -.-. -.. . ..-. --. …. .. .--- -.- .-.. --

N.07

O.08

P.02

Q.00

R.06

S.06

T.09

U.03

V.01

W.02

X.00

Y.02

Z.00

-. --- .--. --.- .-. … - ..- …- .-- -..- -.-- --..

February 1, 2011 38

How Long are Morse Codes on Average?

Not the average of the lengths of the letters: (2+4+4+3+…)/26 = 82/26 ≈ 3.2

We want the average a to be such that in a typical real sequence of say 1,000,000 letters, the number of dots and dashes should be about a·1,000,000

The weighted average:

(freq of A)·(length of code for A)

+ (freq of B)·(length of code for B)

+ …

= .08·2 + .01·4 + .03·4 + .04·3+… ≈ 2.4February 1, 2011 39

Data vs. InformationMessage sequence:

“yea,” “nay,” “yea,” “yea,” “nay,” “nay” …

In ASCII, 3·8 = 24 bits of data per message

But if the only possible answers are “yea” and “nay,” there is only 1 bit of information per message

Entropy is a measure of the information content of a message, as opposed to its size

Entropy of this message sequence = 1 bit/msg

February 1, 2011 40

Squeezing out the “Air”Suppose you want to ship pillows in boxes

and are charged by the size of the box

Lossless data compression

Entropy = lower limit of compressibility

February 1, 2011 41

Claude Shannon (1916-2001)

A Mathematical Theory of Communication (1948)

February 1, 2011 42

Communication over a Channel

February 1, 2011 43

Source Coded Bits Received Bits Decoded Message

S X Y T Channel symbols bits bits symbolsEncode bits before putting them in the channelDecode bits when they come out of the channel

E.g. the transformation from S into X changes“yea” --> 1 “nay” --> 0

Changing Y into T does the reverseFor now, assume no noise in the channel, i.e. X=Y