on inherent complexity of computation, by attila szegedi

On Inherent Complexity of Computation

Attila Szegedi@asz

THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISION. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE.

I won’t tell you anything humankind doesn’t already

know for 40 years.

I won’t tell you anything humankind doesn’t already

know for 40 years.

I won’t tell you anything humankind doesn’t already

know for 40 years.

This is not a Lisp talk.

I won’t tell you anything humankind doesn’t already

know for 40 years.

This is not a Lisp talk. No, seriously.

How complex is your system?

Is it this?

Hardware

Operating system

Application

Configuration

Persistent data

Runtime state

Is it maybe more like this?

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime state

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime state

Electricity

Electric impulses on a wire

Heat

The electricity to heat part is easy.

100011110001011010100100000101111100000100100010101001010100101000010111111000110111011000110101010

1001000101010010101001010000101111110001101110110001100000110011110000110001100000100110111011111110110110010111001001000001010000110010111100101101101110000111011101100101110111011101001101100111100110101000011110111111011101110100110010111011101110100101101100110011001011101110110011111101001101000111010100000110000101010001001100001111010011001011110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011100111110101100101100111000001000111100110110101001010101001111001011110010111011011001011110010111010100000100000111100001100001110001111010001100101101111110010101110110010101110110001110111100000101000101010111000101110101110111011101001110101101001101010110001011011010000110111111101111100101111001011001011100100101101100001011110011110101000001010000100100010100001011111101011011101100111011101101011011011100011110101110001011101011101110111010011101011101111011101100011100011010101100010110110100001101001110111011001111100010110000111000111101011111010100000110100011101001110100111000011101010111110111111001111100101110010111010111101111111010111101001011101100101110010110111111110001101101110110011100101110000110001110111011100001101000111000010101001100110111111000111100001111010011010011101111110111011101010000011010001110100111010011100001110101011111011111100111110010111001011101011110111111101011110100101110110010111001011011111010100001111000011100011110100011001011011011000011110111111011101110100111001011011111101100111010100000110110111000011111000101101110000111001111100101111101110011110000110000101010001011111000111000011010011110010110010111100111110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011110001110101100101100111000001000111100110110101001010101011011000011110010111100111101010000010000011100011110001111001011110000111010010110110001011101110110001111011111100100110100111011101100111101010000111101111110111011011101100101110001111101001101001110111111011101110101000001100011110110011011111110011110010110101000011110111111011101110100110010111011101110100101101101010011110011110000110010111101010000011101001100101111100011101001011111101000111010011011011101100111011100000110001111010001100001111001011100111100101111010011110110101011010100100011010110111100010101010

GET / HTTP/1.1

HTTP/1.1 301 Moved PermanentlyContent-Length: 0Date: Thu, 02 May 2013 09:13:23 GMTServer: Apache/2.2.17 (Ubuntu)X-Powered-By: PHP/5.3.5-1ubuntu7.11X-Pingback: http://geekout.ee/xmlrpc.phpLocation: http://geekout.ee/Cache-Control: max-age=300Expires: Thu, 02 May 2013 09:18:23 GMTVary: Accept-EncodingConnection: closeContent-Type: text/html; charset=UTF-8

GET / HTTP/1.1

HTTP/1.1 301 Moved PermanentlyContent-Length: 0Date: Thu, 02 May 2013 09:13:23 GMTServer: Apache/2.2.17 (Ubuntu)X-Powered-By: PHP/5.3.5-1ubuntu7.11X-Pingback: http://geekout.ee/xmlrpc.phpLocation: http://geekout.ee/Cache-Control: max-age=300Expires: Thu, 02 May 2013 09:18:23 GMTVary: Accept-EncodingConnection: closeContent-Type: text/html; charset=UTF-8

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime state

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime stateC

ompl

exity

Abs

trac

tion

How many layers are there? Are they

inherent? Are they real?

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime state

Hardware

Operating systemVirtual machine

Application

A virtual machine takes bytecode and transforms it into something resembling a native application on the OS/hardware.

Hardware

Virtual machine

Application

Then there are some attempts to even eliminate the operating system.

Hardware

And some to go all the way down to

BitCoin craze

MHash/J

Generic CPU 0.1-1.14

GPU 2.4

FPGA 25

ASIC 160

https://en.bitcoin.it/wiki/Mining_hardware_comparison

BitCoin craze

MHash/J

Generic CPU 0.1-1.14

GPU 2.4

FPGA 25

ASIC 160

https://en.bitcoin.it/wiki/Mining_hardware_comparison

These are actual products

Hardware

HypervisorOperating systemVirtual machine

Application serverApplication

Plugins

ConfigurationPersistent data

Runtime state

Application

Configuration

In languages where code is first-class data, you don’t have to distinguish between the two.

Application = Configuration

• In Lua, recommended data serialization format is Lua source code.

• You can just eval() JSON in JavaScript.

• Lisp. ‘nuff said.

• Standard disclaimer about having your program execute unverified external code.

No, seriously, what is computation?

Computation is a transition from a more

likely to a less likely state.

(Decrease in physical entropy; increase in information entropy.)

Wait what?• In information theory, entropy is the

measure of uncertainty of state.

• In physics, it’s the amount of missing information to precisely describe the state of the system.

• The two usages of the term are therefore exactly opposite.

• Way to troll each other, fellow scientists!

This is very likely This is highly unlikely

This is very likely

•An empty database•An uninitialized disk•RAM when power is turned on

•Quiet network

This is very likely This is highly unlikely

This is highly unlikely

•A populated database•A populated disk•RAM after substantial activity

•Active network

Humans are part of the computation

• Having an ongoing record of people’s thoughts and events persisted in a social network database is less likely than not having them.

• Just think about the combined effort of creating Twitter/Facebook/etc databases

• both people building and operating it, and

• people pouring data into it.

Humans are part of the computation

Humans are part of the computation

• Thinking of boundaries of identities in real world can be quite fascinating.

Humans are part of the computation

• Thinking of boundaries of identities in real world can be quite fascinating.

• Where does a computer end?

Humans are part of the computation

• Thinking of boundaries of identities in real world can be quite fascinating.

• Where does a computer end?

• Where does a person end?

Humans are part of the computation

• Thinking of boundaries of identities in real world can be quite fascinating.

• Where does a computer end?

• Where does a person end?

• Can we influence another person in such a way that part of us actually lives in them?

Humans are part of the computation

Humans are part of the computation

• Where does a computer end?

Humans are part of the computation

• Where does a computer end?

• Where does a person end?

Humans are part of the computation

• Where does a computer end?

• Where does a person end?

• Can we influence another person in such a way that part of us actually lives in them?

Humans are part of the computation

Humans are part of the computation

• Where does a computer end?

Humans are part of the computation

• Where does a computer end?

• Where does a person end?

Humans are part of the computation

• Where does a computer end?

• Where does a person end?

• Can we influence another person in such a way that part of us actually lives in them?

Humans are part of the computation

Arrow of time

• Transitioning from more likely to less likely gives us an arrow of time.

• We naturally presume that more likely states precede less likely ones.

• Opposite cases happen, and they are associated with idea of destruction.

Is unzip not a computation then?

• It creates more likely (decompressed) from less likely (compressed) data!

• It is still a computation, though.

Is unzip not a computation then?

• You started with not having both the compressed and decompressed data, and you end up with both.

• That’s less likely than just having the compressed data.

Arrow of time revisited

• Future is a quantum superposition of all possible successive quantum states.

• Past, too, is a quantum superposition of all possible preceding quantum states.

• See Wheeler’s Delayed Choice Experiment.

This seemed like a good time for this slide

Thermodynamics in the way

• Any physical process that dissipates heat will be irreversible.

• It’s possible to go:

• fast, hot and irreversible, or

• slow, cool, and reversible.

• (And any shade of gray in between.)

Thermodynamics in the way

• e x t = const

• e := energy dissipation in form of heat

• t := time to operate an electric circuit

• Going slow conserves energy

Why does the arrow of time matter anyway?

Here’s why

Here’s why

A

BA NAND B

Here’s why

A

BA NAND B

Direction

Reversible gate

A

B

A NAND B

Direction

reversing bit

Reversible gate

A

B

A NAND B

Direction

reversing bit

Meandering. Not caring which way it flows.

Running analogy

Tempo

Endurance Distance

Running analogy

Speed

Powerconsumption

Battery life

How fast can we go?

How fast can we go?

• I couldn’t figure out hard data for this…

How fast can we go?

• I couldn’t figure out hard data for this…

• Theoretical upper limit: divide highest power density with lowest energy for a physical state transition; you get “operations/m3/s”.

How fast can we go?

• I couldn’t figure out hard data for this…

• Theoretical upper limit: divide highest power density with lowest energy for a physical state transition; you get “operations/m3/s”.

• Problem with:

How fast can we go?

• I couldn’t figure out hard data for this…

• Theoretical upper limit: divide highest power density with lowest energy for a physical state transition; you get “operations/m3/s”.

• Problem with:

• relativistic effects

How fast can we go?

• I couldn’t figure out hard data for this…

• Theoretical upper limit: divide highest power density with lowest energy for a physical state transition; you get “operations/m3/s”.

• Problem with:

• relativistic effects

• controlling those amounts of energy output

Where are we?

• We could go much hotter and faster.

• Insert science fiction of atomic particle machines that can run few hours of subjective human brain simulation in few femtoseconds before heat-disintegrating.

• We could also go much slower and cooler

• Ain’t nobody got time for that!

100011110001011010100100000101111100000100100010101001010100101000010111111000110111011000110101010

1001000101010010101001010000101111110001101110110001100000110011110000110001100000100110111011111110110110010111001001000001010000110010111100101101101110000111011101100101110111011101001101100111100110101000011110111111011101110100110010111011101110100101101100110011001011101110110011111101001101000111010100000110000101010001001100001111010011001011110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011100111110101100101100111000001000111100110110101001010101001111001011110010111011011001011110010111010100000100000111100001100001110001111010001100101101111110010101110110010101110110001110111100000101000101010111000101110101110111011101001110101101001101010110001011011010000110111111101111100101111001011001011100100101101100001011110011110101000001010000100100010100001011111101011011101100111011101101011011011100011110101110001011101011101110111010011101011101111011101100011100011010101100010110110100001101001110111011001111100010110000111000111101011111010100000110100011101001110100111000011101010111110111111001111100101110010111010111101111111010111101001011101100101110010110111111110001101101110110011100101110000110001110111011100001101000111000010101001100110111111000111100001111010011010011101111110111011101010000011010001110100111010011100001110101011111011111100111110010111001011101011110111111101011110100101110110010111001011011111010100001111000011100011110100011001011011011000011110111111011101110100111001011011111101100111010100000110110111000011111000101101110000111001111100101111101110011110000110000101010001011111000111000011010011110010110010111100111110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011110001110101100101100111000001000111100110110101001010101011011000011110010111100111101010000010000011100011110001111001011110000111010010110110001011101110110001111011111100100110100111011101100111101010000111101111110111011011101100101110001111101001101001110111111011101110101000001100011110110011011111110011110010110101000011110111111011101110100110010111011101110100101101101010011110011110000110010111101010000011101001100101111100011101001011111101000111010011011011101100111011100000110001111010001100001111001011100111100101111010011110110101011010100100011010110111100010101010

This is digital computation

Digital computation

• Binary inputs go in, binary output goes out

• Not all of the input precedes the output (think streams).

• Later input can be shaped by earlier output (client resubmits a cookie received from server)

Digital computation

• A particular computation (or “program”) is mapping of all possible inputs to all possible outputs.

“” -> “10001111000101101010010000010111110000010010001010100101010010100”“0” -> “0100010101100101110110110101101”“1” -> “010001010110100000010001010111110010101000011100101”“01” -> “111100010110101001000001011111000001001000101010010101001010”...

• We don’t write programs this way. Right?

• We recognize patterns and map classes of inputs to classes of outputs.

• It’s basically a compressed representation.

• If the number of patterns is not finite, you’re describing a random process.

Kolmogorov complexity

• Defined for strings.

• The length of a program that produces the given string.

• The programming language doesn’t matter

• ‘cause you can just prefix your program with an interpreter written in another language, for a constant difference.

• Can also apply to infinite sequences, such as our “enumerate all programs”

Inherent complexity of a computation

Inherent complexity of a computation

Inherent complexity of a computation

• Inherent complexity of the computation: the length of the most compact form that can encode its full input-output mapping.

Inherent complexity of a computation

• Inherent complexity of the computation: the length of the most compact form that can encode its full input-output mapping.

• Incidentally, a string is “random” if it’s shorter than any program that can generate it.

• Oh, wait, Kolmogorov complexity is a non-computable function.

• Oh, wait, Kolmogorov complexity is a non-computable function.

• Oh, wait, Kolmogorov complexity is a non-computable function.

• Consequence: no matter how good your optimizing compiler is, you can never prove that there’s not an even better one.

• Oh, wait, Kolmogorov complexity is a non-computable function.

• Consequence: no matter how good your optimizing compiler is, you can never prove that there’s not an even better one.

• a.k.a “Full Employment Theorem” for compiler writers.

“” -> 1001000101010010101001010000101111110001101110110001100000110011110000110001100000100110111011111110110110010111001001000001010000110010111100101101101110000111011101100101110111011101001101100111100110101000011110111111011101110100110010111011101110100101101100110011001011101110110011111101001101000111010100000110000101010001001100001111010011001011110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011100111110101100101100111000001000111100110110101001010101001111001011110010111011011001011110010111010100000100000111100001100001110001111010001100101101111110010101110110010101110110001110111100000101000101010111000101110101110111011101001110101101001101010110001011011010000110111111101111100101111001011001011100100101101100001011110011110101000001010000100100010100001011111101011011101100111011101101011011011100011110101110001011101011101110111010011101011101111011101100011100011010101100010110110100001101001110111011001111100010110000111000111101011111010100000110100011101001110100111000011101010111110111111001111100101110010111010111101111111010111101001011101100101110010110111111110001101101110110011100101110000110001110111011100001101000111000010101001100110111111000111100001111010011010011101111110111011101010000011010001110100111010011100001110101011111011111100111110010111001011101011110111111101011110100101110110010111001011011111010100001111000011100011110100011001011011011000011110111111011101110100111001011011111101100111010100000110110111000011111000101101110000111001111100101111101110011110000110000101010001011111000111000011010011110010110010111100111110101000001010100110100011101011011001000001100001100101000001001101110000111110011000001100101100001100011100111000001100001110011110101100011110001110101100101100111000001000111100110110101001010101011011000011110010111100111101010000010000011100011110001111001011110000111010010110110001011101110110001111011111100100110100111011101100111101010000111101111110111011011101100101110001111101001101001110111111011101110101000001100011110110011011111110011110010110101000011110111111011101110100110010111011101110100101101101010011110011110000110010111101010000011101001100101111100011101001011111101000111010011011011101100111011100000110001111010001100001111001011100111100101111010011110110101011010100100011010110111100010101010

You can’t cheat

complexity(input) + complexity(decoder)

• Sometimes, part of the decoder is external.

• Sometimes, part of the decoder is external.

• It can be in your head.

• Sometimes, part of the decoder is external.

• It can be in your head.

• Again, human is part of thecomputation.

Complex look is sometimes a simple pattern

Complex look is sometimes a simple pattern

Complex look is sometimes a simple pattern

SzegediButterfly1 {init:  z = #pixelloop:  float x = real(z)  float y = imag(z)  z = sqr(y) - sqrt(abs(x)) +            (sqr(x) - sqrt(abs(y))) * 1i + #pixelbailout:  |z| <= @Bailoutdefault:  param Bailout    caption = "Bailout Value"    default = 127.0  endparam}

Complex look is sometimes a simple pattern

Complex look is sometimes a simple pattern

Complex look is sometimes a simple pattern

SzegediButterfly2 {init:  z = #pixelloop:  float x = real(z)  float y = imag(z)  z = sqr(x) - sqrt(abs(y)) +           (sqr(y) - sqrt(abs(x))) * 1i + #pixelbailout:  |z| <= @Bailoutdefault:  param Bailout    caption = "Bailout Value"    default = 127.0  endparam}

Even if you could find the most compact

form…• … you probably wouldn’t want to use it.

• It would be even less humanly maintainable than flipping bits in a zipped source file.

• Ever seen 256-byte 8-bit CPU demos?

You can always reduce complexity

• You always have excess complexity in your system.

• It’s worth thinking of how to reduce it.

• Pattern recognition (refactoring).

• Favor environments that don’t force you into artificial layer boundaries. Code is data, data is code.

• Patterns can be more overarching without artificial boundaries.

Artificial boundaries hobble you

• Favor environments that don’t force you into artificial layer boundaries. Code is data, data is code.

• Patterns can be more overarching without artificial boundaries.

• Java program loads an XML file containing a flow graph and some embedded JavaScript?

• You have a problem.

Take aways

• Be sensible with the effort and the result.

• Don’t overshoot into compact-but-unreadable territory.

• Hitting on the minimally complex representation is as likely as inventing the perpetuum mobile. (Thermodynamics metaphor again.)

Image CreditsSnowy branches: http://www.flickr.com/photos/aeioux/2398264997/XKCD comic: http://xkcd.com/297/Complex gizmo: http://www.flickr.com/photos/michaelheiss/2871996129/Electric heater: http://www.flickr.com/photos/jocelynb/426268348/Layer cake: http://www.flickr.com/photos/julessilver/3259734572Abacus: http://www.flickr.com/photos/skidder/37675092Airplanes: http://www.flickr.com/photos/good_day/198611998/Meandering: http://www.flickr.com/photos/31856336@N03/7831522814/Headphones: http://www.flickr.com/photos/doremigirl/8362322435Kolmogorov Lecture: http://en.wikipedia.org/wiki/File:Kolm_complexity_lect.jpg

All used images are Creative Commons licensed, and used according to their terms of license.