The Digital DelugeLecture 1

Learning in RetirementDavid Coll

Professor EmeritusDepartment of Systems and Computer

Engineering

Winter 2009

Welcome to “The Digital Deluge”

• My name is David Coll and

• In the next few weeks I’m going to try and give you a feel for what’s going on in a world awash in digital information.

• We’ll look at – the jargon, – the sources, – the volumes, – the technology, – the societal impact, and – how some are coping.

• This is a lecture series about the flood of information created by new technologies.

• The problem in offering a course about this subject is that there is way too much information to handle.

• But – we have ways of coping• For example

• Clear and simple instructions …

First - My Co-ordinates

• Professor Emeritusin the Department of Systems and Computer Engineering, Carleton University

• dccinfo@sympatico.ca• http://www.sce.carleton.ca/faculty/coll.htm• 613-225-4229

Information

• The world is awash in INFORMATION.• Our senses: sight, hearing, smell, touch, and

taste are bombarded with inputs from all sides

• Information is essential for our survival.• But …• Let’s narrow it down somewhat

• We might limited ourselves to information that comes to us through a medium: mediated communications – Print

• newspapers, magazines, books, flyers …– Posters– Radio– Television– Recordings – music, images, videos– The Telephone

• As well as through • the Internet and • the World Wide Web

• We will exclude information We will exclude information communicated to us through spectral communicated to us through spectral channels by mediums wearing star-channels by mediums wearing star-studded gowns and too many studded gowns and too many bracelets …bracelets …

Introduction

• The rapid growth and deployment of digital information processing and communications technology is changing our world, the way we do business, and the way we live.

• Information be it text, data, images, music, speech, or television is acquired, stored, processed, communicated, received, and presented in digital form, i.e., as numbers.

• The resulting torrent of information is so

vast that some have dubbed it the “The Digital Deluge”.

• del·uge – 1

• a: an overflowing of the land by water • b: a drenching rain

– 2: an overwhelming amount or number, as in:received a deluge of offers

The Deluge from Various Perspectives

The extent, rapidity of acceptance, and pervasiveness of digital technology is almost (?) beyond comprehension.

The Digital Deluge

• “Data is omnipresent. Everywhere we go, we encounter it, in our business transactions, subscription lists, email, etc.”

• “To put the growth of data in perspective, every 18 months the processing capacity of the world doubles, but at the same time data has been doubling every nine months.”

• http://itmanagement.earthweb.com/datbus/article.php/1495861

The Digital Deluge

• “One of the scariest statistics in recent memory is IDC’s prediction that the amount of digital data between now and 2010, reaching a whopping 988 billion gigabytes.”

• “(That’s a figure so huge, it would be helpful to have one of those goofy yet helpful references to put it into context. How many football fields it would fill, for example.)”

• http://www.itbusinessedge.com/blogs/tve/?p=98

ScienceDaily (Nov. 8, 2007)

• “Most people have a few gigabytes of files on their PC.

• In the next decade, astronomers expect to be processing 10 million gigabytes of data every hour from the Square Kilometre Array telescope”.

• http://www/sciencedaily.com/releases/2007/11/071107091524.htm

• “When the Large Hadron Collider [at CERN] starts running in 2008, it will be the biggest scientific instrument in the world.

• Thousands of scientists across the planet will be clamouring for access to the streams of data that will come out of the instrument.

• The LHC will produce about 15 Petabytes of data every year, which is more than 1000 [times] the amount of information in book form printed every year around the world”.

• If written to CDs, the stack of CDs would be about 20 km in height!

• http://gridcafe.web.cern.ch/gridcafe/GridatCERN/gridatcern.html

From IDC Report

• “With a compound annual growth rate of almost 60%, the digital universe is growing faster and is projected to be nearly 1.8 zettabytes (1,800 exabytes) in 2011, a 10-fold increase over the next five years

• “At 281 billion gigabytes (281 exabytes), the digital universe in 2007 was 10% bigger than originally estimated

• IDC’s new research shows the digital universe is growing more rapidly than original estimates as a result of accelerated growth in worldwide shipments of– digital cameras, – digital surveillance cameras, and – digital televisions

• as well as – a better understanding of information

replication trends.

• They didn’t mention traffic to and from cell phones , iPhones, Blackberries and the ilk..

• The digital universe in 2007 was equal to almost 45 gigabytes (GB) of digital information for every person on earth – or the equivalent of over 17 billion 8 GB iPhones.

• Other fast-growing corners of the digital universe include those related to – Internet access in emerging countries– sensor-based applications– data centers supporting “cloud computing”– social networks comprised of digital content

created by many millions of online users.

The Digital Deluge

• What’s it all about and what can we do about it?

• Maybe we can not only survive the Digital Deluge, but use it to our advantage.

• Hey! Didn’t Noah’s kids learn to water ski behind the Arc?

Let’s Start with Five Questions

• Where does information come from?• How does it get to be “digital”?• How does it get to us?• Why is it a problem?• What is being done to cope with it?

Where Does the Information Come From?

Information Generators• Arts• Literature• History• Language• Education• Religion• Science and Technology• Government/Bureaucracy• Commerce

• Administration• Medicine and Pharmacology• Geography and Cartography• Entertainment• News• …• …• …

Traditional Sources

• Media– Print: Books, Newspapers– Film– Radio and TV Broadcast

• Accumulated Knowledge– Libraries– Repositories– Folk Lore

• Publications– Books, Newspapers, Magazines,

Professional Journals

• Radio and TV• Personal Communications and

Correspondence• Sensors and Instrumentation

• .

• Libraries and Repositories– Publications, Reference Material, Dictionaries,

Encyclopedia, Catalogs

• Personal Creation– Records

• Bills, Accounts, Investments, Wills, Warranties, Contracts, …

– Libraries • Books, Magazines, Clippings, …

– Diaries and Writings– Recordings

• Audio, Photo, Film, Video• Computer Files

How Does It Get To Us?• Purchase and Own• Delivery Services

– Mail, Newspapers, Magazines• Broadcast Services

– Radio– Television

• Message Services– Telegraph– Teletype

• Data Communications– File Transfers, Data Collection– Transactions

Where did this “Deluge” come from?

• The major reason the world is awash in information in digital form is the

COMPUTER• Actually the ability to represent and process

information on digital form

• But, that is only half the story.

• An isolated computer is just a fancy adding machine or typewriter with memory

• So, the deluge results from the fact that all the world’s computers are connected!– Well, most all.

• The ability to communicate information is an essential factor behind the deluge, and

• Communications is done “digitally”..

What’s the Problem?

• There is so much information/data*.– Entertainment, medical, scientific research,

educational, literature, news, personal …

• We are not always sure about just what “information” there is in the data

*“information” often implies “knowledge” whereas we are talking about the volumes of data that are generated – whether they make sense or not.

The Culprit?

• A mixture of – The Internet

• networked computers, ubiquitous communications,

– Applications• the WWW, information processing

– and Smart Machines

• Enabled by the evolution of common digital technologies throughout.

So What’s New?

• A common technological basis for the– acquisition, – distribution and – handling

• of information in digital formats.• A basis that has very recently reached

levels of maturity that have unleashed information in a flood of incomprehensible immensity.

• That’s the Punch Line• But Why The Deluge Now?

• Because we can!

ELECTRONICS COMMUNICATIONS

DIGITAL TECHNOLOGY

HARDWARE SOFTWARE

Enabling Technologies

• But, first

• Let us pause for a short break ….

What will we do?

• We’ll look at sources of the information explosion in areas that affect us:– Mobile telephones– Digital cameras – Music: iPods, CDs, DVDs– HDTV – The Internet and Web Applications – Libraries and other repositories – Smart terminals– Home automation – Scientific Research, and so on.

• We will look at what “digital” means– How the adoption of digital technology is used to

create information, store and process it, communicate and display it.

• Why we are deluged with digital information, i.e., – What’s behind the sudden onslaught of

information, where this surge is coming from, how it’s to be handled, and what the future holds.

• Then, hopefully we can discuss some ways of coping, and even enjoying the deluge –

• Let’s start with some definitions

Terminology

• Misuse of terminology can lead to lead to misunderstanding.

• Being precise in one's terminology can be important.

• It is also important to understand the historical origin of terms, to better understand their meaning, and to realize how different points of view can modify meaning.

Scale and Scope of the Deluge

• How deep is the flood?• If we are in water that is 6 feet deep we

have a feel for what sort of situation we are facing

• But, what’s an exafoot?• What are these funny names for data rates

and volumes?

• The basic digital (binary) unit is the bit, a single binary digit with values 0 or 1.

• In Digital Communications, one usually refers to data rates in terms of bits per second, or bps.

• Data rates, often mistakenly termed bandwidth, are expressed as the number of bits transmitted per second– as in 56 kbits per second (kbps) or 1.544

megabits per sec (mbps)

Common Data Rates

• Data Rate speeds derive from teletype and telephone services

• Originally, everything was a multiple of the telegraph speed of 75 bits per second– 300, 1200, 2400, 4800, 9600 bps

• Then multiples of the basic telephone sampling speed of 64,000 kbps took over, giving services at the T1 TDM carrier speed of 1.544 Mbps, and multiples of it.

Common Optical FibreTransmission Rates – in Megabytes per second.

OpticalOptical Line RatesLine Rates

OC-1OC-1 51.8451.84

OC-3OC-3 155.52155.52

OC-12OC-12 622.08622.08

OC-48OC-48 2,488.322,488.32

OC-192OC-192 9,953.289,953.28

OC-768OC-768 39,813.1239,813.12

Optical Fibre Transmission Rates- in Megabytes per second- MBps

• The OC192 rate is 9,953.28 megabits per second, or almost 10,000 Mbps = 10 gigabits per second.

• This number is 192 times 51.84 mbps• 51.84 = 810 x 8,000 x 8, or 810 x 64,000• The number of bits per second in a basic

SONET frame used for the transmission of 810 voice channels.

Prefix Symbol Base 10 Base 2

yotta Y +24 +80

zetta Z +21 +70

exa E +18 +60

peta P +15 +50

tera T +12 +40

giga G +09 +30

mega M +06 +20

kilo k +03 +10

hecto h +02 -

deka da +01 -

deci d -01 -

centi c -02 -

milli m -03 -

micro -06 -

nano n -09 -

pico p -12 -

femto f -15 -

atto -18 -

zepto z -21 -

yocto y -24 -

In communications, electronics, and physics, multipliers are defined in powers of 10 from 10-24 to 1024, proceeding in increments of three orders of magnitude (103 or 1,000).

In IT and data storage, multipliers are defined in powers of 2 from 210 to 280, proceeding in increments of ten orders of magnitude (210 or 1,024).

But, how many is that in base 10?*

• Kilo = 1 thousand = 103

• Mega = 1 million = 106

• Giga = 1 billion = 109

• Tera = 1 trillion = 1012

• Exa = 1 quadrillion = 1015

• Peta = 1 quintillion = 1018 • Zetta = 1 sextillion = 1021

* By the “US short scale”: check http://en.wikipedia.org/wiki/Names_of_large_numbers

Computer Memory and Processing

• In digital computers, the basic element of information storage is the byte, an 8-bit binary number.

• The basic storage element is a word, and is usually 8, 16, 32 or 64 bits long, or 1, 2, 4, or 8 bytes.

• It is common to measure the size of a computer memory in terms of powers of 2,

• e.g.• 1 Kilobyte = 210 = 1024 bytes.

Counting Bytes

• Kilobyte: 1 KB = 210 = 1024 bits ~ 1 x 103

• Megabyte: 1MB = 220 bits, or 1,048,576 bits ~ 1 x 106

• Gigabyte: 1GB = 230 bits = 1,073,741,824 bits• Terabyte: 1TB = 240 bits = 1,099,511,627,776

bits• Exabyte: 1 EB =250 bits =

1,125,899,906,842,624 bits ~ 1 x 1015

• Petabyte 1 PB = 260 bits = 1,152,921,504,606,846,976 bits ~ 1 x 1018

Bits and Bytes

• Transmission speeds invariably refer to bits per second

• e.g. 1.544 Mbps

• File sizes refer to bytes • e.g. 1 Megabyte of RAM = 1024 Kilobytes

or 1,048,576 bits; but communications rarely, if ever, refers to bits at all.

The “Digital” Thing: What’s the Big Idea, Anyway?

• ““DigitalDigital” ” is a word (mis)used to describe is a word (mis)used to describe information (often called information (often called datadata) in ) in discretediscrete formform, and the subsequent processing of that , and the subsequent processing of that information, information,

as opposed to information that occurs in a as opposed to information that occurs in a continuouscontinuous, or “, or “analoganalog” form.” form.