First Courses WorkshopDay 2
Mark Guzdial College of Computing
Georgia Institute of [email protected]
http://www.cc.gatech.edu/~mark.guzdialhttp://www.georgiacomputes.org
Workshop Plan-Day 2
9-10:00 am: Explanation of Media Computation class structure.
10:00-10:15: Break 10:15-12:00: Movies in Media Computation:
Implementing video special effects 12:00-1:00: Lunch 1-2:30: Tackling a homework assignment in Media
Computation. Making a movie. 2:30-2:45: Break 2:45-4:15: Introducing computing through robotics in
Python 4:15-5:30: Tackling a homework assignment in robotics.
Follow the light.
Introduction to Media Computation in Python
Python as the programming language Huge issue
Use in commercial contexts authenticates the choice IL&M, Google, Nextel, etc.
Minimal syntax
Looks like other programming languages Potential for transfer
Rough overview of Syllabus
Defining and executing functions Pictures
Psychophysics, data structures, defining functions, for loops, if conditionals
Bitmap vs. vector notations Sounds
Psychophysics, data structures, defining functions, for loops, if conditionals
Sampled sounds vs. synthesized, MP3 vs. MIDI Text
Converting between media, generating HTML, database, and networking
A little trees (directories) and hash tables (database) Movies Then, Computer Science topics (last 1/3 class)
def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor)
def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0)
def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance))
Examples:
Some Computer Science Topics inter-mixed We talk about algorithms across
media Sampling a picture (to scale it) is
the samealgorithm as sampling a sound (to shift frequency)
Blending two pictures (fading one into the other) and two sounds is the same algorithm.
We talk about representations and mappings (Goedel) From samples to numbers (and into
Excel), through a mapping to pixel colors
We talk about design and debugging But they mostly don’t hear us
Computer Science Topicsas solutions to their problems
“Why is PhotoShop so much faster?” Compiling vs. interpreting Machine language and how the computer works
“Writing programs is hard! Are there ways to make it easier? Or at least shorter?” Object-oriented programming Functional programming and recursion
“Movie-manipulating programs take a long time to execute. Why? How fast/slow can programs be?” Algorithmic complexity
Introduction to Java Math operators, printing results, data types, casting,
relational operators, Strings, variables
Introduction to Programming Creating and naming objects
Using a turtle and a world
Creating new Turtle methods Draw simple shapes Using parameters
Syllabus
Syllabus - Continued
Modifying Pictures using Loops One-dimensional
arrays Use for-each, while,
and for loops to Increase/decrease
colors, fake a sunset, lighten and darken, create a negative, and grayscale
Introduction to Media Computation in Java
Syllabus - Continued
Modifying Pixels in a Matrix Two-dimensional
arrays Nested loops Copying, mirroring,
blending, rotation, scaling
Syllabus - Continued
Conditionally Modifying Pixels Boolean expressions Using && and || Replacing a color,
reducing red-eye, edge detection, sepia-toned, posterize, highlight extremes, blurring, background subtractions, chromakey
Syllabus - Continued
Drawing on Pictures Using existing Java
classes Inheritance Interfaces Drawing simple
shapes, drawing text, general copy, general scale, shearing, gradient paint, general blending, clipping
Syllabus - Continued
Modifying all Samples in a Sound 1D arrays Loops Conditional execution Change volume,
normalizing a sound (make it as loud as possible), force to extremes
Syllabus - Continued
Modifying Samples using Ranges Loops Clipping, splicing,
reversing, mirroring
Syllabus - Continued
Combining and Creating Sounds Class and private
methods Composing sounds,
blending sounds, changing frequencies, and creating echoes
Creating sounds Sine Waves, Square
Waves, Triangle Waves MP3 and MIDI
Syllabus - Continued
Creating Classes Identifying objects and classes Defining a class Overloading constructors Creating and initializing an array Creating getters and setters Creating a main method Javadoc comments
Reusing a class via inheritance ConfusedTurtle
ConfusedTurtleTurtle
Syllabus - Continued
Creating and Modifying Text String methods Reading from and writing to
files Handling Exceptions
Creating a form letter Modifying programs Getting text from networks Creating random sentences Using text to shift between
media
Syllabus - Continued
Making Text for the Web Throwing exceptions,
“unnamed” package, HashMap, Generics, and Iterators
Generating HTML Create a web page from a
directory Create a web page from
other web pages Databases Creating a web page from
a database
Syllabus - Continued
Encoding, Manipulating, and Creating Movies Frame-based animations
with simple shapes and text
Special effects – fade out, fake sunset, and chromakey
Syllabus - Continued
Speed What makes programs
fast? Compilers and Interpreters Writing a graphics
interpreter and compiler Searching Algorithms that can’t be
written What makes computers
fast? Clock rates, Storage, Display
Syllabus - Continued
Javascript Syntax User Interfaces Multimedia
Modules and Movies
Using external features in Python Making movies
Implementing digital video special effects
Processing many files at once
Sometimes you want to process a bunch of files without knowing the name of every one. Examples: Putting a title on every picture in a directory. Creating an index page that references every picture
and sound in a directory. Processing every frame in a video, where each frame
is stored as a JPEG frame.
Adding new capabilities: Modules
What we need to do is to add capabilities to Python that we haven’t seen so far.
We do this by importing external modules. A module is a file with a bunch of additional functions and
objects defined within it. Some kind of module capability exists in virtually every
programming language.
By importing the module, we make the module’s capabilities available to our program. Literally, we are evaluating the module, as if we’d typed them into
our file.
Python’s Standard Library
Python has an extensive library of modules that come with it.
The Python standard library includes modules that allow us to access the Internet, deal with time, generate random numbers, and…access files in a directory.
Accessing pieces of a module
We access the additional capabilities of a module using dot notation, after we import the module.
How do you know what pieces are there? Check the documentation. Python comes with a Library Guide. There are books like Python Standard Library that
describe the modules and provide examples.
The OS Module
The OS module offers a number of powerful capabilities for dealing with files, e.g., renaming files, finding out when a file was last modified, and so on.
We start accessing the OS module by typing: import os
The function that knows about directories is listdir(), used as os.listdir() listdir takes a path to a directory as input.
Using os.listdir
>>> import os>>> print getMediaPath("barbara.jpg")C:\Documents and Settings\Mark Guzdial\My Documents\
mediasources\barbara.jpg>>> print getMediaPath("pics")Note: There is no file at C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\picsC:\Documents and Settings\Mark Guzdial\My Documents\
mediasources\pics>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg',
'students7.jpg', 'students8.jpg']
Writing a program to title pictures
We’ll input a directory We’ll use os.listdir() to get each filename in the
directory We’ll open the file as a picture. We’ll title it. We’ll save it out as “titled-” and the filename.
Titling Pictures
import os
def titleDirectory(dir):
for file in os.listdir(dir):
picture = makePicture(file)
addText(picture,10,10,"This is from CS1315 Spring 2003")
writePictureTo(picture,"titled-"+file)
Okay, that didn’t work
>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")
makePicture(filename): There is no file at students1.jpg
An error occurred attempting to pass an argument to a function.
Why not?
Is there a file where we tried to open the picture? Actually, no. Look at the output of os.listdir() again
>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")
['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg']
The strings in the list are just the base names No paths
Creating paths
If the directory string is in the placeholder variable dir, then dir+file is the full pathname, right?
Close—you still need a path delimiter, like “/” But it’s different for each platform! Python gives us a notation that works: “//” is as a path
delimiter for any platform. So: dir+”//”+file
A Working Titling Program
import os
def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)
Showing it work>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")Processing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students1.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students2.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students5.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students6.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students7.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students8.jpg>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg', 'titled-
students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']
Inserting a copyright on pictures
What if you want to make sure you’ve got JPEG files?import os
def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file if file.endswith(".jpg"): picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)
Say, if thumbs.db is there
>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")Processing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students1.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students2.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students5.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students6.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students7.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//students8.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics//Thumbs.db>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg',
'Thumbs.db', 'titled-students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']
Another interesting module: Random
>>> import random>>> for i in range(1,10):... print random.random()... 0.82113693141939280.63542667797032460.94600601635201590.9046156965596840.335004644632541870.081249821269405940.07114813768070150.72552173073460480.2920541211845866
Randomly choosing words from a list
>>> for i in range(1,5):
... print random.choice(["Here", "is", "a", "list", "of", "words", "in","random","order"])
...
list
a
Here
list
Randomly generating language
Given a list of nouns,verbs that agree in tense and number,and object phrases that all match the verb,
We can randomly take one from each to make sentences.
Random sentence generator
import random
def sentence(): nouns = ["Mark", "Adam", "Angela", "Larry", "Jose", "Matt", "Jim"] verbs = ["runs", "skips", "sings", "leaps", "jumps", "climbs", "argues",
"giggles"] phrases = ["in a tree", "over a log", "very loudly", "around the bush",
"while reading the Technique"] phrases = phrases + ["very badly", "while skipping","instead of
grading", "while typing on the CoWeb."] print random.choice(nouns), random.choice(verbs),
random.choice(phrases)
Running the sentence generator
>>> sentence()Jose leaps while reading the Technique>>> sentence()Jim skips while typing on the CoWeb.>>> sentence()Matt sings very loudly>>> sentence()Adam sings in a tree>>> sentence()Adam sings around the bush>>> sentence()Angela runs while typing on the CoWeb.>>> sentence()Angela sings around the bush>>> sentence()Jose runs very badly
How much smarter can we make this?
Can we have different kinds of lists so that, depending on the noun selected, picks the right verb list to get a match in tense and number?
How about reading input from the user, picking out key words, then generating an “appropriate response”?
if input.find(“mother”) <> -1: print “Tell me more about your mother…”
Joseph Weizenbaum’s “Eliza”
Created a program that acted like a Rogerian therapist. Echoing back to the user whatever they said, as a
question. It had rules that triggered on key words in the user’s
statements. It had a little memory of what it had said before.
People really believed it was a real therapist! Convinced Weizenbaum of the dangers of computing.
Session with the “Doctor”
>>>My mother bothers me.
Tell me something about your family.
>>>My father was a caterpillar.
You seem to dwell on your family.
>>>My job isn't good either.
Is it because of your plans that you say your job is not good either?
Digital Video Effect: Background subtraction Let’s say that you have a picture of someone,
and a picture of the same place (same background) without the someone there, could you subtract out the background and leave the picture of the person?
Maybe even change the background? Let’s take that as our problem!
Person (Katie) and Background
The core of the algorithm:
if distance(personColor,bgColor) > someValue:
bgcolor = getColor(getPixel(newBg,x,y))
setColor(getPixel(person,x,y), bgcolor)
What else do we need? We need to get all these variables set up
We need to input a person picture, a background (background without person), and a new background.
We need a loop where x and y are the right values We have to figure out personColor and bgColor
Simplifying a little, and specifying a little
def swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) > 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)
Trying it with a jungle background
What happened?
It looks like we reversed the swap If the distance is great, we want to KEEP the pixel. If the distance is small (it’s basically the same thing),
we want to get the NEW pixel.
Reversing the swap
def swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)
Better!
But why isn’t it alot better?
We’ve got places where we got pixels swapped that we didn’t want to swap See Katie’s shirt stripes
We’ve got places where we want pixels swapped, but didn’t get them swapped See where Katie made a
shadow
How could we make it better?
What could we change in the program? We could change the threshold “someValue” If we increase it, we get fewer pixels matching
That won’t help with the shadow If we decrease it, we get more pixels matching
That won’t help with the stripe
What could we change in the pictures? Take them in better light, less shadow Make sure that the person isn’t wearing clothes near
the background colors.
Another way: Chromakey Have a background of a known
color Some color that won’t be on
the person you want to mask out
Pure green or pure blue is most often used
I used my son’s blue bedsheet This is how the weather people
seem to be in front of a map—they’re actually in front of a blue sheet.
Chromakey recipe
def chromakey(source,bg): # source should have something in front of blue, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))
Can also do this with getPixels()
def chromakey2(source,bg): # source should have something in front of blue, bg is the new background for p in getPixels(source): # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))
Example results
Just trying the obvious thing for Red
def chromakey2(source,bg): # source should have something in front of red, bg is the new background for p in getPixels(source): if getRed(p) > (getGreen(p) + getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))
Doesn’t always work as you expect
Let’s try that with green
def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness + blueness if getGreen(p) > getBlue(p) + getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))
The same definition of green doesn’t work
What happened?
The towel isn’t just green The green of the towel has lots of blue and red in it.
Use MediaTools to figure out a new rule that makes sense.
Tweaking Chromakey
def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness AND blueness if getGreen(p) > getBlue(p) and getGreen(p) > getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))
That looks better
Movies, animations, and video…oh my! We’re going to refer generically to captured
(recorded) motion as “movies.” This includes motion entirely generated by graphical
drawings, which are normally called animations. This also includes motion generated by some kind of
photographic process, normally called video.
Psychophysics of Movies:Persistence of Vision What makes movies work is yet another limitation of our
visual system: Persistence of vision
We do not see every change that happens in the world around us.
Instead, our eye retains an image (i.e., tells the brain “This is the latest! Yup, this is still the latest!”) for a brief period of time. If this were not the case, you would be aware of every time that
your eye blinks because the world would “go away” for a moment.
16 frames and it’s motion
If you see 16 separate pictures in one second, and these pictures are logically sequenced, That is, #2 could logically follow from the scene in #1. 16 pictures of completely different things doesn’t work,
You will perceive the pictures as being in motion. 16 frames per second (fps), 16 pictures in a
second, is the lower bound for the sensation of motion.
Beyond 16 fps
Early silent pictures were 16 fps. Motion picture standards shifted to 24 fps to make sound
smoother. Videocameras (digital video) captures 30 fps How high can we go?
Air force experiments suggest that pilots can recognize a blurb of light in 1/200th of a second!
Video game players say that they can discern a difference between 30 fps and 60 fps.
Bottomlines: Generate at least 16 fps and you provide a sense of motion. If you want to process video, you’re going to have 30 fps to
process (unless it’s been modified elsewhere for you.)
Processing movies
Our frames are going to be JPEG pictures. One JPEG file per frame.
So, if we’re going to be processing movies, we’re going to generating or processing sequences of JPEG files.
Using MediaTools
To generate a series of frame pictures in a folder from an MPEG file.
To play a folder of frame pictures and to save it as a JMV file. (JPEG Movie format.)
To play JMV or MPEG movies.
MPEG? QuickTime? AVI? JMV?
MPEG, QuickTime, and AVI are compressed movie formats. They don’t record every frame. Rather, they record some key frames, and then store data about
what parts of the screen change on intervening frames. MPEG is an international standard, from the same people who
invented JPEG. AVI is a Microsoft standard. QuickTime is an Apple standard.
JMV is a file consisting of JPEG frames in an array. All frames represented
Generating other kinds of movies
Other tools can generate other kinds of movie formats.
QuickTime Pro (http://www.apple.com/quicktime) can read a sequence of JPEG images and produce MPEG, AVI, or QuickTime movies.
ImageMagick (open source toolkit) can also read a sequence of JPEG images and produce MPEG movies.
Why do we compress movies?
Do the math: One second of 640x480 pixels at 30 fps 30 (frames) * 640 * 480 (pixels) = 9,216,000 pixels With 3 bytes of color per pixel, that’s 27,648,000
bytes or 27 megabytes of information per second. For a 90 minute feature movie (short), that’s 90 * 60 *
27,648,000 = 149,299,200,000 bytes (149 gigabytes)
A DVD stores 6.47 gigabytes of data. So even on a DVD, the movie is compressed.
MPEG movie = MPEG frames plus MP3 soundtrack An MPEG movie is actually a series of MPEG
frames composed with an MP3 soundtrack. It’s literally two files stuck together in one.
We’re not going to deal with sound movies for now. The real challenge in doing movie processing is
generating and manipulating frames.
Get the frames in order
Many tools (including os.listdir()) can process frames in order if the order is specified.
We specify the order by encoding the number of the frame into the name. If you put in leading zeroes so that everything is the
same length, the order is alphabetical as well as numerical.
Simple Motiondef movingRectangle(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")
A Few Frames
frame00.jpg frame02.jpg frame50.jpg
The trick here is all mathematics
if frame < 50: #Less than 50, move down
# Generate new positions each frame number
addRectFilled(canvas,frame*10,frame*5, 50,50,red)
if frame >= 50: #Greater than 50, move up
addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red)
When frame = 1, addRectFilled(canvas,10,5,50,50,red)
When frame = 2, addRectFilled(canvas,20,10,50,50,red)
When frame = 49, addRectFilled(canvas,490,285,50,50,r
ed) When frame = 50,
50-(50-50) = 50 addRectFilled(canvas,500,250,50,50,r
ed) When frame = 51,
50-(51-50)=50-1=49 addRectFilled(canvas,490,285,50,50,r
ed) When frame = 99,
50-(99-50)=50-49=1 addRectFilled(canvas,10,5,50,50,red)
Important cool thing: You can draw past the end of the picture!
addText, addRect, and the rest of the drawing tools will work even if you go beyond the edge of the drawing. Drawings will clip what can’t be seen in them, so you
don’t get an array out of bounds error. This is a big deal, because it means that you don’t
have to do complicated math to see when you’re past the end of the drawing.
But only for the drawing functions. If you set pixels, you’re still on your own to stay in range.
Making a tickertapedef tickertape(directory,string): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) #Start at right, and move left addText(canvas,600-(frame*10),100,string) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")
tickertape(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\tickertape","Welcome to CS1315!")
Can we move more than one thing at once? Sure!
def movingRectangle2(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Let's have one just moving around addRectFilled(canvas,100+ int(10 * sin(frame)),4*frame+int(10* cos(frame)),50,50,blue) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")
addRectFilled(canvas,100+ int(10 * sin(frame)), 4*frame+int(10* cos(frame)),50,50,blue)
What’s going on here? Remember that both sine and cosine vary between
+1 and -1. Int(10*sin(frame)) will vary between -10 and +10 With cosine controlling y and sine controlling x, should
create circular motion frame=1
x is 108, y is 9
frame=2 x is 109, y is 4
Frames from two motions at once
Moving something else:Remember this?
def copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas
To move Barb’s face around, we have to do this for each frame, moving the target each time.
Note: This will take a while
On my fastest computer, it takes over a minute. Think about it: Why? We’ll get to it next week.
You might to know where you are. Print isn’t too useful here.
Print doesn’t print until the program is done. There is a JES function called printNow() that takes a string and
will print it immediately. That way, we can see what frame number we’re at.
Most important thing to know: DID WE GET PAST FRAME ONE? (Very important to know as code gets slower!)
Moving Barb’s head
def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")
moveahead(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbshead")
My, isn’t that gorey!
Can’t we make it easier to read? Can we just deal with the parts that we care
about? Maybe we could use sub-functions?
At least for the writing out of the frame.
Using subfunctions
def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color =
getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,canvas)
def writeFrame(num,directory,framepict): # Have to deal with single digit vs.
double digit frame numbers differently framenum=str(num) if num < 10:
writePictureTo(framepict,directory+"//frame0"+framenum+".jpg")
if num >= 10:
writePictureTo(framepict,directory+"//frame"+framenum+".jpg")
What if we have over 100 frames?
def writeFrame(num,directory,framepict): # Have to deal with single digit vs. double digit frame numbers
differently framenum=str(num) if num < 10: writePictureTo(framepict,directory+"//frame00"+framenum+".jpg") if num >= 10 and num<100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg") if num >= 100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg")
This will work with moveahead() and other functions—it’s generally useful
Using real photographs
Of course, we can use any real photographs we want.
We can use any of the techniques we’ve learned previously for manipulating the photographs.
Even more, we can use the techniques in new ways to explore a range of effects.
Slowly making it (very) sunset
Remember this code?def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7) What if we applied this to create frames of a movie, but
slowly increased the sunset effect?
SlowSunset
def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the
loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas)
def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99)
Not showing you writeFrame() because you know how that works.
Just one canvas repeatedly being manipulated
SlowSunset frames
Fading by background subtraction
Remember background subtraction?def swapbg(person, bg, newbg,threshold): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < threshold: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)
One change here is that the threshold is now an input.
Use the frame number as the threshold
def slowfadeout(directory): bg = makePicture(getMediaPath("wall.jpg")) jungle = makePicture(getMediaPath("jungle2.jpg")) for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("wall-two-people.jpg"))
#outside the loop! printNow("Frame number: "+str(frame)) swapbg(canvas,bg,jungle,frame) # Now, write out the frame writeFrame(frame,directory,canvas)
SlowFadeout
Dealing with real video
We really can’t deal with real video. Dealing with each frame takes a lot of processing. If you were going to process each frame as fast as it
was coming in (or going out), you’d have 1/30th of a second to process each frame!
We cheat by Saving each frame as a JPEG image Processing the JPEG images Convert the frames back to a movie
The original kid-in-bg-seq movie
Let’s have Mommy “watching”
We’ll paste Barb’s head into each frame. We’ll use os.listdir to process all the frames of
the kid sequence.
MommyWatchingdef mommywatching(directory): kiddir=r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\kid-in-bg-seq" barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) frame = 0 for file in os.listdir(kiddir): if file.endswith(".jpg"): frame = frame + 1 printNow("Frame number: "+str(frame)) framepic = makePicture(kiddir+"//"+file) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3, frame*3+((200-45)/2) ): sourceY = 25 for targetY in range(frame*3, frame*3+((200-25)/2) ): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(framepic,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,framepic)
We process each frame, and copy Mommy’s head to the frame, just like we animated in a line before onto a blank canvas.
MommyWatching
Lightening a picture
I took some video of a puppet show in black light.
Very hard to see the puppets. Your eye can pick them up, but the camera can’t. Recall earlier discussion: Your eye can detect
luminance changes that no media can replicate.
Dark-fish2 sequence
How I did the processing
First try, lighten every pixel. Didn’t work. Made all the black whiter as well as the colors No improvement in contrast
Second try, explore under MediaTools first Black parts are really black Lighter parts have really low number values So:
Look for any pixel less black than black (threshold=8) Lighten it a couple values
Lightenfish
import osdef lightenFish(directory): framenum = 0 for framefile in os.listdir(getMediaPath("dark-fish2")): framenum = framenum + 1 printNow("Frame: "+str(framenum)) if framefile.endswith(".jpg"): frame=makePicture(getMediaPath("dark-fish2")
+"//"+framefile) for p in getPixels(frame): color = getColor(p) if distance(color,black)>8: color=makeLighter(color) color=makeLighter(color) setColor(p,color) writeFrame(framenum,directory,frame)
Original sequence again
Same frames after lightening
That took forever!
On a 500 Mhz G4, this took almost a minute per frame to process! Over 160 frames
I started it at 3 pm, checked to make sure that the first few frames looked good, went to a meeting, and collected the result after 4:30
Why?
Why does movie processing take so long? Why does sound processing seem to go so fast?
Why can Photoshop do these things faster than we can in Python?
What makes software fast, or slow? See next lecture…
CS1 Robotics with Python
Microsoft Research has funded the Institute for Personal Robotics in Education Tucker Balch, Directing
Monica Sweat, Developing GT’s CS1 Joint between Bryn Mawr and
Georgia Tech Doug Blank (Developing Myro) Deepak Kumar (Teaching their first
CS1 with robots) http://www.roboteducation.org
Curriculum Goals
Make entry into computer science more accessible Revamp CS1 & CS2 curricula Use personal robots as a “vehicle” Fresh approach to introducing computing Influence robot design based on past experience Influence software development
Pilot CS1 in Spring 2007
Offered at Bryn Mawr (Deepak Kumar) and GaTech (Monica Sweat)
Use the Parallax Scribbler robot Wireless bluetooth interface designed by
GaTech enables Scribbler interface Myro software running under Dr. Python (IDE) to
control Scribbler Course will provide additional data on curriculum
design
Curriculum Development Strategy
Step One: Figure out something that students will want to do.
Step Two: Teach the CS and Robotics to make that happen.
Goals: Relevance Motivation Context for Transferrable Learning
Example CS1 Exercises: Early Focus on variables, arguments, functions, giving
commands to a computer, sequencing commands
Personalize your robot (use colors, pens, stickers, etc.). Give it a personality, and a specific “move” Then experiment with difference basic robot movement
behaviors (go forward, backward, spin, turn, etc.) and learn how to upload and play a tune. Design a series of steps for your robot to perform a dance while playing a tune.
Test the sensitivity and range of IR sensors for light detection, obstacle detection. The latter will vary depending on the reflectance of the object being sensed... Put these results in Excel to introduce ideas of computational
science: Gathering data from sensors and analyzing it.
Example CS1 Exercises: Later Iteration Focus on iterating, sequencing within an iteration, limiting iteration
Experiment with simple behaviors like going forward for a while and then backward. Repeat these behaviors several times. Does the robot track the same space or are the movements shifting its space? A good way to test these would be to insert a pen and let the robot draw its path each time. This will help observe and get used to variations in your robots motor behaviors.
Write a loop to input a sound (could be on laptop instead of robot) then play it back. Do this in a group of robots. Make a sound, allowing the group behavior and dynamics to create a cacophony.
Write programs for the robot to draw a square, a circle, a figure-8, a 5 point star...and other shapes...a spiral?
Example CS1 Exercises: Powerful Representation and Interpretation Using some other data structure or sensed data as a command to drive
other behavior
Create a file with commands like "left on 30 \n right on 30 \n beep" Write a program to read these commands and execute them – in other words, define a mini robot language and write an interpreter for it. Gives us the opportunity to address string processing, a typical CS1 topic.
Have one robot "call out" audio commands (perhaps specific tones, via the speaker), and have other robots "hear" the commands (via microphones) and execute the commands.
Follow a line of symbols/fiducials, where the symbols (colors or scan codes) can be interpreted as music. Start the robot on the line, and as it encounters the markings, it plays a note. The spacing is the timing of the songs. Put one after another, and they play in a round. Have two go together (on two lines) and they play a harmony. The robot could also write the music with a pen. First robot writes the song, second robot plays it.
Parallax Scribbler Robot
Low cost (~$80, ~$60 in bulk) and features
3 photosensors IR sensors in front IR on chasis Servo controlled motors 3 Programmable LEDs Speaker Battery operated (6 AAs) Serial interface
Assessment Methods
Within term: Initial, midterm, final surveys.
Focus group or individual student interviews. Across term:
Tracking – do they stay in CS?How well do they do?
Installing Myro on Windows
1. Download the latest Myro from http://wiki.roboteducation.org/Windows_Setup
2. Save it, open it, and expand it.
3. Double-click on install.bat
Setting up Bluetooth
Testing Myro
Testing the Robot
In other words:
initialize(“com5”)
Controlling the robot’s motors
Most Myro robot movement functions control the speed (specifically, amount of power) of the motor. motors(1.0,0.5) # Gives full power to motor Left,
# half power to motor Right forward(1.0) #Both motors, full speed ahead
forward(1.0,0.25) #Full forward, for 0.25 seconds backward(0.5) #Backwards, half speed turnLeft (0.1) #Turn left, slowly
turnLeft(0.1,1) #Turn left slowly, for 1 second turnRight(0.25) # Turn right, slightly more quickly stop() #STOP EVERYTHING
same as motors(0,0)
Defining a robot function: Yoyo
def yoyo():forward(1)
wait(1) # Wait one second
backward(1)
wait(1)
stop()
Can just type this in, and then execute it as:
yoyo()
Or can enter it into a module (filename)
Parameterizing our YoYo
def yoyo1(speed):forward(speed)
wait(1) # Wait one second
backward(speed)
wait(1)
stop()
Further parameterizing
def yoyo2(speed, wait):forward(speed)
wait(wait) # Wait a bit
backward(speed)
wait(wait)
stop()
Can do something forever
from myro import *initialize(“com5”)def yoyo2(speed, wait):
forward(speed)wait(wait) # Wait one
secondbackward(speed)wait(wait)stop()
def wiggle(speed,waitTime):rotate(speed)wait(waitTime)rotate(-speed)wait(waitTime)stop()
def dance():while True:
yoyo2(1.0,1)wiggle(1.0,0.2)yoyo2(0.5,0.5)wiggle(0.5,0.5)
Use Control-C to stop this.
BUT WAIT, before you try that…
What are all the ways that we can limit time?for seconds in timer(5):
turnRight(0.1,1)
while timeRemaining(5):turnLeft(0.1,1)
time_to_run = 25 start_time = currentTime() while (start_time + time_to_run > currentTime() ) :
forward(0.5,1)
So, trying our dance:
def dance():while timeRemaining(5):
yoyo2(1.0,1)
wiggle(1.0,0.2)
yoyo2(0.5,0.5)
wiggle(0.5,0.5)
Remember that you still need those two functions in your code!
Using the robot as an object
Robot can be controlled using global functions,or as methods to robot objects.
Review
from myro import *
robot = Scribbler(“com4”)
robot.turnRight(0.25); wait(0.25); robot.stop()
robot.turnRight(0.25,0.25)
robot.forward(0.25); wait(0.25); robot.stop()
robot.forward(0.25,0.25)
def function():
while True:
while timeRemaining(n):
for seconds in timer(5):
Reading the Light Sensors
robot.getLight(0)
robot.getLight(1)
robot.getLight(2)
>>> robot.getLight()
[657, 1453, 1025]
Light sensors
Which one is which?
Exercise: Playing blind man’s bluff 0,1,2: left, right, center? Do the values go up with darkness or with light?
Modeling Animals
How do animals sense light? Why do moths move to the light? How do they know which way to turn to
get there? Does it matter if you see vs. smell?
Let’s model light-seeking behavior
Choosing action depending on senses
New statement: if Allows us to test conditions (logical expressions)
and choose actions based on those conditions. if (some logical test goes here) :
The actions go on the line below, indented.if robot.getLight(0) > 800:
robot.turnLeft(0.25,0.1)
Blocks
Just like the lines after def and while, all the lines indented the same after if are part of the same block
A block is defined in Python by indentation. Lines at the same level of indentation are in the same block.
A block is used to define the body of a function and the loop.
A block is used to define the actions of a successful if test.
Where’s the light?
We can compare against a value, but what we really care about is the relative light values.
if robot.getLight(0) > robot.getLight(2):
print "LEFT!"
Signaling a Turn
def signalingTurn(): left = 0 right = 2 while timeRemaining(10): if robot.getLight(left) < robot.getLight(right): print “Left!" if robot.getLight(right) < robot.getLight(left): print "Right!"
signalingTurn()
Audibly signaling a turn
def signalingTurn(): left = 0 right = 2 while timeRemaining(10): if robot.getLight(left) < robot.getLight(right): robot.beep(0.25,400) if robot.getLight(right) < robot.getLight(left): robot.beep(0.25,800)
signalingTurn()
Making Music
beep(1,400) #Plays at frequency 400 Hz # for 1 second
computer.beep(0.5,440)# Has the computer beep at A-above-middle C
# (440 Hz) for ½ second
Can they play a duet?Try it!
Myro Song Format
Myro has a more music-like format that it supports:s = makeSong(“c 1; d .5; d#4 1/2”)
# C in the fifth octave for a whole note
# D in the fifth octave for a 1/2 note
# D-sharp in fourth octave for a 1/2 note
robot.playSong(s,0.75) # Play this for ¾ second
More Speech Options
computer.speak(message) - computer will say the text message
computer.stopSpeaking() - stop talking computer.setVoice(name) - set the voice to a named
voice computer.getVoice() - get the name of the current voice computer.getVoices() - get alternative set of named
voices computer.playSpeech(filename) - play a speech file
(wav file) computer.saveSpeech(message, filename) - save
speech to a wav file
Exercises
Option #1: Follow the Light Write a function that will turn the robot toward
the light, much as an insect might follow the light. Can you turn based on the amount of light?
Option #2: Make the Robot Dance Write a program to play (or sing or give a
speech) and “dance” your Scribbler Use beep() and playSong to make music Use movements like yoyo and wiggle