impure guide

7/30/2019 Impure Guide

1/39

a guide to impurethe human-internet interface

Unlocking, transforming, storing, distributing, and switching about are ways of revealing.

from The Question Concerning Technlogy

Martin Heidegger

Impure is an internet application that allows people to be part of the information revolution.With impure is possible to get information from very different sources; from user owneddata to diverse feeds in internet, including social media data, real time or historical finan-cial information, images, news, search queries and many more. Impure is a tool to be intouch with data around internet, to deeply understand it. Within a modular logic interfaceyou can quickly link information to operators, controls and visualization methods, bringingall the power of the comprehension of information and knowledge to the not programmersthat want to work with information in a professional way. Among other possibilities, impure

allows you to:

- easily read data from diverse sources and repositories

- load your own data(text files, excel, xml...)

- visualize it in a wide range of ways (more than 100 visualization methods so far)

- process it.. compare it... mix it.. filter it... (more than 300 controls and operations so far)

- publish and share your projects


2/39

impure is intuitive; you dont have to type any code. Instead, you will link modules to cre-ate information flows, that begins with feeds or inputs and ends with processed data orvisualizations. In the middle, interactive controls allow you to choose or modify parametersand to see the results immediately.

impure makes sense of internet / impure is your brain inside internet / the internet tool /talk with internet / the knowledge tool

Basic functionality

Impure is about modules that can be connected. Each module have receptors and somehave an emitter. Each receptor or emitter receives or sends an specific type of data, repre-sented by in icon. Data types are also called data structures.

Among the most simple data structures that can be shared between modules are: String,Number, List and NumberList. These are also the most common.

There are different types of modules: data structures, operators, controls and visualizators.

You can search them on the library, and drag them to the impure space.


3/39

You can search modules in the library by selecting tags. Modules that are more relatedwith the selection (because they have all or some of the selected tags) will appear in firstpositions.


4/39


5/39

data structures modules

A module that contain information. They only have an emitter.

Though there exists many data structures, only some basic data structures can beplaced as modules on an impure space. In the library you will recognize them by asmall arrow on their icons.

operators

Modules with one or more receptors that perform some operations and return theresult on one emitter.


6/39

controls (under the tag control)

Modules that allow some kind of interaction or executes some complex task (such adownload data). Among controls a very useful type of modules are the apis, thatsearch information in different places on internet and create data structures.

visualizators

Modules that receive information and visualize it. They can also allow interactionand some visualizators have an emitter.

An impure file have modules that bring information (loaders, apis, data structures withtyped information), modules that process information (operators, controls), and modulesthat visualize information (visualizators).


7/39

inheritance

Some data structures inheritate from other. For instance NumberList inheritate from List.That means that NumberList its also a List, and each receptor that works with List will also

accept NumberList (and StringList, DateList, Table, ...)

The example of the image shows how a NumberList is reversed. The operator reverseListreceive a List and off course it can reverse any kind of List, including a NumberList, aStringList or a Table.

Thats why sender and receiver icons doesnt match.

impure code

Each impure file has a code that you can copy to clipboard by clicking the blue C button(top right). You can also open the code editor with the red C button.

This is useful to easily share impure files. You can send the code by mail. To compile acode paste it into the code editor and click the compile arrow on the top.

Some examples presented here come with code, so you can compile them easily.


8/39

universal operators (aka polymorphism)

Some operators receive and give Object as data structure. It actually means that the op-erator allow different types of data structures, and the data structure it receives determinesthe data structure it returns. The best example is the operator addition which is able tooperate a lot of different data structures combination. Among them:

Number Number Number

Number NumberList NumberList(adds the first number to all values on the NumberList)

String String String

(concatenation)

String StringList StringList

(concatenates the first String to all the Strings on the StringList)

Point Polygon2D Polygon2D

(vector sum of the Point to all the Points on the Polygon2D)

etc...

Another interesting universal operator is interpolation that will try to create an interpolatedobject with two objects and a number between 0 to 1. The data structure of the obtainedobject is off course the same that the two given objects.

required and optional parameters

Modules with receptors may have required and optional parameters. Only when requiredparameters are fulfilled the module will be able to work. Otherwise operators will return nullvalue, and visulizators will not depict nothing.

Optional parameters have grey icons.


9/39

remarkable tags

Some tags group interesting and useful modules and worth a look:

! apiModules under this tag allow communication with internet services such as google, twitter,wikipedia, xignite (finance and market info), delicious, etc... These modules are crutial be-cause they are the main interface between impure and internet.

! generatorOperators with this tag creates complex structures from simple parameters. They are use-ful to quickly introduce data to the space without needing of load it.

! coder and decoder

! filterOperators that take an structure and creates a new one that contains less informationwithout transformation.

! conversionOperators that transforms structures preserving information.

!advanced

Operators that perform advanced tasks.

! universalOperators, controls and visualizator that deal with any (or many) data structures.

! vector


10/39

Data structures under this tag are vectors. Under this tag you find operators and controlsthat deal with vectors. Vectors are a special type of data structure, often numerical, andwhich allow operations such as multiplication of a vector by a number, or addition betweenvectors. Some vectorial operators are very useful such as interpolation. If an structure is avector (a Rectangle or NumberList, for instance) you can take two of them and calculate adata model in between. You can also use the ExponentialConvergence to create continual

transformation over data. Thatsparticular useful when applied to visualizators that readvectors.


11/39

How to quickly bring data to impure

There are three main ways to bring data to an impure space:

# - using an api (under the tag api)

Using an api is one of the fastest ways to obtain rich structured informationfrom internet. For instance, you can download a delicious account, all theimages -with their tags- from a flickr set, the historical market behavior of acompany, the occurrences of some word on twitter accounts last month,etc...

# - generating data (under the tag generator)

Some operators under the tag generator build data structures, such a ran-dom network (you choose the number of nodes and relations), a NumberListwith the same repeated number. Generators are useful to do quick tests.

#


12/39

- loading data from files (FileLoader module)

Perhaps you have tables in excel files or csv format, or some text you wantto analyze. Impure allows you to load the file using the FileLoader moduleand then you might decode it. If the text file you loaded is in csv format youhave to link the FileLoader module to the decoder operator


13/39

How to quickly visualize data

If you already have placed data into an impure space, chances are that you can visualize itimmediately. You have only to put in space the appropriate visualizator and link the data toit.

Of course the visualizations you can choose depends on the type of data you bring to theimpure space.

Some visualizators require more than one data structure.

Lets see some common information structures and some linkable visualizators.

NumberList

# ! Histogram

two NumberLists

# ! ColorScatter


14/39

NumberTable

# ! SimpleNumberTableVisualizator

Network

# ! Oracle


15/39

Tree

# ! ZoomableTreeMap


16/39

How to process information

Sometimes you want to go further than picking a data file and visualize it using a specificvisualization method. Impure allows much more complex processes with more interestingresults:

- to transform data (by operations, normalizations, filtering, parsing, comparing or combin-ing different sources...)

- to use interactive controls that allow parameters or data changes

- to combine controls and visualizators to build an integral interactive space

Working with numbers

basic numeric structures: Number, NumberList, NumberTable

Numbers, besides being the most common data raw material, are highly important be-

cause they are used as parameters for many operators, controls and visualizators. You willoften use the Number module.

NumberLists are Lists (see inheritance, above), and that means it has a lot of availableoperators besides the specific ones for these data structure.

Among the most important operators you can apply to NumberLists are all the normaliza-tions (to maximum, to sum...) that creates a new NumberList, average, mean and standarddeviation whose results are numbers.

You may might want to compare two NumberLists (two companies market behaviors,

trends, country indexes) in order to try to figure out if their are correlated. Yo can then usea Scatter or a ColoredScatter take a glance to correlation. You can also measure its Pear-son correlation value, the most common index of statistical correlation.

NumberTables are very common, its easy to obtain one from an excel file or a .csv file(this file format is pretty common nowadays because it is often the one to be used to sharetables data in open data repositories). But not every Table is made of numbers, some ofthem are a mix of different kinds of data. You can choose the numerical lists and assemblea NumberTable, but much more easy is to use the NumberTableFromTable operator thatwill filter all the non NumberLists of the Table.

A NumberTable is depictable in several ways.


17/39

Working with texts

basic string structures: String, StringListbasic operators: splitString, occurrences

Handle text operations is quite useful, because a lot of queries are made of text.

On the other hand text analysis is the basis for social media analysis. What people aresaying about something? How often? How to know which assets are being related withsome product in the social media ecology? After reading contents from blogs, news, twitterand so on, you might want to analyze the obtained result. Some options are: count the oc-currences of a word, count how much two words occur in the same sentence or para-graph, compare the entire lexicon used by two different sources, visualize the commonwords used in different resources,...

Text analysis is also the key for data mining, parsing and filtering. And when it takes toparse data, these operators become very useful:

stringTransformations

firstTextBetweenStrings

getAllStringsBetweenTwoStrings

replaceSubstring

..building a network from a StringList...

...rematkable works...


18/39

Working with lists

basic lists structures: List, Table, NumberList, StringList

A lot of data structures inheritates from List. And that means that all these structures areable to be operated as Lists. Thats why Lists are so important.

Almost for every data structure it exists a type of List that contain that kind of elements.

Different types of List have different functionalities. For instance its possible to seacrhNodes in ListNode using the id of the Node. NumberLists have a lot of statistical and mathoperators.

Things you will want to do with Lists:

- assamble a List (that is: take some data structures and create a List with them)

- add elements on a List

- remove elements of a List

- concatenate two Lists

- obtain a single element from a List

- obtain a group of elements of List

- sort a List (using different sorting criteria)

Some of the basic operators for a List are:

getElementFromList

getSubList

getSubListFromPositions

pushToList


19/39

Working with tables

basic lists structures: Table, NumberTable

Tables are Lists of Lists.

By convention Tables lists are interpreted as columns. The fact of a Table being a Listmight be confusing for those who are used to work with tables as excel sheets. In excel,columns and rows are treated as equals. In Impure Tables, rows doesnt exists a priori! (offcourse they can be obtained, in fact generated).

So, if you want to obtain a specific list (a column) theres no getListFromTable operator,instead you have to use getElementFromList, because Tables are Lists!

This scheme reveals the structure of a Table, and put it clear that a Table is a List of Lists:


20/39

The image clearly shows that a Table may have Lists of different lengths. This might not bethe most common case, but could happen.

Use the TableAtAGlance for quickly visualize the structure of a Table and its contents.

Some basic and useful operators for Tables:

getElementFromList to obtain a List (column) from a TablegetSubList to obtain a sub-Table

and any List opertor

more specific for Tables:

getRowgetElementFromTable


21/39

Working with networks

basic network structures: Node, NodeList, Network, Tree

Networks are complex structures, at least more complex than lists or tables. But they ex-tremely important. Neurons, genes, phones, people are structured by networks not by lists.

One issue when working with networks is that its not easy to find network data, in opposi-tion to numerical or text data. Major data repositories (see Where to find interesting andspecific data? below) offer data in table formats such as .csv. Formats for networks, in-stead are rare. We expect the arise of social networks and apis for services such as Face-book, will make data networks standards more common.

In the meanwhile impure allows you to load the most common and simple network dataformats: .gdf and grapML.

But what is much more interesting is that impure comprehend several ways to create net-works. Fists of all, when you download a RSS feed, the generated data structure is a Net-work instead of a table or a list. The nodes of this Network are contents and tags. Fromthis Network you can create a new Network of contents (creation relations whenever twocontents share tags) or a Network of tags (creation relations whenever two tags share con-tents).

With a table with two lists and a NumberLists that somehow express the proximity betweenthe pairs of the list you can build a Network.

From a NumberList its also possible to create Networks: comparing each pais of Num-berLists and calcultaing its statistical correlation. or its distance, and creating relationswhen these values surpass a threshold. In general terms, when you have a distance crite-ria for certain type of elements you can build a Network from a List of these elements.

See working with texts for an exampe on how to build a Network from a StringList.


22/39

Working with graphic elements

basic graphic structures: BitmapData, ColorList, ColorScale

Among graphic elements colors are the most important, because they are useful to repre-sent quantities. Some visualizators such a SimpleMap use colors to represent values. Inmany cases its optional. Colors help also to element differentiation. Its easy to create aColorList or a ColorScale for a NumberList.


23/39

Working with geographic information

basic geographic structures: PointGeo, Region, RegionList, FrameGeo, Country


24/39

Working with time information

basic dates structures: Date, IntervalDate, DateList, IntervalDateList


25/39

How to use internet search to obtain valuable data

Multi-search is a powerful technique to obtain information from internet. Search enginessuch as google give you a list of webpages with occurrences of the string you typed.Sometime you want to obtain more information, and also its possible that you dont wantto find information about an specific concept, word or idea, but about a set os related con-cepts.

Imagine you want to know about the similarities and differences between each pair ofinternet browser applications. The conventional way is to search a webpage in whichsomeone else already did the task and made this comparison. You have to be lucky. In theother hand it is much more likely that a lot of people made specific pair comparisons. If wecan ask google to return all the pages in which two browser are compared we will obtainmuch more valuable information. But how to do that? It would be great if we can typesomething like this:

[browser_brand_here] compaired with [browser_brand_here]

or:

[browser_brand_here] is faster than [browser_brand_here]

Or, you can type all the possible combinations. With 5 browsers its not a huge task (only20 possible pairs without comparing browser with their selfs). But results come in sepa-rated windows.. and what if you want to compare 50 car models (2450 different search op-erations)?.

Off course you can do that automatically with impure. First you load or type the list ofbrowsers. Then you combine this List, creating a table with two Lists in which it exists allpossible combination of pairs. You take the first StringList and you add (addition operation)it with the String compaired to, you obtain a new StringList that you will add to the sec-ond StringList of the Table. You add a quotation to the StringList, and yo add the result toanother quotation mark. Now you have a StringList with all the sentences of the kind:

[browser_brand_here] is faster than [browser_brand_here]

Quotation marks are important because they mean tha the seach will be strict, and google

will return webpages in which the complete sentences are finded.

You link this StringList to the InternetMultiNSearchResults.

(


26/39

advanced concepts:

internet perception

internet distance

externality

remarkable words

)


27/39

This great cartoon by http://xkcd.com depict some of the questions its possible to ask tointernet using google search.
http://xkcd.com/http://xkcd.com/


28/39

Working with urls

urls are strings, so you will work with all the string operators.


29/39

Where to find interesting and specific data?

Available well structured and commented data is increasing dramatically. 2010 saw theappearance of many great data repository, most of them being part of transparency poli-

cies lead by some governments. This is the open data revolution we are witnessing.

Since its easy to find tons of interesting data, if you have a specific need it could be veryhard, even impossible.

The coming years we will see the appearance, growing and enhancement of data reposi-tories with relevant information about many realties (not only country specific information).

This a list of increasing factors, with positive interaction between them:

- people producing information

- people needing to share information

- people and institutions needing information

- people and institutions needing tools to understand and add value and knowledgeto information

- people warned about the relevance and they rights of information being public

- people learning new tools to understand, process, visualizae and publish informa-

tion

- institutions sharing and publishing information (by ethical or political means, bynecessity and strategy, or by social pressure)

This is a list of great data repositories:

http://delicious.com/moebio/+%5BdataRepository%5D

A good place to explore when it takes to find interesting data to play with it on impure isthe Guardian data blog: http://www.guardian.co.uk/news/datablog

At Bestiario we are creating a modest but diverse and well organized collection of datafiles: http://delicious.com/datarepository Some of these are used in the impure examples.
http://delicious.com/datarepositoryhttp://delicious.com/datarepositoryhttp://www.guardian.co.uk/news/databloghttp://www.guardian.co.uk/news/databloghttp://delicious.com/moebio/http://delicious.com/moebio/http://delicious.com/datarepositoryhttp://delicious.com/datarepositoryhttp://www.guardian.co.uk/news/databloghttp://www.guardian.co.uk/news/databloghttp://delicious.com/moebio/http://delicious.com/moebio/


30/39

understanding examples 1

You have always access to a list of examples. Some are very simple other are quite hardto understand from a first glance and require very attentive information flow following.

You will open and try to understand all the examples except for the ones contained in theadvanced folder, following this order:

- basic

- visualizators

- apis

- controls

- operators

In each case it will be easy to follow the data flow, from the loaders, apis or generators,continuing with operators and controls, and finishing with visualizators.


31/39

Exercices

exercise 1: from an excel file

Open a excel file. It should have a simple structure: a table with rows and columns. Forthis exercise purposes it should contain at least two columns filled with numbers (exceptfor the first cell that could be the header).

Export this table to a CSV file, and put it on a server or on folder. You have to identify thepath to this file.

Create a new impure space, add a String module and paste the path to the .csv file.


32/39

Now place a FileLoader (tags: control, loader), and link the Strong Module to it. Once thefile is loaded plug a csvToTable operator (tags: decoder, tables).

If the first row of the table are headers, link a Boolean module to the second parameterwith true as value.

So far the impure space is something like that:

In the near future it will exist table visualizators, so far you can only visualize Num-berTable, with is a Table filled with numbers. So, you have to filter the table, taking only thenumber lists and rebuild a new table. There is an operator that do all the job: getNum-berTableFromTable (tags:tables, numeric, filter).

Finally, place a SimpleNumberTableVisualizator (tags: tables, numeric) and like getNum-berTableFromTable to it. Place Boolean modules to change the optional parameters of thevisualizator.


33/39

exercise 2: your facebook friends graph

Follow these directions in order to obtain a .gdf file that contains your facebook graph(friends and relations):http://thepoliticsofsystems.net/2010/03/22/netvizz-facebook-to-gephi/.

You can load a .gdf of a facebook group too.

(about the .gdf standard: http://gephi.org/users/supported-graph-formats/gdf-format/andby the way you can also download gephi application wich is great for networks visulization,and is free )

Simple load the file to the impure space and visualize it with some of the network visualiza-tions impure have.

The most impressive is the ForceDirectedNetworkVisuallizator, although it allows repre-sentation up to 300 nodes.
http://gephi.org/users/supported-graph-formats/gdf-format/http://gephi.org/users/supported-graph-formats/gdf-format/http://thepoliticsofsystems.net/2010/03/22/netvizz-facebook-to-gephi/http://thepoliticsofsystems.net/2010/03/22/netvizz-facebook-to-gephi/


34/39

exercice 3: assets perception on blogs

As commented above, impure could be very useful to read the web, in particular to un-derstand what people perception is about a lot of things. This exercice is an example on

how to read and understand the social media.

Choose two comparable concepts, objects names, or people names. Choose a list of ad-jectives you expect to fit in these concepts, objects or persons -called items from now..

We gonna ask internet what people are saying abut these items, which of the adjectivesare being used with each of them and how much it is used.

Place a string module, with the name of the item, a blank space, the world is and a finalblank space. Something like this:

(be aware of the space after the is)

In other String module write a list of adjectives, or text that complete the sentence xxx is .For instance:

greed, an inspiration, great, the best company, the future, innovative, falling, growing

Each part is separated by a blank space and a choma.

Split this string using the splitString operator (tags: basic, strings, lists). By the default theseparator string is ,, which is the one we need here. The result is a StringList with all theadjectives or objects.

Then use the addition operator (tags: numeric, universal, basic), and add the String xxx is

to the new StringList. This operator will concatenate the first string to each string in theStringList creating a new StringList in which every string has the structure x is y.

This StringList is a list of queries; for each query we want to know how many occurrencesthere are in internet. The proportion of this occurrences will give you an idea of what peo-ple (at least internet content-publisher people) thinks about something. Of course it de-pends a lot of what adjectives and objects you had choose.

Connect the StringList to the InternetMultiNSearchResult control (tags: control, loader,search, strings). It will search each string and create a NumberList with the occurrences ofeach one. This numbers are the same you see as search results below the search bar

when you perform a search task with google.


35/39

Finally use a CirclesTagCloud or a Quadrification to visualize the results (link the StringListand the NumberList).


36/39

exercice 4: number lists comparison

A very common structure is Table with a first column (list) filled with names or objects, andthe others with numbers. If the Table have only two lists: names and values, you can use

the Histogram, the CirclesTagCloud or the Quadrification to visualize it, in this cases youhave to use the getListElement (remember that Table is a List o Lists, so if you read thefirst element of the Table it will be the first List, and so on).

Once you have a Table with objects or names in the first List, and two or more Num-berLists, you may want to use some kind of comparative visualization method.

In order to perform this exercise you have to find a table with a list of names and two ormore number lists.

This data could be interesting:

http://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goals

The main idea in this case is to try to figure out if budget and revenues are correlated ornot.

You load and decode the table. The result should be a Table with at least three Lists. Thenyou place three getElementFromList operators on the space, each one picking a differentList, the first one being a StringList and the other two NumberLists.

Now you have two interesting an immediate options. If values determine a sorting, youmay be interested in see how each NumberList define a different sorting and to comparethe two sorting lists. In the example of maternal mortality its interesting to see the list ofcountries sorted from lees to more mortality in each year, and try to visualize which coun-tries remained in the same positions and which ones changed. For that purpose use theComparativeSort visualizator.
http://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goalshttp://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goalshttp://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goalshttp://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goalshttp://www.guardian.co.uk/news/datablog/2010/apr/12/maternal-mortality-rates-millennium-development-goals


37/39

For instance, from a csv file:

title;classification;year;budget;gross;duration;audience

Aliens;1;1986;18.5;81.843;137;8.2

Armageddon;2;1998;140;194.125;144;6.7As Good As It Gets;2;1997;50;147.54;138;8.1

Braveheart;1;1995;72;75.6;177;8.3

Chasing Amy;1;1997;0.25;12.006;105;7.9

Contact;2;1997;90;100.853;153;8.3

Dante's Peak;2;1997;104;67.155;112;6.7

Deep Impact;2;1998;75;140.424;120;6.4

Executive Decision;1;1996;55;68.75;129;7.3

Forrest Gump;3;1994;55;329.691;142;7.7

Ghost;3;1990;22;217.631;128;7.1

Gone with the Wind;3;1939;3.9;198.571;222;8

Good Will Hunting;1;1997;10;138.339;126;8.5

Grease;3;1978;6;181.28;110;7.3

Halloween;1;1978;0.325;47;93;7.7

Hard Rain;1;1998;70;19.819;95;5.2

I Know What You Did Last Summer;1;1997;17;72.219;100;6.5

Independence Day;2;1996;75;306.124;142;6.6

Indiana Jones and the Last Crusade;2;1989;39;197.171;127;7.8

Jaws;2;1975;12;260;124;7.8

Men in Black;2;1997;90;250.147;98;7.4

Multiplicity;2;1996;45;20.1;117;6.8

Pulp Fiction;1;1994;8;107.93;154;8.3Raiders of the Lost Ark;2;1981;20;242.374;115;8.3

Saving Private ryan;1;1998;70;178.091;170;9.1

Schindler's List;1;1993;25;96.067;197;8.6

Scream;1;1996;15;103.001;111;7.7

Speed 2:Cruise Control;2;1997;110;48.068;121;4.3

Terminator;1;1984;6.4;36.9;108;7.7

The American President;2;1995;62;65;114;7.6

The Fifth Element;2;1997;90;63.54;126;7.8

The Game;1;1997;50;48.265;128;7.6

The Man in the Iron Mask;3;1998;35;56.876;132;6.5Titanic;2;1997;200;600.743;195;8.4

True Lies;1;1994;100;146.261;144;7.2

Volcano;2;1997;90;47.474;102;5.8

I took the first, the fourth and the fifth lists, and the result was:


38/39

Other approach is to try to depict whether or not two NumerLists are correlated. A classicmethod to do that is the Scatter. But if the two NumberLists are sorted (chronologically, forinstance) you can visualize the relation between values with ColoredScatter.

-----

exercise 5: comparing twitter trends

[!] twitter apis doesnt work yet


39/39

understanding examples 2

Now you are prepared to understand the examples in the advanced folder.

Some of them have tricky operations, but in general terms you will be able to understandthe general structure, and to see how complex operations could be performed combiningthe existing ones.

reverse engineering is not that easy with impure. But you can always save the examplesas new files and modify them.

You will recognize some of this examples becuase they are evolutions of some the exer-cises you already did.

impure guide

Documents