python tutorial-mining imgur images

Curiosity Bits - curiositybits.com

• This tutorial is created for social scientists interested in grabbing data from the image-hosting site, Imgur (imgur.com).

• Find out more about Python for mining the social web, please visit Curiosity Bits (curiositybits.com).

• Social-Metrics.org also hosts a series of Python tutorials on aggregating and analyzing Twitter/Facebook data. More at social-metrics.org

CURIOSITY BITS©

Get Imgur Data through Python

http://social-metrics.org/


This tutorial shows you how to download images and the images’ meta-data (i.e., image title, description, source, upload time, etc.).

CURIOSITY BITS©

Imgur.com is a popular site for sharing selfies and images intended for persuasion.


CURIOSITY BITS©

Images will be saved in a designated folder.

The metadata will be saved in a SQLite database


CURIOSITY BITS©

Imgur images are available through keyword search, reddit timelines and public albums.

Here are the examples:• http://imgur.com/search?q=hillary images from the search using the

keyword “hillary”• http://imgur.com/r/transtimelines images from a public reddit timeline

called transtimelines• http://imgur.com/gallery/ROYAZ images from a public album called

ROYAZ

Three sources of Imgur images

http://imgur.com/search?q=hillary


http://imgur.com/r/transtimelines

http://imgur.com/r/transtimelines

http://imgur.com/gallery/ROYAZ

http://imgur.com/gallery/ROYAZ


Final note before we start: for simplicity, I am laying out only the most essential steps. Previous tutorials provide details about how to set up a Python programing environment, please visit curiositybits.com and click the PYTHON tab.

CURIOSITY BITS©


1. Install Anaconda Python (with Spyder and Ipython Notebook)2. Install SQLite Browser3. Install four essential Python packages (imgurpython, sqlalchemy, urllib, sqlite3)4. Register a Imgur client to get client ID and client secret 5. Create a SQLite database for images from keyword search6. Download images from the keyword search 7. Create a SQLite database for images from a reddit timeline8. Download images from a reddit timeline9. Create a SQLite database for images from a public album10. Download images from a public album

CURIOSITY BITS©

Steps


CURIOSITY BITS©

Install Anaconda Python

• store.continuum.io/cshop/anaconda/

You can run Python codes in Spyder, which is a component in Anaconda Python

Or use IPython Notebook, a web-based interactive Python environment

https://store.continuum.io/cshop/anaconda/



CURIOSITY BITS©

This is what Spyder looks like. The left side of the window displays codes. The one at the bottom right shows you the results.

IPython Notebook runs your codes on the web and displays results in your browser.

Install Anaconda Python


CURIOSITY BITS©

http://sqlitebrowser.org

Install SQLite Browser

http://sqlitebrowser.org/




CURIOSITY BITS©

• In the Command Prompt, use the command line pip install, followed by the package name, to install necessary packages.

Required packages• Imgurpython• Sqlalchemy• Urllib• sqlite3

Install four essential Python packages

https://github.com/Imgur/imgurpython

http://www.sqlalchemy.org/

https://docs.python.org/2/library/urllib.html

https://docs.python.org/2/library/sqlite3.html


CURIOSITY BITS©

type the following command lines in the command prompt:

pip install imgurpythonpip install sqlalchemypip install urllibpip install sqlite3

Install four essential Python packages


Imgur Fetcher V1 is a collection of codes used in this tutorial. You can download them at https://github.com/cosmopolitanvan/imgur_curiositybits_v1

On the site, you can also find three examples of SQLite database. The database files end with the extention .sqlite

CURIOSITY BITS©

The Python codes used in the tutorial

https://github.com/cosmopolitanvan/imgur_curiositybits_v1





CURIOSITY BITS©

Sign in your Imgur account, go to https://api.imgur.com/oauth2/addclient and get your “security clearance”.

Register a Imgur client

https://api.imgur.com/oauth2/addclient




CURIOSITY BITS©

You need Client ID and Client secret to grab data from Imgur API

Register a Imgur client


CURIOSITY BITS©

Let’s get images with the keyword “hilliary”


We will download all available images, but at first, let’s get their metadata. By metadata, I mean, attributes related to each image. Examples of the metadata include image title, image description, image upload date, image link (which is what we will use to download images).

Create a SQLite database for images from keyword search



You can right-click the link and download the file with the extension .py. Open the file in Spyder.

Or, copy the entire block of codes into Ipython Notebook.


CURIOSITY BITS©

Use the code named Imgur search v1.py

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20search%20v1.py

CURIOSITY BITS©

You don’t have to change anything in this block of codes. It is used for importing necessary Python packages. Think of packages as apps running on iOS.

Open Imgur search v1.py in Spyder





CURIOSITY BITS©

This is where you enter the keyword(s) you want to apply to the search. You can have multiple keywords, wrapped in parenthesis, and separated by a comma.


CURIOSITY BITS©

This is where you enter the client ID and client secret generated in the previous step.


CURIOSITY BITS©

No tweaking is needed for this block. But it gives you a sense of what data are to be collected.


CURIOSITY BITS©

Go to line 136, this is where you specify the name of the SQLite database to be saved. If no absolute file path is given, the database will be saved in the same folder with your Python script (Imgur search v1.py).


CURIOSITY BITS©

Imgur images are indexed into multiple pages. Here, 5 means that we are to get five pages of images from the keyword search. You can put 3 or 1 or 2, just play around to see what number gives you the most adequate, while at the same time manageable amount of data.


CURIOSITY BITS©

Now, the tweaking is done! From the menu, chose Run-Configure and hit RUN


CURIOSITY BITS©

This is the SQLite database that contains all the metadata.


CURIOSITY BITS©

Notice that the last column is called multiple_images.

Most image links will end with an extension name of a picture (.jpg, .png, .,bmp, .gif). If that is the case, the value on the column will be NO.

But, some links contain multiple images, and you will see “this link contains multiple images…” warning in the column.


CURIOSITY BITS©

Now, we have a saved SQLite database with image links. The next step is to use Python to download images from those image links

Use the .py file named Imgur search_downloader v1.py

Download images from the keyword search

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20search_downloader%20v1.py




CURIOSITY BITS©

Go to line 72, and make sure the filename there matches the SQLite database we have just created.



CURIOSITY BITS©

Please create a new folder and enter the folder path here.

Image filename is comprised of the image’s unique identifier id from Imgur API.



CURIOSITY BITS©

Again, this script downloads images from links with the extension of .jpg, .png, .gif, .bmp

If an image link ends with anything but an image extension, the script will prompt a reminder message.



CURIOSITY BITS©

Getting image from a reddit timeline is very similar to getting images from keyword search. This time, we will try executing the Python code in Ipython Notebook.

Use the script called Imgur reddit timeline v1.py

Create a SQLite database for images from a reddit timeline

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20reddit%20timeline%20v1.py




CURIOSITY BITS©

Create a new notebook in IPython Notebook and copy and paste the code from Imgur reddit timeline v1.py

Create a SQLite database for images from a reddit timeline






CURIOSITY BITS©

Enter the name of a reddit timeline. In the URL term, a reddit timeline displayed ashttp://imgur.com/r/buffalo

Enter your client ID and client secret.


CURIOSITY BITS©

Click Run Cell

The output will be saved in your IPython Notebooks folder. By default, it is …\Documents\IPython Notebooks


CURIOSITY BITS©

Download images from a reddit timeline

Use the .py file named Imgur reddit_downloader v1.py

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20reddit_downloader%20v1.py




CURIOSITY BITS©


Load the code in IPython Notebooks, Like what you do to download images from keyword search, make sure the file path in the code matches the database you have just created.


CURIOSITY BITS©

Create a SQLite database for images from a public albumExactly as what we do in previous steps. There are only a few places in the script that need tweaking. You need to enter client ID, client secret, the name of the album, the file path of the database, the number of pages to be grabbed. Then you are all good to go!

To create a SQLite database for images from a public album, use the script named Imgur album v1.py

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20album%20%20v1.py

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20album%20%20v1.py


CURIOSITY BITS©

Download images from a public album

To download images from a public album, use the script named Imgur alubm_downloader v1.py

https://github.com/cosmopolitanvan/imgur_curiositybits_v1/blob/master/Imgur%20alubm_downloader%20v1.py




python tutorial-mining imgur images

Education

curiosity bits

curiosity bits images

curiosity bits imgur

curiosity bits http

hillary images

comrtranstimelines images

sources of imgur images

comgalleryroyaz images