building a mini google high performance computing in ruby
DESCRIPTION
TRANSCRIPT
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building Mini-Google in Ruby
Ilya Grigorik
@igrigorik
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
postrank.com/topic/ruby
The slides… Twitter My blog
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Ruby + MathOptimization
PageRank
IndexingExamplesMisc Fun
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank PageRank + Ruby
IndexingExamplesTools
+ Optimization
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Consume with care…everything that follows is based on released / public domain info
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Search-engine graveyardGoogle did pretty well…
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Search pipeline50,000-foot view
Query: Ruby
Results
1. Crawl 2. Index 3. Rank
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Query: Ruby
Results
1. Crawl 2. Index 3. Rank
Bah FunInteresting
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
circa 1997-1998
CPU Speed 333MhzRAM 32-64MB
Index 27,000,000 documentsIndex refresh once a month~ishPageRank computation several days
Laptop CPU 2.1GhzVM RAM 1GB1-Million page web ~10 minutes
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Creating & Maintaining an Inverted Index DIY and the gotchas within
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building an Inverted Index
require 'set'
pages = {"1" => "it is what it is","2" => "what is it","3" => "it is a banana"
}
index = {}
pages.each do |page, content|content.split(/\s/).each do |word|
if index[word]index[word] << page
elseindex[word] = Set.new(page)
endend
end
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building an Inverted Index
require 'set'
pages = {"1" => "it is what it is","2" => "what is it","3" => "it is a banana"
}
index = {}
pages.each do |page, content|content.split(/\s/).each do |word|
if index[word]index[word] << page
elseindex[word] = Set.new(page)
endend
end
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building an Inverted Index
require 'set'
pages = {"1" => "it is what it is","2" => "what is it","3" => "it is a banana"
}
index = {}
pages.each do |page, content|content.split(/\s/).each do |word|
if index[word]index[word] << page
elseindex[word] = Set.new(page)
endend
end
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
Word => [Document]
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Querying the index
# query: "what is banana"p index["what"] & index["is"] & index["banana"]# > #<Set: {}>
# query: "a banana"p index["a"] & index["banana"]# > #<Set: {"3"}>
# query: "what is"p index["what"] & index["is"]# > #<Set: {"1", "2"}>
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
1 32
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Querying the index
# query: "what is banana"p index["what"] & index["is"] & index["banana"]# > #<Set: {}>
# query: "a banana"p index["a"] & index["banana"]# > #<Set: {"3"}>
# query: "what is"p index["what"] & index["is"]# > #<Set: {"1", "2"}>
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
1 32
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Querying the index
# query: "what is banana"p index["what"] & index["is"] & index["banana"]# > #<Set: {}>
# query: "a banana"p index["a"] & index["banana"]# > #<Set: {"3"}>
# query: "what is"p index["what"] & index["is"]# > #<Set: {"1", "2"}>
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
1 32
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Querying the index
# query: "what is banana"p index["what"] & index["is"] & index["banana"]# > #<Set: {}>
# query: "a banana"p index["a"] & index["banana"]# > #<Set: {"3"}>
# query: "what is"p index["what"] & index["is"]# > #<Set: {"1", "2"}>
{"it"=>#<Set: {"1", "2", "3"}>,"a"=>#<Set: {"3"}>,"banana"=>#<Set: {"3"}>,"what"=>#<Set: {"1", "2"}>,"is"=>#<Set: {"1", "2", "3"}>}
}
What order?
[1, 2] or [2,1]
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building an Inverted Index
require 'set'
pages = {"1" => "it is what it is","2" => "what is it","3" => "it is a banana"
}
index = {}
pages.each do |page, content|content.split(/\s/).each do |word|
if index[word]index[word] << page
elseindex[word] = Set.new(page)
endend
end
Hmmm?
PDF, HTML, RSS?Lowercase / Upcase?
Compact Index?Stop words?Persistence?
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Ferret is a high-performance, full-featured text search engine library written for Ruby
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
require 'ferret'include Ferret
index = Index::Index.new()
index << {:title => "1", :content => "it is what it is"}index << {:title => "2", :content => "what is it"}index << {:title => "3", :content => "it is a banana"}
index.search_each('content:"banana"') do |id, score|puts "Score: #{score}, #{index[id][:title]} "
end
> Score: 1.0, 3
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
require 'ferret'include Ferret
index = Index::Index.new()
index << {:title => "1", :content => "it is what it is"}index << {:title => "2", :content => "what is it"}index << {:title => "3", :content => "it is a banana"}
index.search_each('content:"banana"') do |id, score|puts "Score: #{score}, #{index[id][:title]} "
end
> Score: 1.0, 3
Hmmm?
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
class Ferret::Analysis::Analyzerclass Ferret::Analysis::AsciiLetterAnalyzerclass Ferret::Analysis::AsciiLetterTokenizerclass Ferret::Analysis::AsciiLowerCaseFilterclass Ferret::Analysis::AsciiStandardAnalyzerclass Ferret::Analysis::AsciiStandardTokenizerclass Ferret::Analysis::AsciiWhiteSpaceAnalyzerclass Ferret::Analysis::AsciiWhiteSpaceTokenizerclass Ferret::Analysis::HyphenFilterclass Ferret::Analysis::LetterAnalyzerclass Ferret::Analysis::LetterTokenizerclass Ferret::Analysis::LowerCaseFilterclass Ferret::Analysis::MappingFilterclass Ferret::Analysis::PerFieldAnalyzerclass Ferret::Analysis::RegExpAnalyzerclass Ferret::Analysis::RegExpTokenizerclass Ferret::Analysis::StandardAnalyzerclass Ferret::Analysis::StandardTokenizerclass Ferret::Analysis::StemFilterclass Ferret::Analysis::StopFilterclass Ferret::Analysis::Tokenclass Ferret::Analysis::TokenStreamclass Ferret::Analysis::WhiteSpaceAnalyzerclass Ferret::Analysis::WhiteSpaceTokenizer
class Ferret::Search::BooleanQueryclass Ferret::Search::ConstantScoreQueryclass Ferret::Search::Explanationclass Ferret::Search::Filterclass Ferret::Search::FilteredQueryclass Ferret::Search::FuzzyQueryclass Ferret::Search::Hitclass Ferret::Search::MatchAllQueryclass Ferret::Search::MultiSearcherclass Ferret::Search::MultiTermQueryclass Ferret::Search::PhraseQueryclass Ferret::Search::PrefixQueryclass Ferret::Search::Queryclass Ferret::Search::QueryFilterclass Ferret::Search::RangeFilterclass Ferret::Search::RangeQueryclass Ferret::Search::Searcherclass Ferret::Search::Sortclass Ferret::Search::SortFieldclass Ferret::Search::TermQueryclass Ferret::Search::TopDocsclass Ferret::Search::TypedRangeFilterclass Ferret::Search::TypedRangeQueryclass Ferret::Search::WildcardQuery
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
ferret.davebalmain.com/trac
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Ranking Results0-60 with PageRank…
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Naïve: Term Frequency
index.search_each('content:"the brown cow"') do |id, score|puts "Score: #{score}, #{index[id][:title]} "
end
> Score: 0.827, 3> Score: 0.523, 5> Score: 0.125, 4
Relevance?
3 5 4
the 4 3 5
brown 1 3 1
cow 1 4 1
Score 6 10 7
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Naïve: Term Frequency
index.search_each('content:"the brown cow"') do |id, score|puts "Score: #{score}, #{index[id][:title]} "
end
> Score: 0.827, 3> Score: 0.523, 5> Score: 0.125, 4
Skew
3 5 4
the 4 3 5
brown 1 3 1
cow 1 4 1
Score 6 10 7
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
TF-IDFTerm Frequency * Inverse Document Frequency
Skew
3 5 4
the 4 3 5
brown 1 3 1
cow 1 4 1
Total # of documents: 10
# of docs
the 6
brown 3
cow 4
Score = TF * IDF
TF = # occurrences / # wordsIDF = # docs / # docs with W
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
TF-IDFScore = 0.204 + 0.120 + 0.092 = 0.416
# of docs
the 6
brown 3
cow 4
3 5 4
the 4 3 5
brown 1 3 1
cow 1 4 1
Total # of documents: 10# words in document: 10
Doc # 3 score for ‘the’:4/10 * ln(10/6) = 0.204
Doc # 3 score for ‘brown’:1/10 * ln(10/3) = 0.120
Doc # 3 score for ‘cow’:1/10 * ln(10/4) = 0.092
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Frequency Matrix
W1 W2 … … … … … … WN
Doc 1 15 23 …
Doc 2 24 12 …
… … … …
…
Doc K
Size = N * K * size of Ruby object
Ouch.
Pages = N = 10,000Words = K = 2,000Ruby Object = 20+ bytes
Footprint = 384 MB
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
NArrayhttp://narray.rubyforge.org/
NArray is an Numerical N-dimensional Array class (implemented in C)
NArray.new(typecode, size, ...)NArray.byte(size,...)NArray.sint(size,...)NArray.int(size,...)NArray.sfloat(size,...)NArray.float(size,...)NArray.scomplex(size,...)NArray.complex(size,...)NArray.object(size,...)
# create new NArray. initialize with 0.# 1 byte unsigned integer# 2 byte signed integer# 4 byte signed integer# single precision float# double precision float# single precision complex# double precision complex# Ruby object
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
NArrayhttp://narray.rubyforge.org/
NArray is an Numerical N-dimensional Array class (implemented in C)
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRankthe google juice
Links as votes
Problem: link gaming
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Random Surferpowerful abstraction
Follow link from page he/she is currently on.
Teleport to a random location on the web.
P = 0.85
P = 0.15
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Surfin’rinse & repeat, ad naseum
Follow link from page he/she is currently on.
Teleport to a random location on the web.
Page K
Page N Page M
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Surfin’rinse & repeat, ad naseum
On Page P, clicks on link to K
P = 0.15
P = 0.85
On Page K clicks on link to M
On Page M teleports to X
…
P = 0.85
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Analyzing the Web Graphextracting PageRank
P = 0.6
N
MK
X
P = 0.15
P = 0.20P = 0.05
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
What is PageRank?It’s a scalar!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
What is PageRank?it’s a probability!
P = 0.6
N
MK
X
P = 0.15
P = 0.20P = 0.05
P = 0.6
P = 0.15
P = 0.20P = 0.05
P = 0.6
P = 0.15
P = 0.20P = 0.05
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
What is PageRank?it’s a probability!
P = 0.6
N
MK
X
P = 0.15
P = 0.20P = 0.05
P = 0.6
P = 0.15
P = 0.20P = 0.05
Higher Pr, Higher Importance?
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Teleportation?sci-fi fans, … ?
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Reasons for teleportationenumerating edge cases
N
M
K
X
1. No in-links!
M
2. No out-links!
3. Isolated Web
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Exploring Graphsgratr.rubyforge.com
•Breadth First Search•Depth First Search•A* Search •Lexicographic Search •Dijkstra’s Algorithm •Floyd-Warshall•Triangulation and Comparability detection
require 'gratr/import'
dg = Digraph[1,2, 2,3, 2,4, 4,5, 6,4, 1,6]
dg.directed? # truedg.vertex?(4) # truedg.edge?(2,4) # truedg.vertices # [5, 6, 1, 2, 3, 4]
Graph[1,2,1,3,1,4,2,5].bfs # [1, 2, 3, 4, 5]Graph[1,2,1,3,1,4,2,5].dfs # [1, 2, 5, 3, 4]
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Teleportationprobabilities
N
M
K
X
M
P(T) = 0.03
P(T) = 0.03
P(T) = 0.03
P(T) = 0.03
P(T) = 0.03
P(T) = 0.15 / # of pagesP(T) = 0.03
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank: Simplified Mathematical Def’ncause that’s how we roll
𝐿 = 𝑇 =
0.15𝑁
⋮0.15
𝑁
Assume the web is N pages bigAssume that probability of teleportation (t) is 0.15, and following link (s) is 0.85Assume that teleportation probability (E) is uniformAssume that you start on any random page (uniform distribution L), then
Then after one step, the probability your on page X is:
𝐿 ∗ 𝑠𝐺 + 𝑡𝐸
𝐿 ∗ (0.85 ∗ 𝐺 + 0.15 ∗ 𝐸)
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
G = The Link Graphginormous and sparse
1 2 … … N
1 1 0 … … 0
2 0 1 … … 1
… … … … … …
… … … … … …
N 0 1 … … 1
Link Graph No link from 1 to N
Huge!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
G as a dictionarymore compact…
{
"1" => [25, 26],
"2" => [1],
"5" => [123,2],
"6" => [67, 1]
}
Page
Links to…
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Computing PageRankthe tedious way
Follow link from page he/she is currently on.
Teleport to a random location on the web.
Page K
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Computing PageRankin one swoop
Identity matrix
Don’t trust me! Verify it yourself!
𝑞 = 𝑡 𝐼 − 𝑠𝐺 −1𝐸 = 𝑃1
⋮𝑃𝑛
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Enough hand-waving, dammit!show me the code
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Birth of EM-Proxyflash of the obvious
Hot, Fast, Awesome
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Hot, Fast, Awesome
http://rb-gsl.rubyforge.org/
Click there! … Give yourself a weekend.
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Click there! … Give yourself a weekend. http://ruby-gsl.sourceforge.net/
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank in Ruby6 lines, or less
require "gsl"include GSL
# INPUT: link structure matrix (NxN)# OUTPUT: pagerank scoresdef pagerank(g)
raise if g.size1 != g.size2
i = Matrix.I(g.size1) # identity matrixp = (1.0/g.size1) * Matrix.ones(g.size1,1) # teleportation vector
s = 0.85 # probability of following a linkt = 1-s # probability of teleportation
t*((i-s*g).invert)*pend
Verify NxN
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank in Ruby6 lines, or less
require "gsl"include GSL
# INPUT: link structure matrix (NxN)# OUTPUT: pagerank scoresdef pagerank(g)
raise if g.size1 != g.size2
i = Matrix.I(g.size1) # identity matrixp = (1.0/g.size1) * Matrix.ones(g.size1,1) # teleportation vector
s = 0.85 # probability of following a linkt = 1-s # probability of teleportation
t*((i-s*g).invert)*pend
Constants…
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank in Ruby6 lines, or less
require "gsl"include GSL
# INPUT: link structure matrix (NxN)# OUTPUT: pagerank scoresdef pagerank(g)
raise if g.size1 != g.size2
i = Matrix.I(g.size1) # identity matrixp = (1.0/g.size1) * Matrix.ones(g.size1,1) # teleportation vector
s = 0.85 # probability of following a linkt = 1-s # probability of teleportation
t*((i-s*g).invert)*pend
PageRank!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Ex: Circular Webtesting intuition…
N
K
X P = 0.33
pagerank(Matrix[[0,0,1], [0,0,1], [1,0,0]])> [0.33, 0.33, 0.33]
P = 0.33
P = 0.33
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Ex: All roads lead to Ktesting intuition…
N
K
X P = 0.07
pagerank(Matrix[[0,0,0], [0.5,0,0], [0.5,1,1]])> [0.05, 0.07, 0.87]
P = 0.87
P = 0.05
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank + Ferretawesome search, ftw!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
require 'ferret'include Ferret
index = Index::Index.new()
index << {:title => "1", :content => "it is what it is", :pr => 0.05 }index << {:title => "2", :content => "what is it", :pr => 0.07 }index << {:title => "3", :content => "it is a banana", :pr => 0.87 }
1
3
2 P = 0.07
P = 0.87
P = 0.05
Store PageRank
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
index.search_each('content:"world"') do |id, score|puts "Score: #{score}, #{index[id][:title]} (PR: #{index[id][:pr]})"
end
puts "*" * 50
sf_pr = Search::SortField.new(:pr, :type => :float, :reverse => true)
index.search_each('content:"world"', :sort => sf_pr) do |id, score|puts "Score: #{score}, #{index[id][:title]}, (PR: #{index[id][:pr]})"
end
# Score: 0.267119228839874, 3 (PR: 0.87)# Score: 0.17807948589325, 1 (PR: 0.05)# Score: 0.17807948589325, 2 (PR: 0.07)# ***********************************# Score: 0.267119228839874, 3, (PR: 0.87)# Score: 0.17807948589325, 2, (PR: 0.07)# Score: 0.17807948589325, 1, (PR: 0.05)
TF-IDF Search
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
index.search_each('content:"world"') do |id, score|puts "Score: #{score}, #{index[id][:title]} (PR: #{index[id][:pr]})"
end
puts "*" * 50
sf_pr = Search::SortField.new(:pr, :type => :float, :reverse => true)
index.search_each('content:"world"', :sort => sf_pr) do |id, score|puts "Score: #{score}, #{index[id][:title]}, (PR: #{index[id][:pr]})"
end
# Score: 0.267119228839874, 3 (PR: 0.87)# Score: 0.17807948589325, 1 (PR: 0.05)# Score: 0.17807948589325, 2 (PR: 0.07)# ***********************************# Score: 0.267119228839874, 3, (PR: 0.87)# Score: 0.17807948589325, 2, (PR: 0.07)# Score: 0.17807948589325, 1, (PR: 0.05)
PageRank FTW!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
index.search_each('content:"world"') do |id, score|puts "Score: #{score}, #{index[id][:title]} (PR: #{index[id][:pr]})"
end
puts "*" * 50
sf_pr = Search::SortField.new(:pr, :type => :float, :reverse => true)
index.search_each('content:"world"', :sort => sf_pr) do |id, score|puts "Score: #{score}, #{index[id][:title]}, (PR: #{index[id][:pr]})"
end
# Score: 0.267119228839874, 3 (PR: 0.87)# Score: 0.17807948589325, 1 (PR: 0.05)# Score: 0.17807948589325, 2 (PR: 0.07)# ***********************************# Score: 0.267119228839874, 3, (PR: 0.87)# Score: 0.17807948589325, 2, (PR: 0.07)# Score: 0.17807948589325, 1, (PR: 0.05)
Others
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Search*: Graphs are ubiquitous!PageRank is a general purpose hammer
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank + Social GraphGitHub
Username GitCred
==============================
37signals 10.00
imbriaco 9.76
why 8.74
rails 8.56
defunkt 8.17
technoweenie 7.83
jeresig 7.60
mojombo 7.51
yui 7.34
drnic 7.34
pjhyett 6.91
wycats 6.85
dhh 6.84
http://bit.ly/3YQPU
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank + Social GraphTwitter
Hmm…
Analyze the social graph:- Filter messages by ‘TwitterRank’- Suggest users by ‘TwitterRank’- …
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank + Product GraphE-commerce
Link items purchased in same cart… Run PR on it.
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank = Powerful Hammeruse it!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Personalizationhow would you do it?
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
PageRank + Personalizationcustomize the teleportation vector
𝑇 =
0.15𝑁
⋮0.15
𝑁
Teleportation distribution doesn’t
have to be uniform!
yahoo.com is my homepage!
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Gaming PageRankfor fun and profit (I don’t endorse it)
Make pages with links!
http://bit.ly/pagerank-spam
Building Mini-Google in Ruby @igrigorik #railsconfhttp://bit.ly/railsconf-pagerank
Questions?
The slides… Twitter My blog
Slides: http://bit.ly/railsconf-pagerank
Ferret: http://bit.ly/ferretRB-GSL: http://bit.ly/rb-gsl
PageRank on Wikipedia: http://bit.ly/wp-pagerankGaming PageRank: http://bit.ly/pagerank-spam
Michael Nielsen’s lectures on PageRank:http://michaelnielsen.org/blog