introduction to grid computing felix hageloh roberto valenti deployment of a language detector grid...
TRANSCRIPT
![Page 1: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/1.jpg)
Introduction to Grid Introduction to Grid ComputingComputing
Felix HagelohFelix HagelohRoberto Roberto ValentiValenti
Deployment of a Language Detector Deployment of a Language Detector Grid ServiceGrid Service
University of Amsterdam, 02-11-University of Amsterdam, 02-11-20052005
![Page 2: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/2.jpg)
OverviewOverview IntroductionIntroduction Required StepsRequired Steps Our Service Our Service
IntroductionIntroduction The basic ideaThe basic idea Use CaseUse Case InterfaceInterface ImplementationImplementation
Problems EncounteredProblems Encountered Future WorkFuture Work ConclusionsConclusions QuestionsQuestions
![Page 3: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/3.jpg)
IntroductionIntroduction
Our chosen task: Grid ServicesOur chosen task: Grid Services
Task Goals:Task Goals: Build a grid service. Build a grid service. Aggregate the service with another to Aggregate the service with another to
provide additional, higher-level provide additional, higher-level servicesservices
![Page 4: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/4.jpg)
StepsSteps Get access to the systemsGet access to the systems
AuthenticationAuthentication Security issuesSecurity issues
Obtain User CertificateObtain User Certificate Obtain Host CertificateObtain Host Certificate
Implement the serviceImplement the service Create required filesCreate required files
WSDLWSDL QNamesQNames WSDDWSDD JNDIJNDI
Compile and create GAR fileCompile and create GAR file As Globus user:As Globus user:
Deploy service Deploy service Start containerStart container
![Page 5: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/5.jpg)
But you all know this…But you all know this…
So… we jump to our service.So… we jump to our service.
![Page 6: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/6.jpg)
Our ServiceOur Service
![Page 7: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/7.jpg)
Our Service: IntroductionOur Service: Introduction
We were requested to implement a We were requested to implement a useful service which could be useful service which could be integrated on other servicesintegrated on other services
We are AI students so…We are AI students so…
Let’s Merge AI and Grid Computing!!Let’s Merge AI and Grid Computing!!
![Page 8: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/8.jpg)
Our Service: The Basic IdeaOur Service: The Basic Idea
Idea: Language Detection Is a Idea: Language Detection Is a necessary first step in a multitude of necessary first step in a multitude of applicationsapplications
Useful Web Service Examples:Useful Web Service Examples: Email filteringEmail filtering Information retrievalInformation retrieval Spell checkersSpell checkers
Can also be component of an Can also be component of an aggregated grid serviceaggregated grid service
![Page 9: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/9.jpg)
Our Service: The Basic IdeaOur Service: The Basic Idea
What about creating a Language What about creating a Language Detector on the Grid?Detector on the Grid? Training and Testing can be extremely Training and Testing can be extremely
time consuming running on a single time consuming running on a single machinemachine
Data difficult to obtain -> can be Data difficult to obtain -> can be shared on the Gridshared on the Grid Duplicate data for parallel computingDuplicate data for parallel computing
![Page 10: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/10.jpg)
Our Service: Use CaseOur Service: Use Case
Simple Interface:Simple Interface: Receives a Receives a
piece of textpiece of text Returns a string Returns a string
indicating the indicating the languagelanguage
![Page 11: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/11.jpg)
Our Service: Adding StatesOur Service: Adding States
Grid services can have states (as Grid services can have states (as opposed to web services)opposed to web services)
Not necessary for our service but Not necessary for our service but for the learning factorfor the learning factor
Added “dummy” states to our Added “dummy” states to our service:service: Last OperationLast Operation Times UsedTimes Used
![Page 12: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/12.jpg)
Our Service: Statefull Use Our Service: Statefull Use CaseCase
![Page 13: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/13.jpg)
Our Service: InterfaceOur Service: Interface
Requests and ResponsesRequests and Responses
<xsd:element name="detect" type="xsd:string" /> <xsd:element name="detectResponse“ type="xsd:string“ /> <xsd:element name="getLanguageRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getLanguageRPResponse" type="xsd:string" /> <xsd:element name="getLastOpRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getLastOpRPResponse" type="xsd:string" /> <xsd:element name="getTimesUsedRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getTimesUsedRPResponse" type="xsd:int" />
![Page 14: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/14.jpg)
Our Service: InterfaceOur Service: Interface
Port TypesPort Types<portType name="LanguageDetectorPortType" … > <operation name="detect"> <input message="tns:DetectInputMessage" /> <output message="tns:DetectOutputMessage" /> </operation> <operation name="getLanguageRP"> <input message="tns:GetLanguageRPInputMessage" /> <output message="tns:GetLanguageRPOutputMessage" /> </operation> <operation name="getLastOpRP"> <input message="tns:GetLastOpRPInputMessage" /> <output message="tns:GetLastOpRPOutputMessage" /> </operation> <operation name="getTimesUsedRP"> <input message="tns:GetTimesUsedRPInputMessage" /> <output message="tns:GetTimesUsedRPOutputMessage" /> </operation></portType>
![Page 15: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/15.jpg)
Our ServiceOur ServiceImplementationImplementation
![Page 16: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/16.jpg)
Language Detection: Basic Language Detection: Basic IdeaIdea
Essentially based on probabilities of Essentially based on probabilities of character combinationscharacter combinations
Every language has typical character Every language has typical character combinations that are very frequent in combinations that are very frequent in that languagethat language ““th” in englishth” in english ““ij” in dutchij” in dutch
Easy for humans to detect a language Easy for humans to detect a language even when we don’t know that specific even when we don’t know that specific languagelanguage
![Page 17: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/17.jpg)
Language Learning: Standard Language Learning: Standard ProcessProcess
Standard machine learning processStandard machine learning process
![Page 18: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/18.jpg)
Language Learning: Markov Language Learning: Markov ModelsModels
Basic Markov ModelBasic Markov Model
kkthth order Markov Model order Markov Model
n
iiinssssssS
21121
|PP,...,,PP
n
iikiiknssssssssS
21121
,...,|P,...,P,...,,PP
![Page 19: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/19.jpg)
Language Detection: Language Detection: ClassificationClassification
Transitional probabilities estimated Transitional probabilities estimated asas
ClassificationClassification
21
21
21 ,#
,,#,|P
is
iii
iii ss
ssssss
)||P(max|Pmax2,1
3cii
n
ii
cCC
LsssLSC
![Page 20: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/20.jpg)
Language Detection: Language Detection: ExampleExample
The training text for a The training text for a language consists of language consists of the stringthe string
Learned model:Learned model:
test text
( ^^, t, 1.0 )( ^t, e, 1.0 )( te, s, 0.5 )( es, t, 1.0 )( st, , 1.0 )( te, x, 0.5 )( ex, t, 1.0 )( xt, , 1.0 )
the probability of the the probability of the stringstring
would be:would be:
test
P(test|L) = P(t|^^)*P(e|^t)*P(s|te) *P(t|es)*P(_|st) = 1*1*0.5*1*1=0.5
![Page 21: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/21.jpg)
Language Detection: Language Detection: PerformancePerformance
60%
70%
80%
90%
100%
number of words
% c
orr
ec
tly
cla
ss
ifie
d
German 78,91 92,91 96,84 98,15 98,81
English 73,89 90,55 95,07 97,18 98,10
Dutch 68,32 88,66 94,70 97,12 98,09
one two three four five
![Page 22: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/22.jpg)
Problems EncounteredProblems Encountered
Necessary tools had to be installed Necessary tools had to be installed (ANT)(ANT)
Problems on our machine (GRAM)Problems on our machine (GRAM) Conflicts with other teamConflicts with other team Buggy shell script to build gar fileBuggy shell script to build gar file
Sensitive to path lengths/ namesSensitive to path lengths/ names
![Page 23: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/23.jpg)
Future WorkFuture Work
Connect with other servicesConnect with other services Make training and evaluation a grid Make training and evaluation a grid
serviceservice Make it part of a multi lingual retrieval Make it part of a multi lingual retrieval
engineengine
Web interface (interactive)Web interface (interactive)
![Page 24: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/24.jpg)
ConclusionsConclusions
Successfully managed to create Successfully managed to create and deploy our own web serviceand deploy our own web service Broke loose from the tutorial web Broke loose from the tutorial web
service structureservice structure Merged Grid Computing with AIMerged Grid Computing with AI
Got hands on experience with Grid Got hands on experience with Grid applications and structureapplications and structure
A lot of possibilities to integrate A lot of possibilities to integrate and/or extend the implemented and/or extend the implemented serviceservice
![Page 25: Introduction to Grid Computing Felix Hageloh Roberto Valenti Deployment of a Language Detector Grid Service University of Amsterdam, 02-11-2005](https://reader036.vdocument.in/reader036/viewer/2022062408/56649f335503460f94c50a34/html5/thumbnails/25.jpg)
QuestionsQuestions
??