technology for automated assessment: the world-wide-mind
DESCRIPTION
Technology for automated assessment: The World-Wide-Mind. Ciarán O’Leary Dublin Institute of Technology 22 nd May 2003. Quick Overview. Requirements Large amounts of software online Large number of participants in project Diversity is key! Need to - PowerPoint PPT PresentationTRANSCRIPT
Technology for automated assessment:
The World-Wide-Mind
Ciarán O’Leary
Dublin Institute of Technology
22nd May 2003
Quick Overview
• Requirements– Large amounts of software online– Large number of participants in project– Diversity is key!
• Need to– Provide architecture and protocol that will
make this software available– Have a simple entry level – like the WWW
e-Learning Overview
• Reusability of components
• Potential for automated assessment
• Potential for comparisons with other students
Agenda
• World-Wide-Mind– Introduction to goal of project– Definition of a service– Available software
• Blocks World example• Usage in learning environment• Comparison with existing models • Possibilities for future development
Part I
World-Wide-Mind
• Dublin City University– Dr. Mark Humphrys
• Artificial Intelligence Project– Distributed– Artificial Minds
The Jolly Little Creature
Agent
The Jolly Little Creature’s Mind
• Multiple (massively diverse) modules• Since, according to faculty psychologists, the
mental causation of behaviour typically involves the simultaneous activity of a variety of distinct psychological mechanisms, the best research strategy would seem to be divide and conquer: first study the intrinsic characteristics of each of the presumed faculties, then study the ways in which they interact– Jerry Fodor – The Modularity of Mind, p1.
Building JLC’s Mind
• Divide and conquer• Required diversity through multiple
authors– The more authors the more diversity
• Make modules available online– Researchers– Amateurs– Students!
Building JLC’s Mind
• Requirements– Architecture – Standard/protocol to use to integrate
components
• Requirement for protocol– No barriers to entry– Simple– Not tied to any platform
Summary Part I
• World-Wide-Mind• Distributed Artificial Intelligent Project• Success dependent on participation of large
number of very different researchers– Require diversity
• Need an architecture and protocol for components to be plugged together
• Major goal is simplicity
Part II
JLC’s Mind
Mind
Perceive state
Select action
Body
State
Take action
World
State
New state
JLC’s Mind Online
Mind
Perceive state
Select action
Body
State
Take action
World
State
New state
Internet
World-Wide-Mind Architecture
World Server
World Server
Mind ServerMind Server
ClientClient
World-Wide-Mind Architecture
World Server
World Server
World Server
World Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
ClientClient
Service Methods
World Service
• newrun• getstate• takeaction• endrun
Mind Service
• newrun• getaction• endrun
World-Wide-Mind Protocol
• “The term Web services describes a standardized way of integrating Web-based applications using the XML, SOAP, WSDL and UDDI open standards over an Internet protocol backbone. XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available”– http://www.webopaedia.com
• Society of Mind Markup Language (SOML)– Lightweight Web Services
SOML Message
1. <soml version="0.9">2. <request type="getaction" runid="123456">3. <param name="state">4. (0, 0, 0, 9, 7, 8, 2, 0, 4)5. </param>6. <argument name="usemem" value=“true”/>7. </request>8. </soml>
Web Based Client – Java Applet
Web Based Client – Java Applet
Constructing complex minds, online
World Server
World Server
MindM Server
MindM Server
Mind ServerMind Server
Mind ServerMind Server
ClientClient
Who builds the sub-minds?
• Everyone– Professional researchers– Students– Amateurs– Anyone interested in AI!
Who builds the parent minds
• Everyone– Anyone can put software online– Anyone can put a world online– Anyone can use my mind as part of their mind– Best minds are the ones that will be re-used
How does this make the AI problem easier?
• Society A solves problem A
• Society B solves problem B
• Problem A + B is solved by writing parent mind to arbitrate between A and B’s parent minds
Society of Mind
World AWorld A
MindM AMindM A
MindMind MindMind MindMind
World BWorld B
MindM BMindM B
MindMind MindMind MindMind
MindM A+BMindM A+B
World A+BWorld A+BClientClient
Summary Part II
• Distribute the mind on the Internet
• Use server-centric architecture
• Use existing HTTP protocol
• Use simple XML-like messages
• Simple entry level
• Potential for provision of large numbers of components required for project
Part III
World-Wide-Mind in learning environment
• 10 Students presented with an online problem
• Have to interact with the online problem using client software
• Need to create “mind” or solution
First Implementation
StartD
CF
EBA
GoalF
E
D
C
B
A
• Blocks World
Blocks World Service
• Action
empty(1)&ontable(A,2)&clear(A)&ontable(B,3)&on(F,B)&on(D,F)&clear(D)&ontable(E,4)&on(C,E)&clear(C)&empty(5)&gripper()
• State
grip(A)
ungrip(3)
Blocks World Run
Blocks World Server
Blocks World Server
Mind ServerMind Server
ClientClient
getstatetakeactiontakeactiontakeactiontakeaction
getactiongetactiongetactiongetactiongetaction
endrunnewrun
endrunnewrun
Blocks World Service
• Supports– newrun
• Provides runid– getstate
• Provides state in form shown
– takeaction• Receives action sent by client• Returns updated state following execution of action
– endrun• Ends run, clears up server
• All messages carry error code information
Building a blocks world “mind”
• Create service that supports– newrun
• Provides runid• Can also provide startup-parameters
– getaction• Receives state from client (which had got it from world
service)• Provides state in form shown
– endrun• Ends run, clears up server
• All messages should carry error code information
Scoreboard
• Scores of best performing minds recorded in a list on the “world” site– Score is the number of moves needed to stack blocks
• Any mind that interacts with world will have its score recorded and displayed
• Others can then run that mind in the world• Performance based assessment
– Performance of algorithm can be judged from score board
Technical Requirements
• Web Server space• One of…
– Java Servlets– ASP– JSP– PHP– CGI – C++/Perl/any language
• To write own client (or to integrate another mind), need to know how to connect to URL– Simple code in most languages
Connect to URL (Java)
String getResponseFromService(String serviceURL, String request) {
URLConnection connection = new URL(serviceURL).openConnection();
PrintWriter outs = new PrintWriter(connection.getOutputStream());
outs.println(request); // SOML message
outs.close();
StringBuffer returnedText = new StringBuffer();
BufferedReader ins = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = "";
while((line = ins.readLine()) != null) {
returnedText.append(line);
}
return returnedText.toString();
}
Connect to URL (Perl)
sub getResponseFromService() {
$serviceURL = @_[0];
$request = @_[0];
new LWP::UserAgent->request(
POST “$serviceURL”,
Content_Type => 'form-data',
Content => [ q(q) => "$request"])->content;
}
Summary Part III
• Simple blocks world problem online using protocol/architecture developed for World-Wide-Mind
• Software still available online for others to interact with
• Client software also available
• Students put minds online– Minimal knowledge of network programming required
– Only really need to read from stdin and write to stdout.
• Scoreboard records best performing algorithms/minds
Part IV
Comments on Architecture
• Lightweight web services– Low entry level
• Protocol simple
• Technical requirements minimal
– Software continually available– Problem can be re-used remotely by others
• At the very least it can be tested remotely
Comments on Architecture
• Comparison with World-Wide-Web– Students put project work (HTML pages/PDF)
online (DCU/DIT)– Others can review their project work– Very simple entry level
• Before there were tools (MS Front Page) there was simple (terrible) HTML
Reusability
• Why don’t we use other peoples software?– Platform issues– Programming language isues
• Its in our favour for others to make software available• What is their incentive?
– If you’re putting it online anyway…
• Learning objects– SCORM
• PEARL• TOIA (IMS QTI compatible tools)
Summary Part IV
• We need to re-use other peoples software
• Local installation has many issues
• Complex standards require tools
• Low entry level
Part V
Future Development
• Existing standard– Portal site will provide
• Sample code
• Instructions on creating services
• Services
• Client software
Future Development
• Possible– Integration at higher levels with other standards
• SOAP
• XML
• WSDL
• DAML
– e-Learning standards• PEARL uses HTTP
Questions?