technology for automated assessment: the world-wide-mind
DESCRIPTION
Technology for automated assessment: The World-Wide-Mind. Ciarán O’Leary Dublin Institute of Technology 22 nd May 2003. Quick Overview. Requirements Large amounts of software online Large number of participants in project Diversity is key! Need to - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/1.jpg)
Technology for automated assessment:
The World-Wide-Mind
Ciarán O’Leary
Dublin Institute of Technology
22nd May 2003
![Page 2: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/2.jpg)
Quick Overview
• Requirements– Large amounts of software online– Large number of participants in project– Diversity is key!
• Need to– Provide architecture and protocol that will
make this software available– Have a simple entry level – like the WWW
![Page 3: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/3.jpg)
e-Learning Overview
• Reusability of components
• Potential for automated assessment
• Potential for comparisons with other students
![Page 4: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/4.jpg)
Agenda
• World-Wide-Mind– Introduction to goal of project– Definition of a service– Available software
• Blocks World example• Usage in learning environment• Comparison with existing models • Possibilities for future development
![Page 5: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/5.jpg)
Part I
![Page 6: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/6.jpg)
World-Wide-Mind
• Dublin City University– Dr. Mark Humphrys
• Artificial Intelligence Project– Distributed– Artificial Minds
![Page 7: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/7.jpg)
The Jolly Little Creature
Agent
![Page 8: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/8.jpg)
The Jolly Little Creature’s Mind
• Multiple (massively diverse) modules• Since, according to faculty psychologists, the
mental causation of behaviour typically involves the simultaneous activity of a variety of distinct psychological mechanisms, the best research strategy would seem to be divide and conquer: first study the intrinsic characteristics of each of the presumed faculties, then study the ways in which they interact– Jerry Fodor – The Modularity of Mind, p1.
![Page 9: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/9.jpg)
Building JLC’s Mind
• Divide and conquer• Required diversity through multiple
authors– The more authors the more diversity
• Make modules available online– Researchers– Amateurs– Students!
![Page 10: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/10.jpg)
Building JLC’s Mind
• Requirements– Architecture – Standard/protocol to use to integrate
components
• Requirement for protocol– No barriers to entry– Simple– Not tied to any platform
![Page 11: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/11.jpg)
Summary Part I
• World-Wide-Mind• Distributed Artificial Intelligent Project• Success dependent on participation of large
number of very different researchers– Require diversity
• Need an architecture and protocol for components to be plugged together
• Major goal is simplicity
![Page 12: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/12.jpg)
Part II
![Page 13: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/13.jpg)
JLC’s Mind
Mind
Perceive state
Select action
Body
State
Take action
World
State
New state
![Page 14: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/14.jpg)
JLC’s Mind Online
Mind
Perceive state
Select action
Body
State
Take action
World
State
New state
Internet
![Page 15: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/15.jpg)
World-Wide-Mind Architecture
World Server
World Server
Mind ServerMind Server
ClientClient
![Page 16: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/16.jpg)
World-Wide-Mind Architecture
World Server
World Server
World Server
World Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
Mind ServerMind Server
ClientClient
![Page 17: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/17.jpg)
Service Methods
World Service
• newrun• getstate• takeaction• endrun
Mind Service
• newrun• getaction• endrun
![Page 18: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/18.jpg)
World-Wide-Mind Protocol
• “The term Web services describes a standardized way of integrating Web-based applications using the XML, SOAP, WSDL and UDDI open standards over an Internet protocol backbone. XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available”– http://www.webopaedia.com
• Society of Mind Markup Language (SOML)– Lightweight Web Services
![Page 19: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/19.jpg)
SOML Message
1. <soml version="0.9">2. <request type="getaction" runid="123456">3. <param name="state">4. (0, 0, 0, 9, 7, 8, 2, 0, 4)5. </param>6. <argument name="usemem" value=“true”/>7. </request>8. </soml>
![Page 20: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/20.jpg)
Web Based Client – Java Applet
![Page 21: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/21.jpg)
Web Based Client – Java Applet
![Page 22: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/22.jpg)
Constructing complex minds, online
World Server
World Server
MindM Server
MindM Server
Mind ServerMind Server
Mind ServerMind Server
ClientClient
![Page 23: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/23.jpg)
Who builds the sub-minds?
• Everyone– Professional researchers– Students– Amateurs– Anyone interested in AI!
![Page 24: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/24.jpg)
Who builds the parent minds
• Everyone– Anyone can put software online– Anyone can put a world online– Anyone can use my mind as part of their mind– Best minds are the ones that will be re-used
![Page 25: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/25.jpg)
How does this make the AI problem easier?
• Society A solves problem A
• Society B solves problem B
• Problem A + B is solved by writing parent mind to arbitrate between A and B’s parent minds
![Page 26: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/26.jpg)
Society of Mind
World AWorld A
MindM AMindM A
MindMind MindMind MindMind
World BWorld B
MindM BMindM B
MindMind MindMind MindMind
MindM A+BMindM A+B
World A+BWorld A+BClientClient
![Page 27: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/27.jpg)
Summary Part II
• Distribute the mind on the Internet
• Use server-centric architecture
• Use existing HTTP protocol
• Use simple XML-like messages
• Simple entry level
• Potential for provision of large numbers of components required for project
![Page 28: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/28.jpg)
Part III
![Page 29: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/29.jpg)
World-Wide-Mind in learning environment
• 10 Students presented with an online problem
• Have to interact with the online problem using client software
• Need to create “mind” or solution
![Page 30: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/30.jpg)
First Implementation
StartD
CF
EBA
GoalF
E
D
C
B
A
• Blocks World
![Page 31: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/31.jpg)
Blocks World Service
• Action
empty(1)&ontable(A,2)&clear(A)&ontable(B,3)&on(F,B)&on(D,F)&clear(D)&ontable(E,4)&on(C,E)&clear(C)&empty(5)&gripper()
• State
grip(A)
ungrip(3)
![Page 32: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/32.jpg)
Blocks World Run
Blocks World Server
Blocks World Server
Mind ServerMind Server
ClientClient
getstatetakeactiontakeactiontakeactiontakeaction
getactiongetactiongetactiongetactiongetaction
endrunnewrun
endrunnewrun
![Page 33: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/33.jpg)
Blocks World Service
• Supports– newrun
• Provides runid– getstate
• Provides state in form shown
– takeaction• Receives action sent by client• Returns updated state following execution of action
– endrun• Ends run, clears up server
• All messages carry error code information
![Page 34: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/34.jpg)
Building a blocks world “mind”
• Create service that supports– newrun
• Provides runid• Can also provide startup-parameters
– getaction• Receives state from client (which had got it from world
service)• Provides state in form shown
– endrun• Ends run, clears up server
• All messages should carry error code information
![Page 35: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/35.jpg)
Scoreboard
• Scores of best performing minds recorded in a list on the “world” site– Score is the number of moves needed to stack blocks
• Any mind that interacts with world will have its score recorded and displayed
• Others can then run that mind in the world• Performance based assessment
– Performance of algorithm can be judged from score board
![Page 36: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/36.jpg)
Technical Requirements
• Web Server space• One of…
– Java Servlets– ASP– JSP– PHP– CGI – C++/Perl/any language
• To write own client (or to integrate another mind), need to know how to connect to URL– Simple code in most languages
![Page 37: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/37.jpg)
Connect to URL (Java)
String getResponseFromService(String serviceURL, String request) {
URLConnection connection = new URL(serviceURL).openConnection();
PrintWriter outs = new PrintWriter(connection.getOutputStream());
outs.println(request); // SOML message
outs.close();
StringBuffer returnedText = new StringBuffer();
BufferedReader ins = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = "";
while((line = ins.readLine()) != null) {
returnedText.append(line);
}
return returnedText.toString();
}
![Page 38: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/38.jpg)
Connect to URL (Perl)
sub getResponseFromService() {
$serviceURL = @_[0];
$request = @_[0];
new LWP::UserAgent->request(
POST “$serviceURL”,
Content_Type => 'form-data',
Content => [ q(q) => "$request"])->content;
}
![Page 39: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/39.jpg)
Summary Part III
• Simple blocks world problem online using protocol/architecture developed for World-Wide-Mind
• Software still available online for others to interact with
• Client software also available
• Students put minds online– Minimal knowledge of network programming required
– Only really need to read from stdin and write to stdout.
• Scoreboard records best performing algorithms/minds
![Page 40: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/40.jpg)
Part IV
![Page 41: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/41.jpg)
Comments on Architecture
• Lightweight web services– Low entry level
• Protocol simple
• Technical requirements minimal
– Software continually available– Problem can be re-used remotely by others
• At the very least it can be tested remotely
![Page 42: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/42.jpg)
Comments on Architecture
• Comparison with World-Wide-Web– Students put project work (HTML pages/PDF)
online (DCU/DIT)– Others can review their project work– Very simple entry level
• Before there were tools (MS Front Page) there was simple (terrible) HTML
![Page 43: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/43.jpg)
Reusability
• Why don’t we use other peoples software?– Platform issues– Programming language isues
• Its in our favour for others to make software available• What is their incentive?
– If you’re putting it online anyway…
• Learning objects– SCORM
• PEARL• TOIA (IMS QTI compatible tools)
![Page 44: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/44.jpg)
Summary Part IV
• We need to re-use other peoples software
• Local installation has many issues
• Complex standards require tools
• Low entry level
![Page 45: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/45.jpg)
Part V
![Page 46: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/46.jpg)
Future Development
• Existing standard– Portal site will provide
• Sample code
• Instructions on creating services
• Services
• Client software
![Page 47: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/47.jpg)
Future Development
• Possible– Integration at higher levels with other standards
• SOAP
• XML
• WSDL
• DAML
– e-Learning standards• PEARL uses HTTP
![Page 48: Technology for automated assessment: The World-Wide-Mind](https://reader036.vdocument.in/reader036/viewer/2022070404/56813b5c550346895da452d8/html5/thumbnails/48.jpg)
Questions?