aquaint testbed john aberdeen, john burger, conrad chang, scott mardis the mitre corporation ©...
TRANSCRIPT
AQUAINT Testbed
John Aberdeen, John Burger,Conrad Chang, Scott Mardis
The MITRE Corporation
© 2002, The MITRE Corporation
Testbed Activities• Testbed
– Provide access to Q&A systems on classified data.– Solicit user feedback and conduct user studies.
• Testweb– Provide public access to a broad variety of Q&A capabilities.– Evaluate systems and architectures for inclusion in testbed.
• User Studies– Obtain feedback from users on utility and usability
• Architecture support
Testbed
• Classified network (ICTESTNET)– access to users, data, scenarios will be restricted
• Evaluate systems prior to installation– Testweb is a good first step – MITRE installations are more than rehearsal
Testbed Progress
• MITRE installations– QUIRK (CYCorp/IBM) – Javelin (CMU)– LCC
• Assessing– Portability to the IC environment– Maintainability– Features– Resources– Configuration
Deployment Issues
• Restrictive version requirements• Fixed pathname/hostname issues• Configure scripts
• Remote debugging (installation, configuration) is inefficient
• Differences in machine configuration, environment• Unknown or unrealized dependencies• Schedule conflicts
• Firewalls and other security issues
Developing a Testbed Vision
• USNR Analyst Task• MITRE gains first-hand knowledge about:
– analyst’s working environment• tools• methods of analysis
– nature of analyst’s tasking– long-term analysis goals
• MITRE obtains expert feedback about:– suitability of QA tools for analysis tasks– representativeness of CNS scenarios– methodology for further tool assessment– methodology for future user studies
Plan for USNR Visit• Discussions of analyst's history with analysis tasks• Hands-on presentation of QA and Search tools• MITRE and USNR analyst jointly work through 1 to 2 CNS
scenarios using QA tools, search tools, telephone, etc.• Analyst formally works through 1 to 3 more CNS
scenarios on their own– MITRE monitors the activity through logging and survey
instruments– The analyst provides feedback so adjustments can be provided as
necessary
• The analyst and MITRE will jointly analyze the results• MITRE and the analyst develop a draft test plan
Testweb
Q/Asystem
CNS
TREC2002
Javelin, LCC,Qanda, TextMap
Q/APortal/Demo
Q/A repository
Google API
Othercollections
IRservice
Clarityservice
UsersQ/APortal/Demo
•Loose federation of systems
•Simple services over TCP/HTTP/SOAP
Q&A Demo
CNS Data
http://aquaint.mitre.org/cns
UMASS Clarity
• Many thanks to Steve Cronen-Townsend & Bruce Croft• Clarity measures …• Clarity service
– SOAP interface– Access via Google Java client – Data sets: AQUAINT CNS, TREC-11 (soon)
• User interface– For demonstration, simple output of clarity score– To come, low clarity warning
CNS Search w/ UMASS Clarity
http://aquaint.mitre.org/clarity
IR Service
• Utilizing Google SOAP API• Tied SOAP to Lucene search• Data sets: AQUAINT CNS, TREC-11• Access via Google Java client
Q/A Architecture
Meeting: April 22, 2003 @ IBM in Hawthorne, NY
Eric Nyberg (CMU), David Ferrucci & Chris Welty (IBM), Scott Mardis & John Burger (MITRE)
Topics:– Javelin– PIQUANT– Catalyst– UIMA– AQUAINT Testbed
Some Directions
• Architecture is important– Enable collaboration– May be task dependent
• Common Components/Subsystems– Question analysis– Query formulators– IR– Taggers – Knowledge sources
MITRE’s Mission on Architecture
• Encourage collaborations– Promote regularization of interfaces
• Facilitate broader use of best components
• Advocate for users (IC analysts) and user tasks
• Coordinate with Ex. Comm.
MITRE’s Roles on Architecture
Services
CommunitySupport
APIs, Standards
InfrastructureResources
Data Resources
MITRE’s Roles on Architecture
• User logs, including Q&A, simple search, documents viewed, etc.
• Extended WN & XWN APIs
• Query formulation API• Corpora URIs
Services,Libraries
APIs,Standards
InfrastructureResources
Data Resources
APIs, Standards:CommunitySupport
MITRE’s Roles on Architecture
• IR• Clarity• Unit conversions ?• Time normalization ?
CommunitySupport
InfrastructureResources
Data Resources
Services, Libraries:
APIs,Standards
Services,Libraries
MITRE’s Roles on Architecture
• Modified CNS corpus• Ontology library ?
CommunitySupport
InfrastructureResources
Data Resources:
APIs,Standards
Services,Libraries
DataResources
MITRE’s Roles on Architecture
• Middleware tools ?
Infrastructure Resources:
APIs,Standards
Services,Libraries
DataResources
CommunitySupport
InfrastructureResources
MITRE’s Roles on Architecture
• Tool website (index)• Wiki for architecture ?• Mailing lists
InfrastructureResources
Community Support:
APIs,Standards
Services,Libraries
CommunitySupport
DataResources
Component Offerings
• LCC– Ontology tools (development & browsing)– Extended WordNet (XWN)
• ISI– Question analysis (QTarget)– Question paraphraser– Query formulator– Web-based IR– Answer identifier (extractor)
Component Offerings (cont’d)
• IBM– NE tagger (Resporator – 80 types)
• CyCorp– Answer sanity checking
– Taxonomy (well-developed for CNS/WMD)– Unit conversion– Cyc concept paraphrases
• UMASS– Clarity
Component Needs ?
• IR subsystems• User context & history representation• More Ontologies & Tools• Answer combination• Entity normalization (time, location, organization)• Structured data services (DBs)• Other media
Many Faces of AQUAINT• Leverage all of AQUAINT’s successes• Expose a greater range of interfaces to analysts
– They may find something they can use now– Make the Q/A system more transparent
• Exploiting the middle ground– Queries that are more complex than boolean search, but more
structured than free natural language questions