assistive context-aware toolkit - g3ictg3ict.org/design/js/tinymce/filemanager/userfiles/file/2015...
TRANSCRIPT
Assistive Context-Aware Toolkit Sai Prasad Anticipatory Computing Lab Intel Labs
Opening the Door for Help
Asking What Matters
Striving for Independence
A Holistic Solution
6
Assistive Context Aware Toolkit (ACAT)
Mute
Contextual Menus Based on app
Speak
Search Files
Assistive Context Aware Toolkit (ACAT)
Launch Applications
Main Control
ACAT Available on Open Source
Assistive Context-Aware Toolkit (ACAT)
8
http://01.org/ACAT
ACAT Architecture
Assistive Context-Aware Toolkit (ACAT)
• Applications only link with the Core and Extension libraries
• All extensions are dynamically discovered and loaded.
9
Overview
Assistive Context-Aware Toolkit (ACAT)
10
ACAT Libraries
ACAT Core Library
Assistive Context-Aware Toolkit (ACAT)
Contains all the core management components of ACAT
11
ACAT Libraries
ACAT Core Library Components
Assistive Context-Aware Toolkit (ACAT)
12
Animation Management Scans widgets on panel
Widget Management Manages widget layout and appearance
Panel Management Creates/Displays Panels
Theme Management Manages colors of Panels
ACAT Libraries
Word Prediction Management Manages word predictors* for next-word prediction
*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)
ACAT Core Library Components (contd.)
Assistive Context-Aware Toolkit (ACAT)
13
Hello world
Text-to-Speech Management Converts text to speech
Talk Window Management Creates, manages the Talk window
ACAT Libraries
ACAT Core Library Components (contd.)
Assistive Context-Aware Toolkit (ACAT)
14
Agent Management Manages Application Agents which provides contextual information from applications
One Application Agent per application
• E.g. Notepad Application Agent, MS Word Application Agent
ACAT Libraries
ACAT Core Library Components (contd.)
Assistive Context-Aware Toolkit (ACAT) 15
Actuator Management Manages Actuators and handles trigger events
Proximity Sensor Assistive Switch Camera
Actuators
ACAT Libraries
ACAT Extension Library
Assistive Context-Aware Toolkit (ACAT)
Contains base classes for Application Agents and Utility classes
16
ACAT Libraries
Assistive Context-Aware Toolkit (ACAT)
17
Extensions
ACAT Extensions
Assistive Context-Aware Toolkit (ACAT)
Extensions are plug-in’s that extend the capability of ACAT
18
Extensions
Extensions Discovery and Instantiation
Assistive Context-Aware Toolkit (ACAT)
ACAT Core loads Extensions on startup.
The manager for each Extension category traverses the depth of the top-level folder for that category.
1. Discovers all DLL’s under the folder
2. Enumerates all the .NET classes that implement the interface for the category.
- E.g. IWordPredictor for all Word Predictor extensions.
3. Caches the .NET Types of the extension classes
4. Uses .NET Reflection to create instances of the classes from the Types
19
Extensions
Assistive Context-Aware Toolkit (ACAT)
20
Extension
Development
Requirements
Assistive Context-Aware Toolkit (ACAT)
• Windows 7 or higher
• Visual Studio 2012 or higher • Code written entirely in C#
• Microsoft .NET Framework 4.5
• Microsoft Office Interop Assemblies • Bundled with Visual Studio Professional/Premium/Ultimate
• Optional Visual Studio Add-ons • StyleCop (free) to enforce .NET coding standards
• CodeMaid (free) to cleanup/reorganize the code
• ReSharper (paid) to optimize the code
21
Extension Dev.
Creating an Extension
Assistive Context-Aware Toolkit (ACAT)
1. Create a Visual Studio DLL project
2. Add references to ACATCore.dll and ACATExtension.dll.
3. Implement IExtension and the interface for the Extension category, e.g., IWordPredictor
4. Compile the DLL.
5. Deploy the DLL to the folder for the Extension category. E.g. Deploy a Word Predictor extension to <RunDir>\Extensions\Default\WordPredictors\MyWordPredictor\MyExtension.dll
6. Modify relevant ACAT XML files to configure the Extension.
• Depends on the Extension category
22
Extension Dev.
Extension Invoker
Assistive Context-Aware Toolkit (ACAT)
• Extension DLL’s are not directly referenced in any Project.
• Access methods/properties in the Extension objects through the Extension category Interface. E.g. IWordPredictor.Predict()
• To access custom methods/ properties
- Use .NET through the ExtensionInvoker helper class.
- Implement the IExtension interface.
- The GetInvoker() method returns an ExtensionInvoker object.
- Use this object to access custom methods/properties
- E.g. To get the name of the file selected from the FileBrowser extension through the SelectedFile custom property:
var file = fileBrowser.GetInvoker().GetStringValue("SelectedFile");
23
Extension Dev.
Assistive Context-Aware Toolkit (ACAT)
24
UI
Extensions
Overview
Assistive Context-Aware Toolkit (ACAT)
Three types of Panels
25
Scanners Menus Dialogs
UI
Examples:
Alphabet Scanner Cursor Scanner Punctuations Scanner . . .
Examples:
Contextual Menus ACAT Main Menu ACAT Settings Menu . . .
Examples:
General Settings Dialog Voice Settings Dialog Scanner Settings Dialog . . .
Panels
Assistive Context-Aware Toolkit (ACAT)
• All Panels are .NET Forms.
• Visual look-and-feel of the Panels, the animation sequences, the transition are all configured through a Panel Configuration file.
• Enables easy customization of the Panels without having to change the code
• Forms can be reused for different Panel configurations.
26
UI
Panel Configuration File
Assistive Context-Aware Toolkit (ACAT)
27
WIDGET ATTRIBUTES SECTION - Configure attributes such as the text, fonts,
colors
<WidgetAttributes>
<WidgetAttribute name="B1"
label="("
value="@CmdGoBack"
fontname="ACAT Icon"
fontsize="16"/>
<WidgetAttribute name="B2"
label="-"
value="-"
...
WIDGET LAYOUT SECTION
- Define widget hierarchy - Logically group widgets into boxes, rows
and columns
<Layout>
<Widget class="RowWidget" name="Row1">
<Widget class="ScannerButton" name="B1"/>
<Widget class="ScannerButton" name="B2"/>
...
...
<Widget class="ScannerButton" name="B7"/>
</Widget>
<Widget class="RowWidget" name="Row2">
...
...
</Widget>
</Layout>
ANIMATIONS SECTION
- Define scanning sequences - Transition between scanning sequences - Actions to initiate when the user activates a
widget
<Animations>
<Animation name="TopLevelRotation"
start="true"
iterations="@HalfScanIterations">
<Widget name="Row1“ onSelect="transition(Row1)"/>
<Widget name="Row2“ onSelect="transition(Row2)"/>
</Animation>
<Animation name="Row1"
iterations="@ColumnScanIterations"
onEnd="transition(TopLevelRotation)">
<Widget name="B1"/>
...
...
<Widget name="B14"/>
</Animation>
</Animations>
UI
Assistive Context-Aware Toolkit (ACAT)
28
Word Predictor
Extensions
Overview
Assistive Context-Aware Toolkit (ACAT)
• Auto-complete and next-word prediction speed up typing. - ACAT sends the text from the current sentence to the Word Predictor
- Word predictor uses this to make smart next-word prediction.
- ACAT displays a list of suggested words returned by the Word Predictor
- User selects words from the list.
• On average, < 20% of characters need to be typed.
29
Word Predictors
Creating a Word Predictor Extension
Assistive Context-Aware Toolkit (ACAT)
30
1. Create a Visual Studio DLL project, say, MyWordPredictor
2. Add references to ACATCore.dll and ACATExtension.dll.
3. Add a class MyWordPredictor derived from IWordPredictor
4. Implement these functions in MyWordPredictor:
- Init() to initialize the Word Predictor
- Predict() to return a list of suggested words. The text from the current sentence is passed to this function
- Learn()to enhance the Word Prediction model by learning the user’s writing style
- Dispose() to dispose the Word Predictor objects
5. Build and deploy MyWordPredictor.dll to:
<RunDir>\Extensions\Default\WordPredictors\MyWordPredictor
Word Predictors
Assistive Context-Aware Toolkit (ACAT)
31
Actuator
Extensions
Actuators Overview
Assistive Context-Aware Toolkit (ACAT)
• Actuators are used to drive the UI.
• Triggers from Actuators are translated into actions.
- E.g. Typing a letter, choosing a menu item.
• Each Actuators is an ACAT Extension.
• Actuators interface with the underlying hardware. For e.g.
- USB for HID Devices or Webcams
• ACAT has helper classes to interface with
- USB HID Devices
- TCP/IP Servers/Clients
• Support included for facial gesture recognition, proximity sensor, accelerometer ring
32
Actuators
Actuator Switches
Assistive Context-Aware Toolkit (ACAT)
• Each Actuator is composed of Switches.
- E.g. Left and right mouse buttons are two switches on a mouse
- For gesture input, each gesture can be treated as a Switch
• Events are associated with Switches
- Switch Down, analogous to Mouse LButtonDown
- Switch Up, analogous to Mouse LButtonUp
- Switch Trigger, analogous to Mouse LButtonClick
• Commands can be mapped with Switch trigger events.
- Commands executed when the trigger event occurs
- Command Examples: Select highlighted UI element, Delete previous word, select first word from prediction list
- Enables multi-modal interactions by mapping gestures to commands.
• Commands mapped through XML configuration files.
33
Actuators
Creating an Actuator Extension
Assistive Context-Aware Toolkit (ACAT)
1. Create a Visual Studio DLL project, say, MyActuator
2. Add references to ACATCore.dll and ACATExtension.dll.
3. Add a class MyActuator derived from ActuatorBase
4. Add a class MyActuatorSwitch derived from ActuatorSwitchBase
5. Implement these functions in MyActuator: - Init() to initialize the Actuator
- CreateSwitch() to return a MyActuatorSwitch object - OnSwitchActivated(), OnSwitchDeactivated(),
OnSwitchTriggered()to trigger Actuator switch events. - Dispose() to dispose the Actuator objects
6. Build and deploy MyActuator.dll to: <RunDir>\Extensions\Default\Actuators\MyActuator
34
Actuators
Proximity Sensor Actuator
Assistive Context-Aware Toolkit (ACAT)
35
RealSense Actuator
Assistive Context-Aware Toolkit (ACAT)
36