EMBODIED AGENTS

IN AUGMENTED & VIRTUAL REALITIES

Course E6998-004, Dept. of Computer Science, Columbia University, Fall 2002

Prof. Kris R. Thórisson, Ph.D.


	LECTURE NOTES LECTURE 5 PART 1 Architecture & Integration October 31, 2002

PART 2

1	Concepts
	Why do we need architectures?
	Ymir: quick summary + Bird's-eye view
	Blackboards
	Trace examples

2	Why Do We Need An Architecture?
	A.I. (and multimodal communicative systems design) is about complexity management Resource management Real-time control
	Most psycological and A.I. research has worked on isolated parts of the psyche natural language emotion planning logic spatial knowledge & action propositional knowledge etc.
	We need a coherent architectural plan to make use of these isolated pieces of information

3	Architectural Goals
	Holistic architectures can provide communicative interfaces that exploit ... Synergy Naturalness Flexibility
	System Characteristics Wanted: Communication Natural Language Manual Gesture Facial Expression Body Language Flexible Turn Taking Dialogue Knowledge Topic Knowledge Space Navigation Manipulation Planning Task Planning Execution Integration

4	Ymir Approach
	General -- not "vertical", special-function
	Integration -- not divide-and-conquer
	Use Existing Solutions -- create new where needed
	"Expert System/Engineering Approach" -- create system by hand
	Use blackboards extensively

5	Origin of Blackboards
	Oliver Selfridge - 1959: "Daemons" paper
	HEARSAY-II speech recognition, CMU - 1980
	Widely used in AI since

6	Ymir
	Three Blackboards: Functional Sketchboard Content Blackboard Motor Feedback Blackboard

7	Functional Sketchboard


	"Functional Sketches" of user actions: “Functional” because we need to figure out the purpose of what the user is doing so we can react properly in real-time E.g.: The user makes a gesture. If the gesture is deictic, we need to look in the direction that she’s pointing. The function of the gesture is to direct our attention in a certain spatial way; however, we may not know what’s there to be seen, it may not even be a deictic gesture - hence “sketches”
	The FS is based on the hypothesis that a functional analysis is a precursor to (higher-priority than) multimodal interpretation and real-time multimodal action generation

8	Content Blackboard


	Communication between the Process Control Layer (PCL) and the Content Layer (CL) This is where process control and content interpretation and generation meet
	Processes in the PCL post messages about their own activities and read messages about activities in the CL
	Processes in the CL post messages about their own activities and read messages about what’s happening in the PCL

9	Motor Feedback Blackboard


	Communication between Action Scheduler and Content Layer (CL) Process Control Layer (PCL)
	Data: status of ongoing motor actions executing in the AS The CL, and to some extent the PCL, contain modules that can make decisions about the progress The Reactive Layer handles feedback loops that are too short to take any advantage of this feedback

10	Blackboard Benefits
	Design Easier to build new system functionalities Easier to handle backward compatibility Easier to create traces of system operation Localizes debugging A natural fit to modular design Multiple BBs: Enables hierarchies of APIs Better support for multiple developers in a team
	Efficient method to deal with ... data represented at multiple levels of detail system complexity asynchronous processing
	Isolation of communicative data and data processing
	Supports incremental, opportunistic processing
	Better forward migration support

11	Gandalf
	26 Perceptual Modules
	35 Decision Modules
	83 Behavior Modules
	1 1/2 Knowledge Base

	Design Basis - Speech & Dialogue Examples: [Searle ‘69, Goodwin ‘81, Grosz & Sidner ‘86, Grice ‘89, Pierrehumbert & Hirschberg ‘90] - Face-to-Face Interaction Examples: [Duncan ‘72, Goodwin ‘81, Clark ‘92] - Gaze Examples: [Kahneman ‘73, Cooper ‘74, Kleinke ‘86] - Gesture Examples: [ Effron ‘41, Ekman & Friesen ‘78, Kendon ‘80, Goodwin ‘86, McNeill ‘92]

12	Ymir BB Trace Example 1
	This example shows the series of system events that happen during and after the user's multimodal communicative act "Delete that box [deictic gesture]"

13	Ymir BB Trace Example 2
	This graph shows internal states of Gandalf during interaction over a 16 second interval (each vertical line marks a second). The highly rythmic gaze pattern observed at the top of the graph indicates a bad eye calibration. Notice that modules 1 through 34 all relate to the user’s behavior

14	Blackboard Implementations
	Most are based on event-triggers
	Ymir blackboards: 'passive' - modules (agents) determined when to access, what to retrieve, etc.
	More specialized implementations include BBs for planning, with three-part storage: Plans, planning data, planning heuristics

15	Other Architectures
	Subsumption architecture [Brooks] Spreading Activation [Maes] Augmented finite state machines Non-hierarchical: Considers all behaviors potentially relevant at all times Extending them grows system complexity exponentially Using blackboards - simplifies?
	POSH (parallel-rooted, ordered, slip-stack, hierarchical) reactive plans [Bryson] Use a stack to store plans and sub-plans Can be implemented in a blackboard framework May simplify expansion of plans Significantly simplifies addtion of supporting and/or unrelated knowledge sources and processes (e.g. emotion systems and NL processing)

PART 2 ->

2002©K.R.Thórisson