EMBODIED AGENTS

IN AUGMENTED & VIRTUAL REALITIES

Course E6998-004, Dept. of Computer Science, Columbia University, Fall 2002
Prof. Kris R. Thórisson, Ph.D.
 
 
 

 

LECTURE NOTES

LECTURE 4 PART 1

Decisions

September 26 / October 24, 2002

 

  
     









1

Concepts

  

 

Decisions

  
 

Deciders

  

 

Cascaded Deciders

  
     










2

What are Decisions for?

  

 

A bridge between perception and action

  
 

Examples of things to make decisions about in dialogue:

  

 

  • Where to pay attention, where to look

  • What kinds of data are important

  • What data to select

  • What to do if we’re under time pressure

  • When to speak

  
 
 










3

What is a Decision?

  

Decisions are necessarily boolean events:

 
 

There is no way to make a decision half-way

  

 

If you make a decision half-way (or 3/4th way or 12/27th way) you are not making a decision, you are in fact the very definition of someone who can't make a decision

  

 

Perceptual Information -> Crisp (boolean) decision

  
 

 

  










4

Decisions in Multimodal Dialog

  

Multi-layered

  • Spans 5 orders of magnitude in time, from 100 ms (e.g. gaze control) to hours (topic progression)

  • Seamlessness

  • Effortless

 

  • Dialogue participants only have decide what to say - other things (facial expressions, turn-taking, etc.) are run as ‘background processes’

  
 

  • Decisions to schedule and prepare for predicted events

  
     










5

Systems to Date: Problems with Decisions

  

Problems in 'Multimodal Natural Dialogue' [Thorisson, Koons, Bolt, 1992]

  

  • No sense of time

  
 

  • Decisions only made at one level of granularity, according to a fixed sequence of events, leading to stilted, unnatural turn-taking

  
 

  • If interpretation works, the system will act in a deterministic way, not taking context into consideration

  
     










6

Systems to Date: Problems with Decisions

  
 

J. Jr. [Thorisson, 1994]

  
 

  • All decisions are modeled as transitions between pre-determined states

  
 

  • Difficult or impossible to add meta-decisions (i.e. decisions about decisions)

 
 

  • Difficult to add new states -> throws the whole system off balance

  
 

  • Perception, decision and motor control cannot be coerced into the same (sometimes integrated) mechanism: FSMs

  
 

 









7

Techniques for Decisions

  

Boolean logic

  • Simple

  
 

Fuzzy logic

  • Perception: Classify concepts into ‘fuzzy’ sets

  • Action: Use output to control action

  • Handy method for a lot of language-related problems, e.g. what does it mean for an object to be ‘next to’ something?

  
 

Neural models

  • Argument: ‘The brain uses neurons, hence, artificial neural networks are the way to go’

  • Main problem: complexity, too low-level

  
     












8

Techniques for Decision
(a.k.a. action selection)

  
 

Behavior-based AI

  • Does away with explicit decision step

  • Direct connection between sensors and actuators

  • Often also connects knowledge to actuators in little pockets

  
 


Problems

  • When you push the decision making into the sensors, and the knowledge into the actuation, it becomes very difficult to manage knowledge (it tends to become "hard coded")

  • Which is why behavior-based AI is good at reactive behavior control and bad at handling natural language

  
 
 
  







9

Deciders in Ymir

  
 

Separate the decision process from ‘cognition’

  
 

Model decision as a set of distributed processes that look at pieces of perceptual & internal data and initiate actions

  
 

Input:

  • All deciders can take multimodal input

  
 

Read Blackboards for...

  • perceptual data

  • state information about the agent’s mind

  
 

Output:

  • Single, boolean output: a decision to act

  • A decision to move the body is a behavior request - an intention to act, a non-deterministic event

  
 
 
     







10

Deciders in Ymir

  
 

2 types: Covert, Overt

  • Overt Decision Modules—those that initiate overt actions

  • Covert Decision Modules—those that only change the internal state

  
 

Overt Decisions (may) result in a body movement

  
 

Covert Decisions (always) result in an internal process happening

  
     





11

Deciders in Ymir

  
 

Multiple levels of granularity

  • ‘close-eyes’ (1 dof, simple timing)

  • ‘smile’ (12 dof, complex timing)

  • ‘activate-[MI1, MI2, ... MIn]’ (simple internal event)

  • ‘parse-now’ (complex internal event)

  
 

Time-bound

  • Uses time-stamps to be real-time aware

  
 

Cascaded

  • Enable deciders to make many decisions about one goal

  
     














12

Decider Example 1

  
 

Parse-Speech-1

  
 

  • Covert (“internal”) Decider

  
 

  • Role: initiate parsing of user’s speech

  
 

  • Conditions: (user-gives-turn T)(user-speech-avail T)

  
 

  • Reset: (I-give-turn T)

  
 

  • Expected-Lifetime: <turn> OR <ts>

  
     













13

Decider Example 2

  
 

Hesitate-1

  
 

  • Overt (“external”) Decider

  
 

  • Role: show hesitation

  
 

  • Condition: when the agent takes turn but has not decided what to say yet

  
 

  • Input: (dial-on true)(I-have-turn true)(usr-spch-avail true) (user-speaking false)(time-since ‘user-speaking 700)(CL-act-avail false)

  
 

  • Reset conditions: (I-give-turn true)

  
 

  • Expected-Lifetime: 1200 ms

  
     










 

14

More Decider Examples

  
 

  • Move-motor-X-to-pos-Y

  • Blink, Nod

  • Show-I-agree

  • Look-away-from-user, Look-at-object-[X]

  • Show-I’m-taking-turn

  • Show-I-know-user-is-addressing-me

  • Deliver-multimodal-response

  
 


 

 

 

 

15

Cascaded Deciders

  
 

Used to enable a character to react to a number of contingencies related to a single problem

  
 

Situation:

  • The agent has one Decider to show that he’s pointing at something.

  • The Decider is called Show-User-Im-Referring-To-CoPresent-Object.

  • The Decider is triggered, but the user fails to see the action.

  • The agent notices that the user has failed to see the pointing, which means the conversation will fail

  • The agent has no other Deciders to deal with the situation, hence, the conversation will fail.

  • What can we - the agent designers - do?

  
 

 





16

Cascaded Deciders

  
 

The failure of one decision triggers another one in a sequence of decisions, all aimed at achieving the same goal.

  
 


 
 

 
 

Exampe: Answering a question

  • Decider-1: If User asks a question, answer within 1 sec

  • Decider-2: If Decider-1 fails, show hesitation within 1.1 sec, then answer as soon as possible

  • Decider-3: If Decider-2 fails, show increased alertness, give indication of thougth, answer as soon as possible

  • Decider-4: If Decider-3 fails, and user is still waiting for a response, appologize and summarize state of response generation

  
     







PART 2 ->

2002©K.R.Thórisson