EMBODIED AGENTS

IN AUGMENTED & VIRTUAL REALITIES

Course E6998-004, Dept. of Computer Science, Columbia University, Fall 2002
Prof. Kris Thórisson, Ph.D.

READINGS

Sept.
LINKS & REFERENCES
 
12
MULTIMODAL PERCEPTION
 
19

PERCEPTION / EMBODIMENT / MOTOR CONTROL

 
26

MOTOR CONTROL / DECISION MAKING

 
Oct.
   
10

DECISION MAKING / MULTIMODAL COMMUNICATION

 
17

MULTIMODAL DIALOGUE / ARCHITECTURE

 
24

INTEGRATION & ARCHITECTURES

  • McCarthy et al.'s 2002 paper on integrative architecture [html] [pdf dowload]
  • Thórisson's 1999 paper on the Ymir architecture
  • Bryson's 2001 thesis chapter 3 on hybrid architectures [download thesis]

    Supporting material:
  • Maes' 1990 paper on spreading activation: P. Maes. "Situated Agents Can Have Goals." Designing Autonomous Agents. Ed: P. Maes. MIT-Bradford Press, 1991. Also published as a special issue of the Journal for Robotics and Autonomous Systems,Vol. 6, No 1, North-Holland, June 1990.
 
31

INTEGRATION & ARCHITECTURES

 

 
Nov.
   
7

MULTIMODAL KNOWLEDGE REPRESENTATION

  • No readings

 

 
14

INTEGRATION & ARCHITECTURES

  • Chapter 5 from Newell's Unified Theories of Cognition.

 

 
21

INTEGRATION & ARCHITECTURES

  • No readings

 

 
Dec.
   
5
 
INFO

Reading the assigned papers is a prerequisite for attending the classes. Typically 3-5 papers are assigned for each class (approx. 3-5 hours of reading time). Additional material will be provided in certain cases; these are optional.

Good books on topcis related to the course, and to A.I. in general (note: list subject to expansion):

Minsky, M. (1986). Society of Mind. NY: Simon & Schuster.
Newell, A. (1990). Unified Theories of Cognition. MA: Harvard.
Handbook of Artificial Intelligence, second ed. (1992). Edited by Stuart C. Shapiro. New York: John Wiley & Sons.

Multimodality in Language & Speech Systems (2002). B. Granstron, D. House, I. Karlsson (Eds.). Amsterdam: Kluwer Academic Publishers.
Embodied Conversational Agents (2000). J. Cassell, J. Sullivan, S. Prevost, E. Churchill (Eds.). MA: M.I.T. Press

 

 

 

TERM PROJECTS

Name
Description
Due Date
1st paper

 

The first course assignment is a paper on the student's choice of one of the following topics:

- Steps towards more integrated AI systems.
- Real-time versus deliberation in systems with perception-action loops.
- Is systems integration necessary to "get AI right"?
- Are physical robots necessary for understanding cognition?
- Is embodiment a necessary ingredient to understand human cognition?
- Can languages like KQML help build smarter AI systems?
- Uses of embodied agents in augmented realities.
- Challenges in putting embodied agents into virtual and augmented realities.
- Benefits of studying embodied agents in augmented realities.
- Collaboration in augmented reality systems with embodied agents.
- The role of simulation in the future of AI.
- When will HAL-9000 be ready? Why?
- Embodied agents in entertainment.
- Critical analysis of implemented emotion simulation systems.
- Unified theories of cognition: Status report 2002.
- The golden screw in A.I.: Should we keep looking, and if so, where?
- Integration of reactive and reflective mechanisms.
- Intelligent agents incapable of natural language: Good for anything?
- Limitations of current dialogue models vis a vis embodied agents.

Half of the papers will be selected by the instructor to be presented at the end of the term by their authors (see below).

The paper should be 4000-6000 words long (7-15 pages), include references, and generally be in the style of journal publications (see e.g. AAAI Magazine or IJAAI for a examples). Grading will be based on the quality of the ideas presented and the quality of writing.

 

Oct. 10
2nd paper

The second term paper will be on a topic of your choice. You will need to do a proposal -- an outline (about 1 paragraph) of what you would like to write about. (The topic must of course be highly relevant to the topics of this course.)

Once your topic has been approved you have until the end of the term to finish the paper.

The paper should be 5000-7000 words long, include references, and generally be in the style of journal publications (follow, as closely as possible, the style of quotations used by J. Bryson in Chapter 3 of her thesis). Grading will be based on the quality of the ideas presented and the quality of writing, and how close to a professional publication the paper gets. Please submit both pdf and hard-copy (please conserve trees by printing on both sides of the pages).

Oct 31

Programming project phases

 

Phase 1. The programming project (see below) starts off with three proposasl, which has to be reviewed and approved by the instructor before implementation is started. Each proposal should be 1 paragraph in length, outlining roughly the topic and its motivation and components. As ready-made components, assume you have available (i.e. ready to integrate with minimal effort) a speech recognizer, a speech synthesizer, and a computer vision processor that can detect movement, and probably more (details forthcoming). You also will have the ability to sense speech-on/off within 50 ms of actual occurence. It's possible we will have facial animation available too. Any ideas for outside componetns are welcome - please bring them in early.

In our class we are focusing on (1) embodiment, (2) agents, (3) communication/interaction, (4) architectural integration, (5) virtual worlds, (6) augmented reality, (7) animation; and the emphasis is roughly in that order. So if you get between 2 and 4 of those elements into your demo idea, then you're on the right track. Ideally, at least two of them should be from the first four listed.

Phase 2. Integration plan, expected results, and final (functional) form of the implemented system is outlined by the team, as well as rough task assignments for the team members. For each major tast there will be chosen a Lead and a Support member -- the Lead is in the "hot seat", the support member is there to make sure there is backup if help is needed.

Phase 3. Each team member writes up a description and breakdown (as long and detailed as necessary) of their own personal/individual contribution to the project, identifying potential difficulties and challenges, and proposing solutions to deal with them, and who they need to collaborate with. The description should contain an outline of the module(s) and their functionalities that will be coded, along with the messages that the module(s) take as input and produce as output, and which module(s) affect and are affected by it. The more detail in the messages' content and syntax, the better for the team. To produce this the student must interact with other team members to settle the design. Notice that individual descriptions may have overlap -- students are encouraged to collaborate on project parts that have a significant overlap (for example, a Lead and support member on project A can collaborate on a description of project A, to be included in both of their descriptions handed in). Team Leads are responsible for making sure that all modules in the project can handle iminimal interaction. Included should be a project plan with measurable milestones, based on the outline for phase 2.

Phase 4. This is the implementation phase. Here the Leads for each task make sure that the milestones are reached, that all team members who need to collaborate are talking, and helps facilitate interactions, avoiding any obstacles along the way, and allerting the instructor of potential or actual problems. The Leads, in collaboration with the instructor, help revise the project, reassign tasks, etc., as necessary as the projects progress.

 


Programming project

 

The project must be highly relevant to the topic of the course -- i.e. it should deal with two or more of: integration, interaction, embodiment, and full perception-action loop systems (no standard"off-the-shelf" or "isolationist" topics will be accepted).

Projects will be done by groups of two, three and four students (size of groups will be partially determined by class size). "Lone ranger" projects are discouraged.

It is highly desirable that students work with systems that have already been developed. (To give an idea: A good example of this is the Sphinx speech recognizer from CMU.) Students are encouraged to work with existing research projects at Columbia whenever possible, with the aim of placing them in the context of the topics of this course.

 

Nov. 21
Paper presentation

 

The instructor will select a subset (or all, if time permits) of the papers for presentation by their authors. Selection will be based on merit; precedence will be given to those whose topics provide additional and/or alternative insights into the subjects covered in the course. Each presentation will be 15 minutes long and include 5 minutes of discussion/Q&A.

 

Nov. 14
Nov. 21
Project presentation

 

The instructor will select a subset (or all, if time permits) of the programming projects for presentation by the authors/teams. The length of each presentation will be be in accordance with the size of the group; each presentation will include a discussion/Q&A session.

 

Nov. 28
Dec. 5
Project guidelines

 

HOW TO CREATE A GREAT DEMO
- SOME PRACTICAL GUIDELINES


First, some facts about showing your demo to the "outside world":
1. People will criticize your demo. This is a fact of life. The main comment you'll hear is "But why can't I just _____", and "Sure that's cool, but I can already do that on my Amiga 64 in a way I think is better"; -- they're challenging the very core of your idea.
2. People will get stuck on silly things that have nothing to do with the point of the demo.
3. People will not understand the point of your demo. They either miss the point completely, or think that the emphasis is in a different place from where you want it to be.

Solutions:
1. Make sure the core of your idea has no (major) holes in it. It has to have a reason (why did you spend all that time on it?). Sometimes you can get away with just saying "it's an artpiece", but then it better be very cool. The best thing is to make the demo show a useful idea (the utilitarian approach).
2. Cut away everyhing that does not help people focus on your core idea. A lof of features in demos are like the room temperature: Unless it bothers you, you don't notice.
3. Making a demo is very much like writing an abstract: First you think of a scenario, then you consider all the elements that are necessary for it to work, then you select a small representative set of things as stand-ins for those elements, and then you implement those. As a result, to address all of the above, your demo (read: project) should be very distilled. Cut away any extraneous things that could take away the attention from the important points.