Go Inside The Dome That Could Give Robots Super Senses

[Narrator] As advanced as robots have gotten

over the last few years, they can still be awkward.

The thing is, robots have a hard time reading us.

They can tell when they're dealing with humans, sure,

but do they know what a hug is?

Or what this gesture means?

How 'bout the difference between

a handshake and a fist bump?

But they're getting better.

In part thanks to this strange contraption.

The Panoptic Studio at Carnegie Mellon University.

The Panoptic Studio is a motion capture dome

loaded with 500 2D and 3D cameras

which capture footage from every conceivable angle.

The idea is that collecting loads of data on body language,

can lead to robots that'll better interact with humans.

So one panel has 24 smaller cameras,

and they are connected to this two devices

which consolidate all the information

as a single giant video.

One of the very important thing in the hardware is about

how to synchronize all these cameras.

[Narrator] It's an exciting approach,

because to do this kind of motion capture before,

you had to stick markers on a person

like they do in Hollywood.

What the Panoptic Studio captures

is a more natural form of human interaction.

What you end up with is a system

that can capture complex movements

in social interactions in astounding detail.

And this data could teach robots

to better understand human behavior.

Like group dynamics or even artistic performances.

(soft music)

Researchers here have collected a petabyte

of motion capture data.

That's one million gigabytes of subtle

and not so subtle body language.

With that data, they're training AI

to recognize human movements like this in real time.

That'll be pivotal for robots that work in sensitive fields

like heath care and law enforcement,

where body language telegraphs a whole lot of information.

Of course most robots aren't equipped with 500 cameras.

So the researchers actually streamline the technology

to run in real time through a webcam.

Our ultimate goal is to provide such ability

in a better life way system.

For example, if you want to build a robot.

Robot may have just one camera

or at most two or three cameras.

[Narrator] For robots to continue improving,

they'll have to recognize subtle changes

in our nonverbal behavior, because there's so much more

to communicating than just speaking.

So soon enough, the machines will be better able

to read your body language and expressions.

That is if they can manage to remain standing.

(robot crashing)