Feature Request: Performance Input / Multimodal Perception Bridge From Prompting A Performance To Performing The Prompt

Current voice input still loses
a substantial part of human
communication.

A person does not communicate
through words alone.

Human expression also contains:

• prosody
• rhythm
• pauses
• breathing
• emphasis
• laughter
• hesitation
• vocal pressure
• emotional dynamics
• timing

When video is available,
additional information becomes visible:

• facial expression
• gaze
• eye movement
• head movement
• gestures
• posture
• body tension
• movement dynamics

Much of the required sensor technology
already exists.

High-quality microphones,
cameras,
face tracking,
motion capture,
gaze tracking,
voice analysis
and performance capture

can already record
an increasingly detailed representation
of human expression.

The problem is what happens afterwards.

A rich human performance
is often reduced to:

Human Performance
↓
Audio Signal
↓
Speech-to-Text
↓
Text
↓
AI

This creates
a major information bottleneck.

The sensors may already be
relatively close to human perception,

but much of the information
they capture is discarded
before the AI can use it.

The next step should therefore
not simply be better Speech-to-Text.

It should be:

SPEECH-TO-PERCEPTION

and, in its full form:

PERFORMANCE INPUT

THREE INPUT LEVELS

TEXT

Transfers semantic content.

VOICE

Transfers semantic content
plus vocal performance:

prosody,
rhythm,
dynamics,
pauses,
breathing,
laughter,
emphasis

and other paralinguistic information.

VIDEO-VOICE / PERFORMANCE INPUT

Transfers semantic content,
vocal performance
and visible physical performance
together.

Audio and video
should not simply be treated
as separate files.

They should remain synchronized
in time

and be interpreted
as parts of one performance.

INFORMATION THAT SHOULD BE PRESERVED

1. Verbatim Transcript

What was actually said?

1. Voice / Prosody

Volume,
tempo,
rhythm,
emphasis,
vocal pressure,
pauses

and changes over time.

1. Paralinguistic Events

Laughter,
breathing,
sighing,
whispering,
crying,
shouting,
vocal breaks

and similar events.

1. Facial Performance

Eyes,
mouth,
facial tension,
gaze fixation,
asymmetry

and subtle facial changes.

1. Body Performance

Head,
shoulders,
hands,
gestures,
posture

and movement dynamics.

1. Temporal Development

How do all of these signals
change together over time?

1. Interpretation Layer

What kind of character performance
does this combination suggest?

A critical distinction
must remain between:

OBSERVATION

and

INTERPRETATION.

For example:

Observation:

slow speech
→ long pause
→ slight smile
→ fixed gaze
→ sudden laughter
→ rapid increase
in vocal intensity

Interpretation:

the performed character appears
increasingly unpredictable,
threatening
or unstable

The interpretation
should never replace
the original performance data.

REPRESENTATIVE USE CASE:
THE JOKER PROBLEM

A character such as the Joker
from Batman demonstrates
the problem extremely well.

Imagine an actor performing
such a character

with all of the intelligence,
depth,
unpredictability,
humour,
menace

and destructive intensity
associated with the role.

If that performance reaches the AI
only as a text transcript,

the words may survive

while much of the character
disappears.

The transcript may lose:

• the timing before a sentence
• sudden acceleration
  or slowing of speech
• the type and rhythm of laughter
• breathing
• long pauses
• changes in vocal pressure
• gaze fixation
• facial asymmetry
• head and shoulder movement
• body tension
• the contrast between
  apparent control
  and sudden loss of control
• the exact moment
  when an ordinary sentence
  becomes disturbing

The semantic transcription
can therefore be completely correct

while the dramatic meaning
is severely reduced.

If the AI receives
only the transcript,

it receives the dialogue.

It does not necessarily receive
the character.

Character is not created
by vocabulary alone.

It emerges from
the interaction of:

LANGUAGE

• VOICE
• RHYTHM
• FACE
• BODY
• TIMING

FROM PROMPTING A PERFORMANCE
TO PERFORMING THE PROMPT

Today a user may have to write:

“The character starts calmly,
holds eye contact,
pauses,
smiles slightly,
suddenly accelerates,
begins laughing,
becomes increasingly intense

and then returns abruptly
to complete stillness.”

But if the user can already
perform this behaviour

in front of a microphone
and camera,

why discard that information

and reconstruct it afterwards
as text?

The proposed workflow would be:

Human Performance
↓
Audio + Video + Timing
↓
Multimodal Performance Analysis
↓
Character Intent
↓
Generated Voice

• Facial Performance
• Body Performance
  ↓
  Generated Character / Scene

The central idea is:

FROM PROMPTING A PERFORMANCE

TO

PERFORMING THE PROMPT.

CONNECTION TO GENERATIVE VIDEO

Performance information
should not necessarily stop
at the character itself.

It could influence:

Facial Expression
↓
Body Performance
↓
Camera
↓
Editing
↓
Lighting
↓
Visual Dramaturgy

A change in human performance
could therefore influence

not only the generated face
or voice,

but also the way
the entire scene is directed.

This continues the evolution from:

IMAGE GENERATION
↓
IMAGE DESIGN
↓
IMAGE DIRECTION

TECHNICAL FEASIBILITY

Many of the necessary technologies
already exist

in mature professional environments:

• Face Tracking
• Facial Performance Capture
• Motion Capture
• Eye / Gaze Tracking
• Voice Tracking
• Prosody Analysis
• Real-Time Rendering
• Virtual Production
• Generative Audio
• Generative Video

These technologies
do not necessarily need
to be reinvented.

Experts and software companies
from film,
games,
animation,
audio,
virtual production

and computer vision

could be brought together.

The core development challenge
is the integration layer:

a synchronized architecture
that preserves
and translates human performance
across modalities.

The goal should not be
five separate features.

The goal should be:

A BRIDGE BETWEEN
HUMAN PERFORMANCE
AND GENERATED CHARACTER.

NEW PROFESSIONAL ROLES

This approach could also create
a constructive path

for actors,
voice actors

and other performers
within generative production.

Instead of treating
human performers

only as something
that AI replaces,

they could become
direct PERFORMANCE INPUT.

Possible roles could include:

• Performance Actor
• Character Voice Artist
• AI Performance Director
• Performance Capture Artist
• Character Performance Designer

The model would not have to be:

AI INSTEAD OF ACTORS

It could become:

AI + HUMAN PERFORMANCE

RIGHTS AND GOVERNANCE

Consent,
compensation,
usage period,
licensing

and reuse of a person’s face,
voice

and performance

are important issues.

They should be treated
as a dedicated governance layer.

They should not replace
the technical
and creative discussion itself.

Rights should define
how a technology may be used,

not whether
the underlying capability
should be developed.

Technical development
and rights frameworks

can evolve in parallel.

CORE PRINCIPLE

The decisive next step
is not simply
to add more sensors.

It is to preserve,
synchronize

and understand
the information

the sensors already capture.

Do not transcribe
only what was said.

Preserve how it was performed.

PERFORMANCE IS INFORMATION.

The bridge between humans and AI
begins when the system receives

not only our words,

but also the way
we put those words
into the world.

2 posts - 1 participant

Read full topic