Let the avatar see. Opt-in, opened just-in-time, held in memory only — and what the user sees while it is on.
Perception lets the avatar answer questions about what the camera can see ("what am I holding?", "does this look right?"). It is off unless you turn it on, and turning it on does not open the camera.
Even with perception.camera enabled, the SDK opens the camera only when the
server asks for a look, mid-conversation. It is never opened on connect.
That ordering is the point. A camera that switches on at the start of every call and stays on is a camera the user stops thinking about. One that opens when the avatar needs to see something is one the user can connect to a reason.
perception: { camera: true } is a grant, not a switch. The browser still
shows its own permission prompt the first time, and the user can refuse. Plan
for a session where the avatar never gets to see anything.
Do not collapse these — "permission granted" does not mean "a picture is arriving", and neither means "the session is over".
| State | What it tracks |
|---|---|
| Session open | The camera session exists and the self-view is mounted. Ends only when the call ends. |
| Video on / off | The user's own toggle. Off releases the camera hardware and drops the frame buffer, but keeps the session and the self-view up. |
| Permission | Granted once and remembered. It survives the video toggle and the end of the call, so re-opening never re-prompts. |
If the user switches video off, the SDK will not silently re-open it — only an explicit toggle back on will. A camera the user turned off staying off is not negotiable.
| Property | Value | Why |
|---|---|---|
| Capture rate | 3 fps | Enough to catch a gesture; low enough not to compete with the call for CPU. |
| Rolling buffer | 12 s | So a retroactive question ("what did I just do?") is answerable. |
| Frame size | 512 px longest edge | Downscaled before encoding. The model does not need more, and the upload does not want it. |
| Encoding | JPEG, quality 0.7 | Small enough to send inside a live call's latency budget. |
| Frames sent per look | 3, newest last | At 3 fps that is the last second. Older frames are near-duplicates and only add model latency. |
Frames live in memory and are never persisted. The buffer is a fixed 12-second window that discards anything older on every capture, and it is dropped entirely when video is switched off or the session ends. Nothing is written to disk, and nothing survives the call.
A freshly-opened camera has an empty buffer — getUserMedia resolves before
the hardware yields a decodable frame, typically by more than a second. The SDK
waits for the first real frame rather than sending nothing, so the first look
after a cold grant is slower than later ones.
While a camera session is open, the SDK renders a small live preview of the user's own camera. It is the mechanism by which the user knows the camera is on.
| Prop | Type | Default | Description |
|---|---|---|---|
cameraPreviewCornerOptional | CameraPreviewCorner | "bottom-right" | Which corner the self-view sits in. Ignored in picture-in-picture. |
hideCameraPreviewOptional | boolean | false | Hide the self-view. Discouraged — capture continues either way, so all this removes is the user's ability to see that it is happening. Ignored in picture-in-picture. |
usePictureInPictureOptional | boolean | false | Swap the layout: camera on the stage, avatar in the corner, once a camera session opens. Takes precedence over the two props above. |
pictureInPictureCornerOptional | CameraPreviewCorner | "bottom-right" | The avatar's corner in picture-in-picture. Independent of cameraPreviewCorner, since the two modes place different things. |
Think hard before setting hideCameraPreview. The self-view is a privacy
signal, not decoration — hiding it means the camera is running with nothing on
screen to say so.
The SDK distinguishes the user's intent from everything else, and reports the difference to the server so the avatar does not ask to see something it cannot:
| Event | What happens |
|---|---|
| The user toggles video off | Camera released, buffer dropped, session and self-view stay. A later look can legitimately re-open it only after the user toggles back on. |
| The camera is unplugged, or another app takes it | Treated as an involuntary loss, not a user decision. The feed is reported as off; a later request may re-open it. |
| Permission is revoked mid-call | Both axes collapse — access denied, feed off. The next open would prompt again. |
| No camera hardware, or an insecure context | Reported as unavailable rather than denied. Retrying will not help. |
On devices with more than one camera, the avatar can switch between them. The SDK detects whether a flip is actually possible before advertising it, and keeps the current camera if the target one fails to open — a failed flip never leaves the user with no picture.
Camera perception is supported on React Native, with the same opt-in config and
the same just-in-time opening. The self-view corner and hideCameraPreview work
the same way; picture-in-picture is web-only.
Camera permission on mobile is a native permission — see Permissions for the Expo config-plugin setup.
Before shipping perception, make sure your own product does these — the SDK cannot do them for you:
If you only need the avatar to know about something rather than see it,
addLiveContext is cheaper,
faster, and asks nothing of the user.