# Camera perception
Let the avatar see. Opt-in, opened just-in-time, held in memory only — and what the user sees while it is on.
Perception lets the avatar answer questions about what the camera can see
("what am I holding?", "does this look right?"). It is off unless you turn it
on, and turning it on does **not** open the camera.
To let the avatar see the user's screen rather than their camera, see [Screen share](/docs/sdk/react/screen-share).
```tsx
const config = {
getSessionToken: async () => fetchSessionToken(),
mode: "call",
perception: { camera: true }, // permission to ask — not an instruction to open
avatar: { avatarId: "default" },
};
```
## Just-in-time by design
Even with `perception.camera` enabled, the SDK opens the camera only when the
server asks for a look, mid-conversation. It is never opened on connect.
That ordering is the point. A camera that switches on at the start of every call
and stays on is a camera the user stops thinking about. One that opens when the
avatar needs to see something is one the user can connect to a reason.
> `perception: { camera: true }` is a **grant**, not a switch. The browser still
> shows its own permission prompt the first time, and the user can refuse. Plan
> for a session where the avatar never gets to see anything.
## Three independent states
Do not collapse these — "permission granted" does not mean "a picture is
arriving", and neither means "the session is over".
| State | What it tracks |
| --- | --- |
| Session open | The camera session exists and the self-view is mounted. Ends only when the call ends. |
| Video on / off | The user's own toggle. Off releases the camera hardware and drops the frame buffer, but keeps the session and the self-view up. |
| Permission | Granted once and remembered. It survives the video toggle and the end of the call, so re-opening never re-prompts. |
If the user switches video off, the SDK will **not** silently re-open it — only
an explicit toggle back on will. A camera the user turned off staying off is not
negotiable.
## What is captured
| Property | Value | Why |
| --- | --- | --- |
| Capture rate | 3 fps | Enough to catch a gesture; low enough not to compete with the call for CPU. |
| Rolling buffer | 12 s | So a retroactive question (\"what did I just do?\") is answerable. |
| Frame size | 512 px longest edge | Downscaled before encoding. The model does not need more, and the upload does not want it. |
| Encoding | JPEG, quality 0.7 | Small enough to send inside a live call's latency budget. |
| Frames sent per look | 3, newest last | At 3 fps that is the last second. Older frames are near-duplicates and only add model latency. |
> **Frames live in memory and are never persisted.** The buffer is a fixed
> 12-second window that discards anything older on every capture, and it is
> dropped entirely when video is switched off or the session ends. Nothing is
> written to disk, and nothing survives the call.
A freshly-opened camera has an **empty** buffer — `getUserMedia` resolves before
the hardware yields a decodable frame, typically by more than a second. The SDK
waits for the first real frame rather than sending nothing, so the first look
after a cold grant is slower than later ones.
## The self-view
While a camera session is open, the SDK renders a small live preview of the
user's own camera. It is the mechanism by which the user knows the camera is on.
| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `cameraPreviewCorner` | `CameraPreviewCorner` | `"bottom-right"` | Which corner the self-view sits in. Ignored in picture-in-picture. |
| `hideCameraPreview` | `boolean` | `false` | Hide the self-view. Discouraged — capture continues either way, so all this removes is the user's ability to see that it is happening. Ignored in picture-in-picture. |
| `usePictureInPicture` | `boolean` | `false` | Swap the layout: camera on the stage, avatar in the corner, once a camera session opens. Takes precedence over the two props above. |
| `pictureInPictureCorner` | `CameraPreviewCorner` | `"bottom-right"` | The avatar's corner in picture-in-picture. Independent of cameraPreviewCorner, since the two modes place different things. |
```tsx
```
> Think hard before setting `hideCameraPreview`. The self-view is a privacy
> signal, not decoration — hiding it means the camera is running with nothing on
> screen to say so.
## When the camera goes away
The SDK distinguishes the user's intent from everything else, and reports the
difference to the server so the avatar does not ask to see something it cannot:
| Event | What happens |
| --- | --- |
| The user toggles video off | Camera released, buffer dropped, session and self-view stay. A later look can legitimately re-open it only after the user toggles back on. |
| The camera is unplugged, or another app takes it | Treated as an involuntary loss, not a user decision. The feed is reported as off; a later request may re-open it. |
| Permission is revoked mid-call | Both axes collapse — access denied, feed off. The next open would prompt again. |
| No camera hardware, or an insecure context | Reported as unavailable rather than denied. Retrying will not help. |
## Mood reading
Mood reading comes with the camera: there is nothing to configure. Whenever the
camera is on, the self-view shows a **Detect mood** switch. It is **off** until
the user turns it on. While it is on, the SDK reads their expression **on their
own device** and the avatar adjusts its tone: warmer when they look happy, calmer
and more patient when they look frustrated. It never opens the camera itself, and
it never names the user's mood unless they ask.
A read happens only when all of these are true:
| Gate | Where it is set |
| --- | --- |
| Camera perception | `perception.camera: true` in your config, and **Camera control** in the avatar's Tool calling panel. |
| A live camera | The user has granted the camera and their video is on. |
| The user's switch | **Detect mood**, in the self-view. Off by default; the user turns it on and off at any time. |
> **Only a one-word mood leaves the browser** — `happy`, `angry`, `sad`,
> `surprised` or `neutral`. Frames and face measurements stay on the device, and
> the mood is never stored: not in the conversation history, not in transcripts.
### A custom toggle
The built-in switch lives in the self-view. To put one in your own UI, read and
set the same state from `useAvatar` — the two stay in sync.
```tsx
function MoodToggle() {
const { isMoodDetectionAvailable, isMoodDetectionOn, setMoodDetection } =
useAvatar();
if (!isMoodDetectionAvailable) return null;
return (
);
}
```
`isMoodDetectionAvailable` is true once a call is connected with camera perception
on. Reads still only happen while the camera is on and the switch is on.
### What it needs from the page
The face model is downloaded once per device (about 16 MB, cached afterwards) and
runs in a Web Worker, off your page's main thread. A strict Content Security
Policy must allow WebAssembly and `blob:` workers:
```
script-src 'self' 'wasm-unsafe-eval' blob:;
worker-src blob:;
connect-src https://api.avatarfactory.in wss://api.avatarfactory.in blob:;
```
If the model cannot load, mood reading switches itself off for that session; the
call and the camera are unaffected. Mood reading is web-only — on React Native,
`isMoodDetectionAvailable` is always false.
## Front and back cameras
On devices with more than one camera, the avatar can switch between them. The
SDK detects whether a flip is actually possible before advertising it, and keeps
the current camera if the target one fails to open — a failed flip never leaves
the user with no picture.
## React Native
Camera perception is supported on React Native, with the same opt-in config and
the same just-in-time opening. The self-view corner and `hideCameraPreview` work
the same way; picture-in-picture is web-only.
Camera permission on mobile is a native permission — see
[Permissions](/docs/sdk/react-native/permissions) for the Expo config-plugin
setup.
## Privacy checklist
Before shipping perception, make sure your own product does these — the SDK
cannot do them for you:
- **Say so before the call.** The browser prompt asks for the camera; it does
not explain why your avatar wants it.
- **Leave the self-view on.** It is the user's only continuous signal.
- **Give the user a video toggle** in your own UI if you are not using the
built-in controls.
- **Cover it in your privacy policy** — frames are transient, but they are still
images of a person, sent to a model vendor.
- **Tell users about mood reading** if you enable the camera. Reading emotions is
regulated in some places (in the EU it is restricted at work and in education),
so check it fits where and how you deploy.
> If you only need the avatar to know *about* something rather than *see* it,
> [`addLiveContext`](/docs/sdk/react/use-avatar#addlivecontextargs) is cheaper,
> faster, and asks nothing of the user.