WebSocket protocol

Connecting

Open a standard WebSocket to wss://<your-voice-orch>/v1/agents/<publicId>/ws?token=<connectToken>. The token is the same one your /api/voice-token handler mints: short-lived, single-use, scoped to one session. The runtime closes the connection with a 1008 policy-violation code if the token's deployment_id, public_id, or agent_version_id don't match, or if it's been replayed.

const { connectToken, websocketUrl } = await fetch('/api/voice-token', {
  method: 'POST',
}).then((r) => r.json());

const url = new URL(websocketUrl);
url.searchParams.set('token', connectToken);
const ws = new WebSocket(url.toString());

ws.addEventListener('message', (event) => {
  const data = JSON.parse(event.data);
  console.log(data.type, data);
});

Audio format

Mic audio you send up: PCM16, little-endian, 24 kHz, mono, base64-encoded, wrapped in input_audio_buffer.append events. TTS audio you receive: same format, in response.output_audio.delta events. The runtime downsamples to 16 kHz internally for STT and resamples back up for the playback ack, so you don't need to handle sample-rate conversion on your end.

Server → client events

The runtime emits these. Listen in your onmessage handler.

Lifecycle

User-turn events

Assistant-turn events

Tool & usage events

Client → server events

Send these as JSON-encoded WebSocket messages.

End-to-end example: text-only consumer

The simplest possible custom UI: sends typed text, receives assistant text, ignores audio entirely. Useful as a starter for chat-style integrations that don't need mic.

async function startTextSession() {
  const { connectToken, websocketUrl } = await fetch('/api/voice-token', {
    method: 'POST',
  }).then((r) => r.json());

  const url = new URL(websocketUrl);
  url.searchParams.set('token', connectToken);
  const ws = new WebSocket(url.toString());

  ws.addEventListener('open', () => {
    ws.send(JSON.stringify({
      type: 'conversation.item.create',
      item: {
        type: 'message',
        role: 'user',
        content: [{ type: 'input_text', text: 'Hello, what can you do?' }],
      },
    }));
    ws.send(JSON.stringify({ type: 'response.create', response: { modalities: ['audio'] } }));
  });

  ws.addEventListener('message', (event) => {
    const e = JSON.parse(event.data);
    switch (e.type) {
      case 'response.output_text.delta': process.stdout.write(e.delta); break;
      case 'response.output_text.done':  console.log('\n[turn complete]');     break;
      case 'response.done':              ws.close();                              break;
      case 'platform.error':             console.error('error:', e.error);        break;
    }
  });
}

Recommended client design

• Pace audio sends. Send input_audio_buffer.append on a steady ~50 ms cadence; bursts cause the server-side VAD to oscillate.
• Honor barge-in. When you receive platform.barge_in, stop scheduling pending TTS audio frames immediately. Frames you've already scheduled will keep playing for ~50 ms, which is expected.
• Apply a short crossfade. If you concatenate audio deltas naively, you'll hear clicks at chunk seams. The widget applies ~120 ms equal-power crossfade; the same trick works for any client.
• Reconnect on network blips, not on logical errors. A clean WS close from the server with code 1008 means the session was rejected (bad token, unauthorized origin). Don't auto-reconnect. Reconnect only on 1006 (abnormal closure).
• Keepalive at 15 s. Idle WebSockets get reaped by some proxies after 30–60 s of silence. The widget pings every 15 s; do the same in custom clients.