CLIENT SDK

The Client SDK (@aether-stack-dev/client-sdk) runs in the browser and connects to a UE-hosted AStack gateway via WebSocket while your app embeds the signed renderer player separately.

INSTALLATION

$ npm install @aether-stack-dev/client-sdk

CONFIGURATION

// Session credentials are returned by your backend after it calls the control plane
const session = await fetch('/api/start-session', { method: 'POST' }).then(r => r.json());

const client = new AStackCSRClient({
  gatewayWsUrl: session.gatewayWsUrl,     // Preferred per-session WebSocket URL
  gatewayUrl: session.gatewayUrl,         // HTTPS gateway origin fallback
  sessionToken: session.sessionToken,     // Auth token from the control plane
  rendererPlayerUrl: session.rendererPlayerUrl,
  sampleRate: 24000,                      // Audio sample rate (default: 24000)
  fps: 30,                                // Animation frame rate (default: 30)
  enableImageCapture: true,               // Enable camera capture (default: true)
  imageCaptureInterval: 5000,             // Capture interval ms (default: 5000)
});

gatewayWsUrlPreferred WebSocket URL for the UE-hosted customer gateway

gatewayUrlHTTPS gateway origin used when gatewayWsUrl is omitted

sessionTokenSession token returned by the control plane

rendererPlayerUrlSigned Pixel Streaming URL for iframe or tab embedding

workerUrlTemporary migration alias for gatewayUrl; avoid in new code

enableImageCaptureSend camera frames for vision processing

autoReconnectAuto-reconnect on disconnect (default: true)

maxRetriesMax reconnection attempts (default: 5)

reconnectDelayBase delay between reconnects in ms, with exponential backoff (default: 1000)

audioProcessorUrlPath to AudioWorklet processor file (default: '/audio-processor.js')

debugEnable debug logging (default: false)

METHODS

connect(): Promise<void>Establish WebSocket connection to the gateway and authenticate

startCall(options?: CallOptions): Promise<void>Initialize audio context, request mic/camera, start streaming. Options: providers, systemPrompt, priorContext, configOverrides, disableA2F

stopCall(): voidStop audio/video capture, reset blendshapes

disconnect(): voidStop call and close WebSocket connection

sendText(message: string): voidSend text input to the AI (instead of voice)

getGatewayStatus(): GatewayConnectionStatusReturn gateway connection/auth state

getRendererPlayerUrl(): string | nullReturn the signed renderer player URL, if provided

getRendererPlayerState(): RendererPlayerStateReturn renderer player URL/status state separately from gateway auth

getCallStatus(): CallStatusReturns 'idle' | 'starting' | 'active' | 'stopping' | 'error'

getCurrentBlendshapes(): number[]Returns current 52-value blendshape array

isConnected(): booleanReturns true if WebSocket is open

destroy(): Promise<void>Disconnect and clean up all resources

getReconnectAttempts(): numberReturns current reconnection attempt count

STARTCALL OPTIONS

await client.startCall({
  providers: {
    asr: 'self',       // 'self' | 'deepinfra' (default: 'self')
    llm: 'self',       // 'self' | 'deepinfra' (default: 'self')
    tts: 'self',       // 'self' | 'deepinfra' (default: 'self')
  },
  systemPrompt: 'You are a helpful assistant.',
  disableA2F: false,
});

EVENTS

The client extends EventEmitter. Listen for events with client.on(event, callback).

connected() => void

WebSocket connection established

disconnected() => void

WebSocket connection closed

reconnecting(attempt: number) => void

SDK is attempting to reconnect (with attempt count)

reconnected() => void

SDK successfully reconnected after connection loss

error(error: Error) => void

Connection, audio, or processing error

callStarted() => void

Call session started on the worker

callStopped() => void

Call session stopped

speechStarted() => void

User started speaking (VAD detected)

interim(text: string) => void

Interim/partial transcript from ASR

transcript(text: string) => void

Final user speech transcription

utteranceEnd() => void

User finished speaking (EOU detected)

response(text: string) => void

AI response text

responseComplete() => void

Full AI response cycle completed (all chunks sent)

asrError(message: string) => void

ASR processing error occurred

blendshapeUpdate(blendshapes: number[]) => void

52 ARKit blendshape values for avatar animation

playbackStarted() => void

Audio playback began

playbackEnded() => void

Audio playback finished

modelStatus(status: { model_loaded?: boolean }) => void

Model loading status update

creditsExhausted() => void

Account credits have been exhausted

sessionExpired(reason: string) => void

Session has expired (with reason)

gatewayStatusChanged(status) => void

Gateway connection/auth state changed

rendererPlayerStateChanged(state) => void

Renderer player URL/status state changed

EXAMPLE

client.on('transcript', (text) => {
  console.log('User said:', text);
});

client.on('response', (text) => {
  console.log('AI:', text);
});

client.on('blendshapeUpdate', (blendshapes) => {
  // blendshapes is number[52] — ARKit blendshape weights (0-1)
  // Use with VRMAvatar for lip sync
});

client.on('error', (error) => {
  console.error('AStack error:', error.message);
});

AUDIO PLAYER

The AudioPlayer handles audio playback and blendshape synchronization. It is used internally by the client but can also be used standalone.

import { AudioPlayer } from '@aether-stack-dev/client-sdk';

const player = new AudioPlayer(24000); // sample rate

player.enqueue({ audio: arrayBuffer, blendshapes: frames });
player.clearQueue();
await player.destroy();

// Events: blendshapeUpdate, playbackStarted, playbackEnded, queueEmpty, error

REACT HOOKS

Import from @aether-stack-dev/client-sdk/react.

useAStackCSR

const {
  client,         // AStackCSRClient | null
  isConnected,    // boolean
  gatewayStatus,  // gateway connection/auth state
  rendererPlayerUrl, // signed Pixel Streaming/player URL
  callStatus,     // 'idle' | 'starting' | 'active' | 'stopping' | 'error'
  blendshapes,    // number[] (52 values)
  transcript,     // string — latest user speech
  response,       // string — latest AI response
  error,          // Error | null
  characterConfig, // CharacterConfig — spread on VRMAvatar
  connect,        // () => Promise<void>
  disconnect,     // () => void
  startCall,      // () => Promise<void>
  stopCall,       // () => void
  sendText,       // (message: string) => void
} = useAStackCSR({
  session,                 // Full session API response
  autoConnect: false,       // auto-connect on mount (default: false)
});

AVATAR COMPONENTS

import { VRMAvatar } from '@aether-stack-dev/client-sdk/react';

// 3D VRM avatar with Three.js
<VRMAvatar
  blendshapes={blendshapes}    // number[52] from useAStackCSR
  width={400}                   // pixels (default: 400)
  height={400}                  // pixels (default: 400)
  modelUrl="https://xwqtyabmavpacejuypyp.supabase.co/storage/v1/object/public/vrm-templates/5705015963733407866.vrm"
/>

ERROR HANDLING

import { AStackError, ErrorCodes } from '@aether-stack-dev/client-sdk';

client.on('error', (error) => {
  if (error instanceof AStackError) {
    switch (error.code) {
      case ErrorCodes.CONNECTION_LOST:
        // Reconnect logic
        break;
      case ErrorCodes.MEDIA_ACCESS_DENIED:
        // Prompt user for mic/camera permissions
        break;
    }
  }
});

ERROR CODES

INVALID_API_KEYAPI key is invalid or missing

INVALID_TOKENSession token is invalid

AUTHENTICATION_FAILEDAuthentication rejected by server

SESSION_EXPIREDSession has expired

CONNECTION_LOSTWebSocket connection dropped

NETWORK_ERRORNetwork connectivity issue

MEDIA_ACCESS_DENIEDMic/camera permission denied

AUDIO_ERRORAudio processing failure

RATE_LIMIT_EXCEEDEDToo many requests

INSUFFICIENT_CREDITSAccount has no remaining credits