Ultravox SDK

The Ultravox REST API is used to create calls but you must use one of the Ultravox client SDKs to join and end calls. This page primarily uses examples in JavaScript. The concepts are the same across all the different SDK implementations.

Ultravox Session

The core of the SDK is the UltravoxSession. The session is used to join and leave calls.

import { UltravoxSession } from 'ultravox-client';

const session = new UltravoxSession();
session.joinCall('wss://your-call-join-url');

session.leaveCall();

import 'package:ultravox_client/ultravox_client.dart';

UltravoxSession session = UltravoxSession.create();
await _session.joinCall('wss://your-call-join-url');

await session.leaveCall();

import ai.ultravox.UltravoxSession

val session = UltravoxSession()
session.joinCall("wss://your-call-join-url")

session.leaveCall()

import asyncio
import ultravox_client as uv

session = await uv.UltravoxSession()
session.join_call("wss://your-call-join-url")

await session.leave_call()

Methods

The UltravoxSession contains methods for:

Joining/leaving a call
Sending text messages to the agent
Changing the output medium for how the agent replies
Registering client tools
Muting the microphone/speaker

joinCall()

joinCall(joinUrl: string): UltravoxSessionState

Joins a call. Requires a joinUrl (string). Returns an UltravoxSessionState.

leaveCall()

async leaveCall(): Promise<void>

Leaves the current call. Returns a promise (with no return value) that resolves when the call has successfully been left.

sendText()

sendText(text: string): void

Sends a text message to the agent. Requires inputting the text message (string).

setOutputMedium()

setOutputMedium(medium: Medium): void

Sets the agent’s output medium for future utterances. If the agent is currently speaking, this will take effect at the end of the agent’s utterance. Also see muteSpeaker and unmuteSpeaker below.

parameter	description
medium	How replies are communicated. Must be either `'text'` or `'voice'`.

registerToolImplementation()

registerToolImplementation(name: string, implementation: ClientToolImplementation): void

Registers a client tool implementation with the given name. If the call is started with a client-implemented tool, this implementation will be invoked when the model calls the tool.

parameter	description
name	String. The name of the tool. Must match what is defined in `selectedTools` during call creation. If `nameOverride` is set then must match that name. Otherwise must match `modelToolName`.
implementation	`ClientToolImplementation` function that implements the tool’s logic.

ClientToolImplementation

This is a function that:

Accepts parameters → An object containing key-value pairs for the tool’s parameters. The keys will be strings.

Returns → Either a string result, or an object with a result string and a responseType, or a Promise that resolves to one of these.

For example:

  const stock_price = (parameters) => {
    ...  // to be implemented
    return `Stock price is ${value}`;
  };

registerToolImplementations()

registerToolImplementations(implementationMap: { [name: string]: ClientToolImplementation }): void

Convenience batch wrapper for registerToolImplementation.

implementationMap → An object where each key (a string) represents the name of the tool and each value is a ClientToolImplementation function.

isMicMuted()

isMicMuted(): boolean

Returns a boolen indicating if the end user’s microphone is muted. This is scoped to the Ultravox SDK and does not detect muting done by the user outside of your application.

isSpeakerMuted()

isSpeakerMuted(): boolean

Returns a boolen indicating if the speaker (the agent’s voice output) is muted. This is scoped to the Ultravox SDK and does not detect muting done by the user outside of your application.

muteMic()

muteMic(): void

Mutes the end user’s microphone. This is scoped to the Ultravox SDK.

unmuteMic()

unmuteMic(): void

Unmutes the end user’s microphone. This is scoped to the Ultravox SDK.

muteSpeaker()

muteSpeaker(): void

Mutes the end user’s speaker (the agent’s voice output). This is scoped to the Ultravox SDK.

unmuteSpeaker()

unmuteSpeaker(): void

Unmutes the end user’s speaker (the agent’s voice output). This is scoped to the Ultravox SDK.

Client Tools

Ultravox has robust support for tools. The SDK has support for client tools. Client tools will be invoked in your client code and enable you to add interactivity in your app that is driven by user interactions with your agent. For example, your agent could choose to invoke a tool that would trigger some UI change.

Creating Client Tools

Client tools are defined just like “server” tools with three exceptions:

1. “client” not “http”

You don’t add the URL and HTTP method for client tools. Instead, you add "client": {} to the tool definition.

{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_BODY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "client": {}
    }
  ]
}

{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_QUERY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "http": {
        "baseUrlPattern": "https://api.stockmarket.com/v1/price",
        "httpMethod": "GET"
      }
    }
  ]
}

2. Client Registration

Your client tool must be registered in your client code. Here’s a snippet that might be found in client code to register the client tool and implement the logic for the tool.

See SDK Methods for more information.

// Start up our Ultravox Session
uvSession = new UltravoxSession();

// Register our client-side tool
uvSession.registerToolImplementation(
  "stock_price",
  stock_price
);

uvSession.joinCall(joinUrl);

// Function that implements tool logic
const stock_price = (parameters) => {
  ...  // to be implemented
  return `Stock price is ${value}`;
};

3. Only Body Parameters

Unlike server tools (which accept parameters passed by path, header, body, etc.), client tools only allow parameters to be passed in the body of the request. That means client tools will always have parameter location set like this:

"location": "PARAMETER_LOCATION_BODY"

Session Status

The UltravoxSession exposes status. Based on the UltravoxSessionStatus enum, status can be one of the following:

status	description
disconnected	Session is not connected. This is the initial state prior to joinCall.
disconnecting	Session is in the process of disconnecting.
connecting	Session is establishing the connection.
idle	Session is connected but not yet active.
listening	Listening to the end user.
thinking	The model is processing/thinking.
speaking	The model is speaking.

Status Events

The status can be retrieved by adding an event listener to the session status. Building on what we did above:

JavaScript
Flutter

// Listen for status changing events
session.addEventListener('status', (event) => {
  console.log('Session status changed: ', session.status);
});

session.addListener(() {
  print('Status is now ${session.status}');
})

Transcripts

Sometimes you may want to augment the audio with text transcripts (e.g. if you want to show the end user the model’s output in real-time). Transcripts can be retrieved by adding an event listener:

import { UltravoxSession } from 'ultravox-client';

const session = new UltravoxSession();
session.joinCall('wss://your-call-join-url');

// Listen for transcripts changing events
session.addEventListener('transcripts', (event) => {
  console.log('Transcripts updated: ', session.transcripts);
});

session.leaveCall();

Transcripts are an array of transcript objects. Each transcript has the following properties:

property	type	definition
text	string	Text transcript of the speech from the end user or the agent.
isFinal	boolean	True if the transcript represents a complete utterance. False if it is a fragment of an utterance that is still underway.
speaker	Role	Either “user” or “agent”. Denotes who was speaking.
medium	Medium	Either “voice” or “text”. Denotes how the message was sent.

Debug Messages

The UltravoxSession object also provides debug messages. Debug messages must be enabled when creating a new session and then are available via an event listener similar to status and transcripts:

import { UltravoxSession } from 'ultravox-client';

const debugMessages = new Set(["debug"]);
const session = new UltravoxSession({ experimentalMessages: debugMessages });
session.joinCall('wss://your-call-join-url');

// Listen for debug messages
session.addEventListener('experimental_message', (msg) => {
  console.log('Got a debug message: ', JSON.stringify(msg));
});

session.leaveCall();

Debug Message: Tool Call

When the agent invokes a tool, the message contains the function, all arguments, and an invocation ID:

LLM response: Tool calls: [FunctionCall(name='createProfile', args='{"firstName":"Ron","lastName":"Burgandy","organization":"Fixie.ai","useCase":"creating a talking AI news reporter"}', invocation_id='call_D2qQVS8OQc998aMEw5PRa9cF')]

Debug Message: Tool Call Result

When the tool call completes, the message contains an array of messages. Multiple tools can be invoked by the model. This message array will conatain all the calls followed by all the results. These messages are also available via List Call Messages.

Here’s an example of what we might see from a single tool invocation:

Tool call complete.

Result: [
  role: MESSAGE_ROLE_TOOL_CALL ordinal: 6 text: "{\"firstName\":\"Ron\",\"lastName\":\"Burgandy\",\"organization\":\"Fixie.ai\",\"useCase\":\"creating a talking AI news reporter\"}" tool_name: "createProfile" invocation_id: "call_D2qQVS8OQc998aMEw5PRa9cF" tool_id: "aa737e12-0989-4adb-9895-f387f40557d8" ,
  role: MESSAGE_ROLE_TOOL_RESULT ordinal: 7 text: "{\"firstName\":\"Ron\",\"lastName\":\"Burgandy\",\"emailAddress\":null,\"organization\":\"Fixie\",\"useCase\":\"creating a talking AI news reporter\"}" tool_name: "createProfile" invocation_id: "call_D2qQVS8OQc998aMEw5PRa9cF" tool_id: "aa737e12-0989-4adb-9895-f387f40557d8"
]

SDK Implementations

There are currently five implementations of the SDK available:

JavaScript

npm install ultravox-client

Available in the npm registry

Flutter

flutter add ultravox_client

Get it on pub.dev

Python

pip install ultravox-client

More info on PyPi

Kotlin (Android)

Find it on Maven Central

Swift (iOS) [alpha]

Find it on Swift Package Index