Ultravox Client SDK

The Ultravox REST API is used to create calls but you must use one of the Ultravox client SDKs to join and end calls. This page primarily uses examples in JavaScript. The concepts are the same across all the different SDK implementations.

​

Methods

The core of the SDK is the UltravoxSession. The session is used to join and leave calls. The UltravoxSession contains methods for:

• Joining/leaving a call
• Sending text messages to the agent
• Changing the output medium for how the agent replies
• Registering client tools
• Muting the microphone/speaker

​

joinCall()

Joins a call.

joinCall(joinUrl: string, clientVersion?: string): void

​

leaveCall()

Leaves the current call. Returns a promise (with no return value) that resolves when the call has successfully been left.

async leaveCall(): Promise<void>

​

sendText()

Sends a text message to the agent. If deferResponse is set, the agent will not respond (i.e. no LLM generation will be done).

sendText(text: string, deferResponse?: boolean): void

​

setOutputMedium()

Sets the agent’s output medium for future utterances. If the agent is currently speaking, this will take effect at the end of the agent’s utterance.

setOutputMedium(medium: Medium): void

Also see muteSpeaker and unmuteSpeaker below.

​

registerToolImplementation()

Registers a client tool implementation with the given name. If the call is started with a client-implemented tool, this implementation will be invoked when the model calls the tool.

registerToolImplementation(name: string, implementation: ClientToolImplementation): void

  const stock_price = (parameters) => {
    ...  // to be implemented
    return `Stock price is ${value}`;
  };

​

registerToolImplementations()

Convenience batch wrapper for registerToolImplementation.

registerToolImplementations(implementationMap: { [name: string]: ClientToolImplementation }): void 

​

isMicMuted()

Returns a boolen indicating if the end user’s microphone is muted. This is scoped to the Ultravox SDK and does not detect muting done by the user outside of your application.

isMicMuted(): boolean

​

isSpeakerMuted()

Returns a boolen indicating if the speaker (the agent’s voice output) is muted. This is scoped to the Ultravox SDK and does not detect muting done by the user outside of your application.

isSpeakerMuted(): boolean

​

muteMic()

Mutes the end user’s microphone. This is scoped to the Ultravox SDK.

muteMic(): void

​

unmuteMic()

Unmutes the end user’s microphone. This is scoped to the Ultravox SDK.

unmuteMic(): void

​

muteSpeaker()

Mutes the end user’s speaker (the agent’s voice output). This is scoped to the Ultravox SDK.

muteSpeaker(): void

​

unmuteSpeaker()

Unmutes the end user’s speaker (the agent’s voice output). This is scoped to the Ultravox SDK.

unmuteSpeaker(): void

​

Client Tools

Ultravox has robust support for tools. The SDK has support for client tools. Client tools will be invoked in your client code and enable you to add interactivity in your app that is driven by user interactions with your agent. For example, your agent could choose to invoke a tool that would trigger some UI change.

Client tools are defined just like “server” tools with three exceptions:

{
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_BODY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "client": {}
    }
  ]
}

// Start up our Ultravox Session
uvSession = new UltravoxSession();

// Register our client-side tool
uvSession.registerToolImplementation(
  "stock_price",
  stock_price
);

uvSession.joinCall(joinUrl);

// Function that implements tool logic
const stock_price = (parameters) => {
  ...  // to be implemented
  return `Stock price is ${value}`;
};

"location": "PARAMETER_LOCATION_BODY"

​

Session Status

The UltravoxSession exposes status. Based on the UltravoxSessionStatus enum, status can be one of the following:

The status can be retrieved by adding an event listener to the session status.

// Listen for status changing events
session.addEventListener('status', (event) => {
  console.log('Session status changed: ', session.status);
});

​

Transcripts

Sometimes you may want to augment the audio with text transcripts (e.g. if you want to show the end user the model’s output in real-time). Transcripts can be retrieved by adding an event listener:

// Listen for transcripts changing events
session.addEventListener('transcripts', (event) => {
  console.log('Transcripts updated: ', session.transcripts);
});

Transcripts are returned as an array of transcript objects.

​

Debug Messages

The UltravoxSession object also provides debug messages. Debug messages must be enabled when creating a new session and then are available via an event listener similar to status and transcripts:

// Start up our Ultravox Session with Debug Messages Enabled
session = new UltravoxSession({ experimentalMessages: ["debug"] });

// Listen for debug messages
session.addEventListener('experimental_message', (msg) => {
  console.log('Got a debug message: ', JSON.stringify(msg));
});

When the agent invokes a tool, the message contains the function, all arguments, and an invocation ID:

LLM response: Tool calls: [FunctionCall(name='createProfile', args='{"firstName":"Ron","lastName":"Burgandy","organization":"Fixie.ai","useCase":"creating a talking AI news reporter"}', invocation_id='call_D2qQVS8OQc998aMEw5PRa9cF')]

When the tool call completes, the message contains an array of messages. Multiple tools can be invoked by the model. This message array will conatain all the calls followed by all the results. These messages are also available via List Call Messages.

Here’s an example of what we might see from a single tool invocation:

Tool call complete.

Result: [
  role: MESSAGE_ROLE_TOOL_CALL ordinal: 6 text: "{\"firstName\":\"Ron\",\"lastName\":\"Burgandy\",\"organization\":\"Fixie.ai\",\"useCase\":\"creating a talking AI news reporter\"}" tool_name: "createProfile" invocation_id: "call_D2qQVS8OQc998aMEw5PRa9cF" tool_id: "aa737e12-0989-4adb-9895-f387f40557d8" ,
  role: MESSAGE_ROLE_TOOL_RESULT ordinal: 7 text: "{\"firstName\":\"Ron\",\"lastName\":\"Burgandy\",\"emailAddress\":null,\"organization\":\"Fixie\",\"useCase\":\"creating a talking AI news reporter\"}" tool_name: "createProfile" invocation_id: "call_D2qQVS8OQc998aMEw5PRa9cF" tool_id: "aa737e12-0989-4adb-9895-f387f40557d8"
]

​

SDK Implementations