Tools in Ultravox (also known as function calling) are a powerful way to extend your agents' capabilities by connecting them to external services and systems. At their core, tools are simply functions that agents can invoke to perform specific actions or retrieve information.

Ultravox includes built-in tools and you can create custom tools.

Here are some of the things you can do with tools:

Communicate with the Outside world

Lookup the weather, get movie times, create calendar events, or send emails.

Order Lookup

Lookup orders, backordered items, or provide shipment updates.

Knowledge Base

Consult product and support documentation for contextual support.

Create Support Case

Open tailored support cases for human follow-up.

Transfer Call

Hand-off or escalate calls to human support agents.

End Call

End calls due to user inactivity or after successful resolution.

Any functionality you can encapsulate in a function can be exposed to your agents as a tool. Addtionally, Ultravox automatically calls the underlying function so you don't have to sweat gluing things together.
Difference from single-generation LLM APIs

Unlike using tools with single-generation LLM APIs, Ultravox Realtime actually calls your tool. This means you need to do a bit more work upfront in defining tools with the proper authentication and parameters.

Built-in Tools

Ultravox Realtime currently includes the following built-in tools.

Tool NameTool Action
queryCorpusRetrieves relevant information from an existing corpus (AKA knowledge base). Also see Query Corpus.
hangUpTerminates the call. You can also have your own custom tools end the call.
playDtmfSoundsPlays dual-tone multi-frequency (a.k.a dialpad) tones. See DTMF for details on sending and receiving tones in your voice applications.

Aside from being publicly visible, there’s nothing special about built-in tools. If you use list tools, you’ll see they use the same definitions you can create yourself.

Using a Built-in Tool

You give your AI agent access tools when you create the call or create a new call stage.

// Example request body for creating a call with a built-in tool
{
  "systemPrompt":"You are a helpful assistant. When the call naturally wraps up, use the 'hangUp' tool to end the call.",
  "selectedTools":[
    { "toolName": "hangUp" }
  ]
};

// The toolId can also be used
{
  "systemPrompt":"You are a helpful assistant. When the call naturally wraps up, use the 'hangUp' tool to end the call.",
  "selectedTools":[
    { "toolId": "56294126-5a7d-4948-b67d-3b7e13d55ea7" }
  ]
};

Note: Using built-in tools is the same as using any other custom durable tool that you have created except for one difference: you can override built-in tools by using the same name.

For example, if you created a durable tool named “hangUp” and then provide that tool by name (i.e. not by the toolId), then your tool would be used instead of the built-in hangUp tool.

Viewing Tools

To view all tools available to you, including built-in tools, use the List Tools API.

The List Tools API returns all available built-in tools along with any custom tools that you have created.

Custom Tools

Custom tools enable you to communicate with the outside world. Anything that you can do in a function can now be done by your agent via a custom tool.

Adding Tools for an Existing API

If you already run a server with a well-defined OpenAPI spec, you can quickly create tools for all your API endpoints by uploading that spec. Your OpenAPI spec must be either json or yaml format.

Once uploaded, your tools are just like any other durable tool, so you can use, modify, or delete them as you wish.

Creating Your First Custom Tool

Let’s look at creating a tool that sends an email with a summary of the conversation.

There are three steps:

1

Define the Tool

We need to define the tool and provide it to our agent. The name, description, and dynamic parameters we provide here will be seen by the agent so we need to be thoughtful with them.

// Creating a tool called 'sendConversationSummary'
//
// A 'string' parameter named 'conversationSummary'
// is passed in the body of a POST request to https://foo.bar/sendSummary
{
  "systemPrompt": "You are a helpful assistant...",
  "selectedTools": [
    {
      "temporaryTool": {
        "modelToolName": "sendConversationSummary",
        "description": "Use this tool at the end of a conversation to send the caller a summary of the conversation.",
        "dynamicParameters": [
          {
            "name": "conversationSummary",
            "location": "PARAMETER_LOCATION_BODY",
            "schema": {
              "description": "A 2-3 sentence summary of the conversation.",
              "type": "string"
            },
            "required": true
          }
        ],
        "http": {
          "baseUrlPattern": "https://foo.bar/sendSummary",
          "httpMethod": "POST"
        }
      }
    }
  ]
}

What’s happening here:

  • We are adding selectedTools to the request body of the Create Call.
  • There’s a single tool named sendConversationSummary.
  • This tool requires a single dynamic parameter called conversationSummary that is passed in the request body.
  • The tool’s functionality is available via POST at the url https://foo.bar/sendSummary.
  • The tool is a temporary tool, so it will only be available for this call.
2

Implement the Function

Now that we’ve defined the tool, let’s implement the functionality. This is a simplified example using Express.js and imagines a generic email API provider.

const express = require('express');
const router = express.Router();

router.post('/sendSummary', async (req, res) => {
  try {
    const { conversationSummary } = req.body;

    // Send the email using our email provider
    sendEmail(conversationSummary);

    return res.status(200).json({
      message: 'Conversation summary sent successfully. Continue the conversation with the user.'
    });
  } catch (error) {
    return res.status(500).json({
      message: 'Internal server error',
      error: error.message
    });
  }
});

module.exports = router;

This function does the following:

  • Accepts the conversationSummary via a POST.
  • Passes the data along to another function (sendEmail) that will send it via email.
3

Instruct the Agent on Tool Use

The last thing we need to do is provide additional instructions to the agent on how to use the tool. This is primarily done using the tool’s own description along with the systemPrompt. Let’s update what we used in the first step.

const updatedPrompt = `
  You're a friendly and fun guy. You like to chat casually while learning more about the person you're chatting with (name, hobbies, likes/dislikes).

  Be casual. Be fun to chat with. Don't talk too much. Keep your sentences pretty short and fun. Let the user guide the conversation.

  As you chat, try and learn more about the person you are talking to such as their name, hobbies, and their likes/dislikes.

  Once you have all the information, call the 'sendSummary' tool to send a summary of the conversation.
`;
{
  "systemPrompt": updatedPrompt,
  "selectedTools": [
      // Same as before
  ]
}

We’ve updated the system prompt that is used when the Ultravox call is created to instruct the agent when and how to use the tool.

Tool Implementations

Every tool needs exactly one implementation. So far we’ve used http tools, but client and dataConnection tools are also possible.

There are a couple things to note:

Tool Requirements

When you define a tool, you can also specify requirements for agents or calls that use the tool. In particular, you can detail the authentication requirements and any parameters that must be overridden with static values.

Tool Authentication

Ultravox has rich support for tool auth. When creating a tool, you must specify what is required for successful auth to the backend service.

Three methods for passing API keys are supported and are used when creating the tool:

1

Query Parameter

The API key will be passed via the query string. The name of the parameter must be provided when the tool is created.
2

Header

The API key will be passed via a custom header. The name of the header must be provided when the tool is created.
3

HTTP Authentication

The API key will be passed via the HTTP Authentication header. The name of the scheme (e.g. Bearer) must be provided when the tool is created.

Your tool can specify multiple options for fulfilling auth requirements (for example if your server allows either query or header auth). Each option may also contain multiple requirements, for example if your server requires both a user_id and an auth_token for that user.

When defining an agent or creating a call, you pass in the key(s) in the authTokens property of selectedTools. If the tokens you provide satisfy multiple options, the first non-empty option whose requirements are all satisfied will be used. An unauthenticated option, if present, will only be used if no other option can be satisfied.

// Create a tool that uses a query parameter called 'apiKey'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "queryApiKey": {
                "name": "apiKey"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
// Requests will include ?apiKey=your_token_here in the url
{
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

Required parameter overrides

Sometimes your tool will require a parameter to function that you need to have defined when the call is created instead of having the model come up with a value. In these cases, you can require that the parameter be overridden at call creation. For example, the built-in queryCorpus tool requires the corpus id to be specified during call creation.

More information about parameters and their overrides is below.

Tool Parameters

Tool parameters define what gets passed to your implementation when the tool is called. When creating a tool, parameters are defined as one of three types:

1

Dynamic

The model will choose which values to pass. These are the parameters you’d use for a single-generation LLM API.
2

Static

This value is known when the tool is defined and is unconditionally set on invocations. The parameter is not exposed to or set by the model.
3

Automatic

Like “Static”, except that the value may not be known when the tool is defined but will instead be populated by the system when the tool is invoked.

Dynamic Parameters

Dynamic parameters will have their values set by the model. Creating a dynamic parameter on a tool looks like this:

// Adding a dynamic parameter to a stock price tool
// The parameter will be named 'symbol' and will be passed as a query parameter
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "dynamicParameters": [
    {
      "name": "symbol",
      "location": "PARAMETER_LOCATION_QUERY",
      "schema": {
        "type": "string",
        "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
      },
      "required": true
    }
  ]
}
Parameter Overrides

You can choose to set static values for dynamic parameters when you create an agent or start a call. The model won’t see any parameters that you override. When creating a call simply pass in the overrides with each tool, as below. You should also consider overriding the tool name or description to give the model a more specific understanding of what the tool will do in this case.

// Overriding dynamic parameter when starting a new call
// Always set the stock symbol to 'NVDA'
{
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
    "nameOverride": "nvidia_stock_price",
    "descriptionOverride": "Looks up the current stock price for Nvidia."
    "parameterOverrides": {
      "symbol": "NVDA"
    }
  ]
}

Static Parameters

If you have parameters that are known at the time you create the tool, static parameters can be used.

// Adding a static parameter that always sends utm=ultravox
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "staticParameters": [
    {
      "name": "utm",
      "location": "PARAMETER_LOCATION_QUERY",
      "value": "ultravox"
    }
  ]
}
Parameter Overrides

Static parameters can also be overridden when you create an agent or start a call. This is most useful with built-in tools. For example, the built-in queryCorpus tool allows you to statically override max_results.

Automatic Parameters

Automatic parameters are used when you want a consistent, predictable value (not generated by the model) but you don’t know the value when the tool is created.

Here are some of the supported automatic parameters:

knownValueDescription
KNOWN_PARAM_CALL_IDUsed for sending the current Ultravox call ID to the tool.
KNOWN_PARAM_CONVERSATION_HISTORYIncludes the full conversation history leading up to this tool call. Typically should be in the body of a request.
KNOWN_PARAM_CALL_STATEIncludes arbitrary state previously set by tools. See Guiding Agents.
// Adding automatic parameters to a profile creation tool
// There are two parameters added:
// 'call_id' which is sent as a query param
// 'conversation_history' which is sent in the request body
{
  "name": "create_profile",
  "description": "Creates a profile for the current caller",
  "automaticParameters": [
    {
      "name": "call_id",
      "location": "PARAMETER_LOCATION_QUERY",
      "knownValue": "KNOWN_PARAM_CALL_ID"
    },
    {
      "name": "conversation_history",
      "location": "PARAMETER_LOCATION_BODY",
      "knownValue": "KNOWN_PARAM_CONVERSATION_HISTORY"
    }
  ]
}

You can find more on Automatic Parameters in the Base Tool Definition.

Temporary vs. Durable Tools

Custom tools come in two varieties: temporary and durable. There is much more information below but there are a few things to consider right upfront:

1

Creation

Temporary tools are created in the request body when a new call is created. Durable tools are created using the Ultravox REST API.
2

No Functional Difference

There is no functional difference within the context of an Ultravox call between the two tool types.
3

Iteration Speed

Temporary tools are great when you are building out a new application and need to iterate.
4

Reuse & Collaboration

Durable tools are best when you have things dialed in and want to reuse tools across applications and/or work with a team and want to divide ownership of tools from the rest of your app.

The distinction between temporary and durable only applies to custom tools. Built-in tools are always durable.

Temporary Tools

Temporary tools are created each time you create a new Call and exist exclusively within the context of that call. (Temporary tools aren’t visible in the List Tools response for example.)

Iteration is faster when using temporary tools because you don’t have to create/update/delete tools as you build out your application. You can simply adjust the JSON in the request body and start a new call.

{
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_QUERY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "http": {
        "baseUrlPattern": "https://api.stockmarket.com/v1/price",
        "httpMethod": "GET"
      }
    }
  ]
}

Durable Tools

In addition to temporary tools, Ultravox supports the creation of durable tools. There is no functional difference between durable and temporary tools within the context of a call, but durable tools are persisted and can be reused across calls or applications. They shine once you have things dialed in, when you want to share tools across multiple applications, or if you have split responsibilities on the team.

Use the /tools endpoints to work with durable tools.

To use a durable tool in a call, set the toolName or toolId field instead of using a temporaryTool.

For example:

// Request body for creating an Ultravox call with a durable tool
{
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
  ]
}

Dealing with latency

When adding your own tools, it’s important to keep in mind that there’s always a user actively waiting for your tool to respond. Tools need to be (or at least appear) fast to make sense in a real-time context.

Precomputable Tools

In order to make sure you can always understand why your tool was invoked, by default tool invocations are always included in the conversation history if they occur at all. To maintain this invariant when the model produces both text and a tool call, tools cannot be invoked until the agent is done speaking so that we know the agent won’t be interrupted.

This is essential for tools that modify state since there’s no good way to revert changes if the agent is interrupted. However, it’s obviously suboptimal for tools like queryCorpus where we’d like to look up information while the agent is speaking and simply ignore the response if the agent is interrupted. Tools like this can be marked precomputable.

Any tool marked precomputable will be speculatively invoked as soon as the model produces the tool call. When the model produces both text and the tool call, the tool’s latency can be masked by the agent speaking, but if the agent is interrupted there will be no record of the invocation.

In order to safely be marked precomputable, a tool should have three properties:

  1. No state changes. For http tools, GET requests are usually safe while methods like POST are not.
  2. No side effects. Even a GET request is not safe to precompute if it has a side effect! (It’s up to you to decide what counts here though. Side effects like logging probably don’t matter to you for example while any database write likely does.)
  3. Idempotent. The tool must return the same result when called with the same parameters, regardless of when or how many times it is called.

If your tool meets these requirements, you can mark it precomputable using the corresponding field.

Timeouts

While tools are executing, the conversation is essentially frozen. The user can continue talking all they like, but the agent will never respond until after the tool invocation completes. (The agent does have access to anything the user said during tool execution once execution completes.)

To users this may feel like the call was disconnected. In order to avoid that, tools are limited to 2.5 seconds by default. If your tool needs longer than that (and you can’t make it faster!), you can increase the timeout up to 20 seconds by setting the tool’s timeout field. You can also reduce your tool’s timeout if you’d like. The value is a duration in seconds, like 5s for 5 seconds or 0.1s for 100 milliseconds.

For tools that take even longer, consider responding immediately and later using an input_text_message data message with deferResponse with the real tool result. This is easiest with a dataConnection implementation since data connections are also able to send input text messages (and the response is always deferred in that case). Keep in mind that the model will see whatever response you send back initially, so you’ll want to make it clear to the model what’s going on by initially responding with some text like “Tool started. The full response will be available soon.”

Changing Call State

For most tools, the response will include data you want the model to use (e.g. the results of a lookup). However, Ultravox has support for special tool actions that can end the call or change the call stage.

These tool actions require setting a special response type.

Response TypeTool Action
hang-upTerminates the call. In addition to having Ultravox end the call after periods of user inactivity, your custom tool can end the call.
new-stageCreates a new call stage. See here for more.

How you set the response type depends on your tool implementation. HTTP tools set the response type via the X-Ultravox-Response-Type header. Client and data connection tools should set the responseType field in their tool result message.

Controlling Agent Responses to Tools

By default, the agent speaks again immediately after a tool call. This is typically the desired behavior for tools that gather information since the agent can immediately respond based on the information retrieved.

However, this may make less sense for other tools. For example, if your agent is gathering information for the user and you have a tool that allows the agent to store what’s been gathered so far, you may want the agent to speak either before or after the tool but not both.

Ultravox Realtime allows you to define how the agent reacts after a tool call by setting the agent reaction. A default can be set on the tool itself or you can use either the X-Ultravox-Agent-Reaction header (for http tools) or the agent_reaction field on the tool result message (for client and data connection tools) similar to how you’d set a response type (see above).

ReactionDescription
speaksAgent will speak immediately after the tool call returns. This is the default behavior if agent reaction is not set. Should be used for tools that gather information.
listensAgent listens for user input and doesn’t speak.
speaks-onceAgent speaks only if it didn’t speak immediately before the tool call. Prevents agent repeating things before and after the tool call.

Debugging

The Ultravox SDK enables viewing debug messages for any active call. These messages include tool calls.

Best Practices

  1. Clear Naming: Choose a descriptive and unique name for your tool that clearly indicates its function.

  2. Detailed Description: Provide a comprehensive description of what the tool does, including any important details about its usage or limitations. This and the name will help the model decide when and how to use your tool.

  3. Precise Parameters: Define your dynamic parameters carefully, guiding the model to produce values your tool can use. On the implementation side, be careful to validate values and be permissive with allowed presence and formatting if possible.

  4. Error Handling: Consider how your tool will handle errors or unexpected inputs, and document this behavior in the tool’s description.

  5. Iterate Faster: Use temporary tools when you are building your application. Persist durable tools in the system when things have stabilized.

  6. Version Control: When updating tools, consider creating a new version (e.g., “stock_price_v2”) rather than modifying the existing tool. This allows testing of the new tool before impacting new calls made with the prior version of the tool.

  7. Security: Be mindful of security when creating tools, especially when they interact with external APIs. Use appropriate authentication methods and avoid exposing sensitive information.

  8. Testing: Thoroughly test your tools before deploying them in production conversations to ensure they function as expected.