Tools in Ultravox

Temporary vs. Durable Tools

Ultravox supports two types of tools: temporary and durable. There is much more information below but there are a few things to consider right upfront:

Creation → Temporary tools are created in the request body when a new call is created. Durable tools are created using the Ultravox REST API.
No Functional Difference → There is no functional difference within the context of an Ultravox call between the two tool types.
Iteration Speed → Temporary tools are great when you are building out a new application and need to iterate.
Reuse & Collaboration → Durable tools are best when you have things dialed in and want to reuse tools across applications and/or work with a team and want to divide ownership of tools from the rest of your app.

Temporary Tools

Temporary tools are created each time you create a new Call and exist exclusively within the context of that call. (Temporary tools aren’t visible in the List Tools response for example.)

Iteration is faster when using temporary tools because you don’t have to create/update/delete tools as you build out your application. You can simply adjust the JSON in the request body and start a new call.

Creating & Using Temporary Tools

Temporary tools are defined and passed in the request body of the Create Call endpoint. They are available during the current call.

{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_QUERY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "http": {
        "baseUrlPattern": "https://api.stockmarket.com/v1/price",
        "httpMethod": "GET"
      }
    }
  ]
}

Durable Tools

In addition to temporary tools, Ultravox supports the creation of durable tools. There is no functional difference between durable and temporary tools within the context of a call.

Durable tools are persisted and can be reused across calls or applications. They shine once you have things dialed in, when you want to share tools across multiple applications, or if you have split responsibilities on the team.

Creating Durable Tools

You create durable tools either by uploading an OpenAPI spec or via the request body in the Create Tool endpoint. Your OpenAPI spec must be either json or yaml format.

The /tools endpoint in the Ultravox API is for working with durable tools.

Using Durable Tools

To use a durable tool in a call, set the toolName or toolId field instead of temporaryTool. For example:

// Request body for creating an Ultravox call with a durable tool
{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
  ]
}

Tool Authentication

Ultravox has rich support for tools auth. When creating a tool, you must specify what is required for successful auth to the backend service. Three methods for passing API keys are supported and are used when creating the tool:

Query Parameter → The API key will be passed via the query string. The name of the parameter must be provided when the tool is created.
Header → The API key will be passed via a custom header. The name of the header must be provided when the tool is created.
HTTP Authentication → The API key will be passed via the HTTP Authentication header. The name of the scheme (e.g. Bearer) must be provided when the tool is created.

You then pass in the key(s) in the authTokens property of selectedTools when creating a call.

// Create a tool that uses a query parameter called 'apiKey'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "mySeviceApiKey": {
              "queryApiKey": {
                "name": "apiKey"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
{
  "model": "fixie-ai/ultravox-70B"
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

// Create a tool that uses an HTTP Header named 'X-My-Header'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "headerApiKey": {
                "name": "X-My-Header"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
{
  "model": "fixie-ai/ultravox-70B"
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

// Create a tool that uses HTTP Authentication scheme 'Bearer'. When the tool is invoked it will receive an 'Authorization' header with value "Bearer your_token_here".
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "httpAuth": {
                "scheme": "Bearer"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the Bearer token during call creation
{
  "model": "fixie-ai/ultravox-70B"
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

Tool Parameters

Tool parameters define what gets passed in to your backend service when the tool is called. When creating a tool, parameters are defined as one of three types:

Dynamic → The model will choose which values to pass. These are the parameters you’d use for a single-generation LLM API. Can be overridden (see below).
Static → Value is known when the tool is defined and is unconditionally set on invocations. Not exposed to or set by the model.
Automatic → Like “Static”, except that the value may not be known when the tool is defined but will instead be populated by the system when the tool is invoked.

Dynamic Parameters

Dynamic parameters will have their values set by the model. Creating a dynamic parameter on a tool looks like this:

// Adding a dynamic parameter to a stock price tool
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "dynamicParameters": [
    {
      "name": "symbol",
      "location": "PARAMETER_LOCATION_QUERY",
      "schema": {
        "type": "string",
        "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
      },
      "required": true
    }
  ]
}

Parameter Overrides

You can choose to set static values for dynamic parameters when you start a call. The model won’t see any parameters that you override. When creating a call simply pass in the overrides with each tool:

// Overriding dynamic parameter when starting a new call
// Always set the stock symbol to 'NVDA'
{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
    "parameterOverrides": {
      "symbol": "NVDA"
    }
  ]
}

Parameter overrides don’t make sense for temporary tools. Instead of overriding a dynamic parameter, use a static parameter instead.

Static Parameters

If you have parameters that are known at the time you create the tool, static parameters can be used.

// Adding a static parameter that always sends utm=ultravox
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "staticParameters": [
    {
      "name": "utm",
      "location": "PARAMETER_LOCATION_QUERY",
      "value": "ultravox"
    }
  ]
}

Automatic Parameters

Automatic parameters are used when you don’t want the model to specify the value and you don’t know the value when the tool is created. The primary use case for automatic parameters today is for using the call_id that is generated for the current call and then passing it as a unique identifier to your tool. Can also be used to get the current conversation history.

// Adding an automatic parameter to a profile creation tool
{
  "name": "create_profile",
  "description": "Creates a profile for the current caller",
  "automaticParameters": [
    {
      "name": "call_id",
      "location": "PARAMETER_LOCATION_QUERY",
      "knownValue": "KNOWN_PARAM_CALL_ID"
    }
  ]
}

Additional Information

Debugging

The Ultravox SDK enables viewing debug messages for any active call. These messages include tool calls. You can see a live demo of this on our website (make sure to toggle “Debug View” on at the bottom).

Tool Definition Schema

The definition object in the tool creation and update requests follows the BaseToolDefinition schema. Here’s a breakdown of its main components:

description (string): A clear, concise description of what the tool does.
dynamicParameters (array, optional): List of parameters that can be set by the AI model when using the tool. Each parameter is an object containining:
- name (string): The name of the parameter.
- location (string): Where the parameter is used (“PARAMETER_LOCATION_QUERY”, “PARAMETER_LOCATION_PATH”, “PARAMETER_LOCATION_HEADER”, “PARAMETER_LOCATION_BODY”).
- schema (object): JSON Schema definition of the parameter. This typically includes things like type, description, enum values, format, other restrictions, etc.
- required (boolean): Whether the parameter is required.
staticParameters (array, optional): List of parameters that are always set to a known, fixed value when the tool is used. These are unconditionally added when the tool is invoked. These parameters are not exposed to or set by the model. Example: you use an API for various things but want to track which requests come from your Ultravox app so you always append utm=ultravox to the query parameters.
automaticParameters (array, optional): Additional parameters automatically set by the system. Used when the value is not known when the tool is created but that will be known when the tool is called. Example: you want to use the unique call_id from ultravox as a key in your backend and you have the tool include call_id in the request body when your tool’s API is called.
requirements (object, optional): Any specific requirements for using the tool. Currently this is used for security (e.g. API keys or HTTP Auth).
http (object): Details for invoking the tool via HTTP. For server tools.
- baseUrlPattern (string): The base URL pattern for the tool, possibly with placeholders for path parameters.
- httpMethod (string): The HTTP method for the tool (e.g., “GET”, “POST”).
client (object): Declares the tool as a client tool. Exactly one of http or client must be set for a tool.

Best Practices for Creating Tools

Clear Naming: Choose a descriptive and unique name for your tool that clearly indicates its function.
Detailed Description: Provide a comprehensive description of what the tool does, including any important details about its usage or limitations. This and the name will help the model decide when and how to use your tool.
Precise Parameters: Define your dynamic parameters carefully, ensuring that the AI model has all the information it needs to use the tool effectively.
Error Handling: Consider how your tool will handle errors or unexpected inputs, and document this behavior in the tool’s description.
Iterate Faster: Use temporary tools when you are building your application. Persist durable tools in the system when things have stabilized.
Version Control: When updating tools, consider creating a new version (e.g., “stock_price_v2”) rather than modifying the existing tool. This allows testing of the new tool before impacting new calls made with the prior version of the tool.
Security: Be mindful of security when creating tools, especially when they interact with external APIs. Use appropriate authentication methods and avoid exposing sensitive information.
Testing: Thoroughly test your tools before deploying them in production conversations to ensure they function as expected.