Using Tools

​

Built-in Tools

Ultravox Realtime currently includes the following built-in tools.

Aside from being publicly visible, there’s nothing special about built-in tools. If you use list tools, you’ll see they use the same definitions you can create yourself.

​

Using a Built-in Tool

You give your AI agent access tools when you create the call or create a new call stage.

// Example request body for creating a call with a built-in tool
{
  "systemPrompt":"You are a helpful assistant. When the call naturally wraps up, use the 'hangUp' tool to end the call.",
  "selectedTools":[
    { "toolName": "hangUp" }
  ]
};

// The toolId can also be used
{
  "systemPrompt":"You are a helpful assistant. When the call naturally wraps up, use the 'hangUp' tool to end the call.",
  "selectedTools":[
    { "toolId": "56294126-5a7d-4948-b67d-3b7e13d55ea7" }
  ]
};

Note: Using built-in tools is the same as using any other custom durable tool that you have created except for one difference: you can override built-in tools by using the same name.

For example, if you created a durable tool named “hangUp” and then provide that tool by name (i.e. not by the toolId), then your tool would be used instead of the built-in hangUp tool.

​

Viewing Tools

To view all tools available to you, including built-in tools, use the List Tools API.

​

Custom Tools

Custom tools enable you to communicate with the outside world. Anything that you can do in a function can now be done by your agent via a custom tool.

​

Creating Your First Custom Tool

Let’s look at creating a tool that sends an email with a summary of the conversation.

There are three steps:

// Creating a tool called 'sendConversationSummary'
//
// A 'string' parameter named 'conversationSummary'
// is passed in the body of a POST request to https://foo.bar/sendSummary
{
  "systemPrompt": "You are a helpful assistant...",
  "selectedTools": [
    {
      "temporaryTool": {
        "modelToolName": "sendConversationSummary",
        "description": "Use this tool at the end of a conversation to send the caller a summary of the conversation.",
        "dynamicParameters": [
          {
            "name": "conversationSummary",
            "location": "PARAMETER_LOCATION_BODY",
            "schema": {
              "description": "A 2-3 sentence summary of the conversation.",
              "type": "string"
            },
            "required": true
          }
        ],
        "http": {
          "baseUrlPattern": "https://foo.bar/sendSummary",
          "httpMethod": "POST"
        }
      }
    }
  ]
}

const express = require('express');
const router = express.Router();

router.post('/sendSummary', async (req, res) => {
  try {
    const { conversationSummary } = req.body;

    // Send the email using our email provider
    sendEmail(conversationSummary);

    return res.status(200).json({
      message: 'Conversation summary sent successfully. Continue the conversation with the user.'
    });
  } catch (error) {
    return res.status(500).json({
      message: 'Internal server error',
      error: error.message
    });
  }
});

module.exports = router;

const updatedPrompt = `
  You're a friendly and fun guy. You like to chat casually while learning more about the person you're chatting with (name, hobbies, likes/dislikes).

  Be casual. Be fun to chat with. Don't talk too much. Keep your sentences pretty short and fun. Let the user guide the conversation.

  As you chat, try and learn more about the person you are talking to such as their name, hobbies, and their likes/dislikes.

  Once you have all the information, call the 'sendSummary' tool to send a summary of the conversation.
`;
{
  "systemPrompt": updatedPrompt,
  "selectedTools": [
      // Same as before
  ]
}

​

Tool Implementations

Every tool needs exactly one implementation. So far we’ve used http tools, but client and dataConnection tools are also possible.

There are a couple things to note:

​

Tool Requirements

When you define a tool, you can also specify requirements for agents or calls that use the tool. In particular, you can detail the authentication requirements and any parameters that must be overridden with static values.

​

Tool Authentication

Ultravox has rich support for tool auth. When creating a tool, you must specify what is required for successful auth to the backend service.

Three methods for passing API keys are supported and are used when creating the tool:

Your tool can specify multiple options for fulfilling auth requirements (for example if your server allows either query or header auth). Each option may also contain multiple requirements, for example if your server requires both a user_id and an auth_token for that user.

When defining an agent or creating a call, you pass in the key(s) in the authTokens property of selectedTools. If the tokens you provide satisfy multiple options, the first non-empty option whose requirements are all satisfied will be used. An unauthenticated option, if present, will only be used if no other option can be satisfied.

// Create a tool that uses a query parameter called 'apiKey'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "queryApiKey": {
                "name": "apiKey"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
// Requests will include ?apiKey=your_token_here in the url
{
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

// Create a tool that uses a query parameter called 'apiKey'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "queryApiKey": {
                "name": "apiKey"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
// Requests will include ?apiKey=your_token_here in the url
{
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

// Create a tool that uses an HTTP Header named 'X-My-Header'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "headerApiKey": {
                "name": "X-My-Header"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
// Requests will include the header "X-My-Header: your_token_here"
{
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

// Create a tool that uses HTTP Authentication scheme 'Bearer'.
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "myServiceApiKey": {
              "httpAuth": {
                "scheme": "Bearer"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
// Requests will include the header "Authorization: Bearer your_token_here"
{
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

​

Required parameter overrides

Sometimes your tool will require a parameter to function that you need to have defined when the call is created instead of having the model come up with a value. In these cases, you can require that the parameter be overridden at call creation. For example, the built-in queryCorpus tool requires the corpus id to be specified during call creation.

More information about parameters and their overrides is below.

​

Tool Parameters

Tool parameters define what gets passed to your implementation when the tool is called. When creating a tool, parameters are defined as one of three types:

​

Dynamic Parameters

Dynamic parameters will have their values set by the model. Creating a dynamic parameter on a tool looks like this:

// Adding a dynamic parameter to a stock price tool
// The parameter will be named 'symbol' and will be passed as a query parameter
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "dynamicParameters": [
    {
      "name": "symbol",
      "location": "PARAMETER_LOCATION_QUERY",
      "schema": {
        "type": "string",
        "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
      },
      "required": true
    }
  ]
}

Parameter Overrides

You can choose to set static values for dynamic parameters when you create an agent or start a call. The model won’t see any parameters that you override. When creating a call simply pass in the overrides with each tool, as below. You should also consider overriding the tool name or description to give the model a more specific understanding of what the tool will do in this case.

// Overriding dynamic parameter when starting a new call
// Always set the stock symbol to 'NVDA'
{
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
    "nameOverride": "nvidia_stock_price",
    "descriptionOverride": "Looks up the current stock price for Nvidia."
    "parameterOverrides": {
      "symbol": "NVDA"
    }
  ]
}

​

Static Parameters

If you have parameters that are known at the time you create the tool, static parameters can be used.

// Adding a static parameter that always sends utm=ultravox
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "staticParameters": [
    {
      "name": "utm",
      "location": "PARAMETER_LOCATION_QUERY",
      "value": "ultravox"
    }
  ]
}

Parameter Overrides

Static parameters can also be overridden when you create an agent or start a call. This is most useful with built-in tools. For example, the built-in queryCorpus tool allows you to statically override max_results.

​

Automatic Parameters

Automatic parameters are used when you want a consistent, predictable value (not generated by the model) but you don’t know the value when the tool is created.

Here are some of the supported automatic parameters:

// Adding automatic parameters to a profile creation tool
// There are two parameters added:
// 'call_id' which is sent as a query param
// 'conversation_history' which is sent in the request body
{
  "name": "create_profile",
  "description": "Creates a profile for the current caller",
  "automaticParameters": [
    {
      "name": "call_id",
      "location": "PARAMETER_LOCATION_QUERY",
      "knownValue": "KNOWN_PARAM_CALL_ID"
    },
    {
      "name": "conversation_history",
      "location": "PARAMETER_LOCATION_BODY",
      "knownValue": "KNOWN_PARAM_CONVERSATION_HISTORY"
    }
  ]
}

You can find more on Automatic Parameters in the Base Tool Definition.

​

Temporary vs. Durable Tools

Custom tools come in two varieties: temporary and durable. There is much more information below but there are a few things to consider right upfront:

​

Temporary Tools

Temporary tools are created each time you create a new Call and exist exclusively within the context of that call. (Temporary tools aren’t visible in the List Tools response for example.)

Iteration is faster when using temporary tools because you don’t have to create/update/delete tools as you build out your application. You can simply adjust the JSON in the request body and start a new call.

{
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_QUERY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "http": {
        "baseUrlPattern": "https://api.stockmarket.com/v1/price",
        "httpMethod": "GET"
      }
    }
  ]
}

​

Durable Tools

In addition to temporary tools, Ultravox supports the creation of durable tools. There is no functional difference between durable and temporary tools within the context of a call, but durable tools are persisted and can be reused across calls or applications. They shine once you have things dialed in, when you want to share tools across multiple applications, or if you have split responsibilities on the team.

Use the /tools endpoints to work with durable tools.

To use a durable tool in a call, set the toolName or toolId field instead of using a temporaryTool.

For example:

// Request body for creating an Ultravox call with a durable tool
{
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
  ]
}

​

Dealing with latency

When adding your own tools, it’s important to keep in mind that there’s always a user actively waiting for your tool to respond. Tools need to be (or at least appear) fast to make sense in a real-time context.

​

Precomputable Tools

In order to make sure you can always understand why your tool was invoked, by default tool invocations are always included in the conversation history if they occur at all. To maintain this invariant when the model produces both text and a tool call, tools cannot be invoked until the agent is done speaking so that we know the agent won’t be interrupted.

This is essential for tools that modify state since there’s no good way to revert changes if the agent is interrupted. However, it’s obviously suboptimal for tools like queryCorpus where we’d like to look up information while the agent is speaking and simply ignore the response if the agent is interrupted. Tools like this can be marked precomputable.

Any tool marked precomputable will be speculatively invoked as soon as the model produces the tool call. When the model produces both text and the tool call, the tool’s latency can be masked by the agent speaking, but if the agent is interrupted there will be no record of the invocation.

In order to safely be marked precomputable, a tool should have three properties:

1. No state changes. For http tools, GET requests are usually safe while methods like POST are not.
2. No side effects. Even a GET request is not safe to precompute if it has a side effect! (It’s up to you to decide what counts here though. Side effects like logging probably don’t matter to you for example while any database write likely does.)
3. Idempotent. The tool must return the same result when called with the same parameters, regardless of when or how many times it is called.

If your tool meets these requirements, you can mark it precomputable using the corresponding field.

​

Timeouts

While tools are executing, the conversation is essentially frozen. The user can continue talking all they like, but the agent will never respond until after the tool invocation completes. (The agent does have access to anything the user said during tool execution once execution completes.)

To users this may feel like the call was disconnected. In order to avoid that, tools are limited to 2.5 seconds by default. If your tool needs longer than that (and you can’t make it faster!), you can increase the timeout up to 20 seconds by setting the tool’s timeout field. You can also reduce your tool’s timeout if you’d like. The value is a duration in seconds, like 5s for 5 seconds or 0.1s for 100 milliseconds.

For tools that take even longer, consider responding immediately and later using an input_text_message data message with deferResponse with the real tool result. This is easiest with a dataConnection implementation since data connections are also able to send input text messages (and the response is always deferred in that case). Keep in mind that the model will see whatever response you send back initially, so you’ll want to make it clear to the model what’s going on by initially responding with some text like “Tool started. The full response will be available soon.”

​

Changing Call State

For most tools, the response will include data you want the model to use (e.g. the results of a lookup). However, Ultravox has support for special tool actions that can end the call or change the call stage.

These tool actions require setting a special response type.

How you set the response type depends on your tool implementation. HTTP tools set the response type via the X-Ultravox-Response-Type header. Client and data connection tools should set the responseType field in their tool result message.

​

Controlling Agent Responses to Tools

By default, the agent speaks again immediately after a tool call. This is typically the desired behavior for tools that gather information since the agent can immediately respond based on the information retrieved.

However, this may make less sense for other tools. For example, if your agent is gathering information for the user and you have a tool that allows the agent to store what’s been gathered so far, you may want the agent to speak either before or after the tool but not both.

Ultravox Realtime allows you to define how the agent reacts after a tool call by setting the agent reaction. A default can be set on the tool itself or you can use either the X-Ultravox-Agent-Reaction header (for http tools) or the agent_reaction field on the tool result message (for client and data connection tools) similar to how you’d set a response type (see above).

​

Debugging

The Ultravox SDK enables viewing debug messages for any active call. These messages include tool calls.

​

Best Practices

1. Clear Naming: Choose a descriptive and unique name for your tool that clearly indicates its function.

2. Detailed Description: Provide a comprehensive description of what the tool does, including any important details about its usage or limitations. This and the name will help the model decide when and how to use your tool.

3. Precise Parameters: Define your dynamic parameters carefully, guiding the model to produce values your tool can use. On the implementation side, be careful to validate values and be permissive with allowed presence and formatting if possible.

4. Error Handling: Consider how your tool will handle errors or unexpected inputs, and document this behavior in the tool’s description.

5. Iterate Faster: Use temporary tools when you are building your application. Persist durable tools in the system when things have stabilized.

6. Version Control: When updating tools, consider creating a new version (e.g., “stock_price_v2”) rather than modifying the existing tool. This allows testing of the new tool before impacting new calls made with the prior version of the tool.

7. Security: Be mindful of security when creating tools, especially when they interact with external APIs. Use appropriate authentication methods and avoid exposing sensitive information.

8. Testing: Thoroughly test your tools before deploying them in production conversations to ensure they function as expected.