Using Tools

​

Creating Your First Tool

Let’s look at creating a tool that sends an email with a summary of the conversation.

There are three steps:

{
  "systemPrompt": "You are a helpful assistant...",
  "model": "fixie-ai/ultravox",
  "selectedTools": [
    {
      "temporaryTool": {
        "modelToolName": "sendConversationSummary",
        "description": "Use this tool at the end of a conversation to send the caller a summary of the conversation.",
        "dynamicParameters": [
          {
            "name": "conversationSummary",
            "location": "PARAMATER_LOCATION_BODY",
            "schema": {
              "description": "A 2-3 sentence summary of the conversation.",
              "type": "string"
            },
            "required": true
          }
        ],
        "http": {
          "baseUrlPattern": "https://foo.bar/sendSummary",
          "httpMethod": "POST"
        }
      }
    }
  ]
}

const express = require('express');
const router = express.Router();

router.post('/sendSummary', async (req, res) => {
  try {
    const { conversationSummary } = req.body;

    // Send the email using our email provider
    sendEmail(conversationSummary);

    return res.status(200).json({
      message: 'Conversation summary sent successfully. Continue the conversation with the user.'
    });
  } catch (error) {
    return res.status(500).json({
      message: 'Internal server error',
      error: error.message
    });
  }
});

module.exports = router;

const updatedPrompt = `
  You're a friendly and fun guy. You like to chat casually while learning more about the person you're chatting with (name, hobbies, likes/dislikes). 
  
  Be casual. Be fun to chat with. Don't talk too much. Keep your sentences pretty short and fun. Let the user guide the conversation.
  
  As you chat, try and learn more about the person you are talking to such as their name, hobbies, and their likes/dislikes. 
  
  Once you have all the information, call the 'sendSummary' tool to send a summary of the conversation.
`;
{
  "systemPrompt": updatedPrompt,
  "model": "fixie-ai/ultravox",
  "selectedTools": [
      // Same as before
  ]
}

​

Server vs. Client Tools

You can implement your tools as either server (the tool’s functionality is exposed via a URL) or client (the tool’s functionality is implemented in your client application) tools. From the perspective of Ultravox Realtime, there is no difference between server or client tools. When the model chooses to call a tool, Ultravox will simply call the tool however it’s defined.

There are a couple things to note:

{
  "systemPrompt": "You are a helpful assistant...",
  "selectedTools": [
    {
      "temporaryTool": {
        "modelToolName": "sendEmail",
        ...
        "http": {
          "baseUrlPattern": "https://foo.bar/sendEmail",
          "httpMethod": "POST"
        }
      }
    },
    {
      "temporaryTool": {
        "modelToolName": "updateScreen",
        ...
        "client": {}
      }
    }
  ]
}

​

Tool Authentication

Ultravox has rich support for tools auth. When creating a tool, you must specify what is required for successful auth to the backend service.

Three methods for passing API keys are supported and are used when creating the tool:

You then pass in the key(s) in the authTokens property of selectedTools when creating a call.

• Query Parameter
• Header
• HTTP Authentication

// Create a tool that uses a query parameter called 'apiKey'
{
  "name": "stock_price"
  "definition": {
    "description": "Get the current stock price for a given symbol",
    "requirements": {
      "httpSecurityOptions": {
        "options": [
          "requirements": {
            "mySeviceApiKey": {
              "queryApiKey": {
                "name": "apiKey"
              }
            }
          }
        ]
      }
    }
  }
}

// Pass the API key during call creation
{
  "model": "fixie-ai/ultravox-70B"
  "systemPrompt": ...
  "selectedTools": [
    {
      "toolName": "stock_price"
      "authTokens": {
        "myServiceApiKey": "your_token_here"
      }
    }
  ]
}

​

Tool Parameters

Tool parameters define what gets passed in to your backend service when the tool is called. When creating a tool, parameters are defined as one of three types:

​

Dynamic Parameters

Dynamic parameters will have their values set by the model. Creating a dynamic parameter on a tool looks like this:

// Adding a dynamic parameter to a stock price tool
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "dynamicParameters": [
    {
      "name": "symbol",
      "location": "PARAMETER_LOCATION_QUERY",
      "schema": {
        "type": "string",
        "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
      },
      "required": true
    }
  ]
}

​

Parameter Overrides

You can choose to set static values for dynamic parameters when you start a call. The model won’t see any parameters that you override. When creating a call simply pass in the overrides with each tool:

// Overriding dynamic parameter when starting a new call
// Always set the stock symbol to 'NVDA'
{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
    "parameterOverrides": {
      "symbol": "NVDA"
    }
  ]
}

Parameter overrides don’t make sense for temporary tools. Instead of overriding a dynamic parameter, use a static parameter instead.

​

Static Parameters

If you have parameters that are known at the time you create the tool, static parameters can be used.

// Adding a static parameter that always sends utm=ultravox
{
  "name": "stock_price",
  "description": "Get the current stock price for a given symbol",
  "staticParameters": [
    {
      "name": "utm",
      "location": "PARAMETER_LOCATION_QUERY",
      "value": "ultravox"
    }
  ]
}

​

Automatic Parameters

Automatic parameters are used when you don’t want the model to specify the value and you don’t know the value when the tool is created. The primary use case for automatic parameters today is for using the call_id that is generated for the current call and then passing it as a unique identifier to your tool. Can also be used to get the current conversation history.

// Adding an automatic parameter to a profile creation tool
{
  "name": "create_profile",
  "description": "Creates a profile for the current caller",
  "automaticParameters": [
    {
      "name": "call_id",
      "location": "PARAMETER_LOCATION_QUERY",
      "knownValue": "KNOWN_PARAM_CALL_ID"
    }
  ]
}

​

Temporary vs. Durable Tools

Ultravox supports two types of tools: temporary and durable. There is much more information below but there are a few things to consider right upfront:

​

Temporary Tools

Temporary tools are created each time you create a new Call and exist exclusively within the context of that call. (Temporary tools aren’t visible in the List Tools response for example.)

Iteration is faster when using temporary tools because you don’t have to create/update/delete tools as you build out your application. You can simply adjust the JSON in the request body and start a new call.

​

Creating & Using Temporary Tools

Temporary tools are defined and passed in the request body of the Create Call endpoint. They are available during the current call.

{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "temporaryTool": {
      "modelToolName": "stock_price",
      "description": "Get the current stock price for a given symbol",
      "dynamicParameters": [
        {
          "name": "symbol",
          "location": "PARAMETER_LOCATION_QUERY",
          "schema": {
            "type": "string",
            "description": "Stock symbol (e.g., AAPL for Apple Inc.)"
          },
          "required": true
        }
      ],
      "http": {
        "baseUrlPattern": "https://api.stockmarket.com/v1/price",
        "httpMethod": "GET"
      }
    }
  ]
}

​

Durable Tools

In addition to temporary tools, Ultravox supports the creation of durable tools. There is no functional difference between durable and temporary tools within the context of a call.

Durable tools are persisted and can be reused across calls or applications. They shine once you have things dialed in, when you want to share tools across multiple applications, or if you have split responsibilities on the team.

​

Creating Durable Tools

The /tools endpoint in the Ultravox API is for working with durable tools. You create durable tools either by uploading an OpenAPI spec or via the request body in the Create Tool endpoint. Your OpenAPI spec must be either json or yaml format.

​

Using Durable Tools

To use a durable tool in a call, set the toolName or toolId field instead of using a temporaryTool.

For example:

// Request body for creating an Ultravox call with a durable tool
{
  "model": "fixie-ai/ultravox-70B",
  "systemPrompt": ...
  "selectedTools": [
    "toolName": "stock_price",
  ]
}

​

Additional Information

​

Changing Call State

For most tools, the response will include data you want the model to use (e.g. the results of a lookup). However, Ultravox has support for special tool actions that can end the call or change the call stage.

These tool actions require setting a special response type.

How you set the response type depends on if you are using a server/HTTP or a client tool:

Server/HTTP Tools

Server tools must respond with the X-Ultravox-Response-Type header set to either hang-up or new-stage

Client Tools

For client tools, set responseType="hang-up" or responseType="new-stage" on your ClientToolResult object.

​

Debugging

The Ultravox SDK enables viewing debug messages for any active call. These messages include tool calls.

​

Tool Definition Schema

The definition object in the tool creation and update requests follows the BaseToolDefinition schema.

Here’s a breakdown of its main components:

• description (string): A clear, concise description of what the tool does.
• dynamicParameters (array, optional): List of parameters that can be set by the AI model when using the tool. Each parameter is an object containining:
  • name (string): The name of the parameter.
  • location (string): Where the parameter is used (“PARAMETER_LOCATION_QUERY”, “PARAMETER_LOCATION_PATH”, “PARAMETER_LOCATION_HEADER”, “PARAMETER_LOCATION_BODY”).
  • schema (object): JSON Schema definition of the parameter. This typically includes things like type, description, enum values, format, other restrictions, etc.
  • required (boolean): Whether the parameter is required.
• staticParameters (array, optional): List of parameters that are always set to a known, fixed value when the tool is used. These are unconditionally added when the tool is invoked. These parameters are not exposed to or set by the model. Example: you use an API for various things but want to track which requests come from your Ultravox app so you always append utm=ultravox to the query parameters.
• automaticParameters (array, optional): Additional parameters automatically set by the system. Used when the value is not known when the tool is created but that will be known when the tool is called. Example: you want to use the unique call_id from ultravox as a key in your backend and you have the tool include call_id in the request body when your tool’s API is called.
• requirements (object, optional): Any specific requirements for using the tool. Currently this is used for security (e.g. API keys or HTTP Auth).
• http (object): Details for invoking the tool via HTTP. For server tools.
  • baseUrlPattern (string): The base URL pattern for the tool, possibly with placeholders for path parameters.
  • httpMethod (string): The HTTP method for the tool (e.g., “GET”, “POST”).
• client (object): Declares the tool as a client tool. Exactly one of http or client must be set for a tool.

​

Best Practices for Creating Tools

1. Clear Naming: Choose a descriptive and unique name for your tool that clearly indicates its function.

2. Detailed Description: Provide a comprehensive description of what the tool does, including any important details about its usage or limitations. This and the name will help the model decide when and how to use your tool.

3. Precise Parameters: Define your dynamic parameters carefully, ensuring that the AI model has all the information it needs to use the tool effectively.

4. Error Handling: Consider how your tool will handle errors or unexpected inputs, and document this behavior in the tool’s description.

5. Iterate Faster: Use temporary tools when you are building your application. Persist durable tools in the system when things have stabilized.

6. Version Control: When updating tools, consider creating a new version (e.g., “stock_price_v2”) rather than modifying the existing tool. This allows testing of the new tool before impacting new calls made with the prior version of the tool.

7. Security: Be mindful of security when creating tools, especially when they interact with external APIs. Use appropriate authentication methods and avoid exposing sensitive information.

8. Testing: Thoroughly test your tools before deploying them in production conversations to ensure they function as expected.