Crawling Websites

​

Crawl Websites via Web App

1

Create New Source

• Go to Corpora in the Ultravox web application.
• Select the corpus that will hold the new source. If you don’t have one, create a new corpus.
• Click Add Source in the top right corner.

2

Add Details and URLs

• Add a Name and (optionally) a Description for the new source.
• Select Web.
• Set the Crawl Depth. See Crawl Depth to choose the right value.
• Click Add URL for each start URL.

3

Save

• Click Save and wait for your pages to be crawled and ingested. Larger crawls take longer.

​

Crawl Websites via API

CORPUS_ID="your-corpus-id"

curl -X POST \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Example Help Center",
    "crawl": {
      "startUrls": [
        "https://www.example.com/help",
        "https://support.example.com"
      ],
      "maxDepth": 2
    }
  }' \
  "https://api.ultravox.ai/api/corpora/$CORPUS_ID/sources"

​

Crawl Depth

• Depth 1 crawls /help only.
• Depth 2 crawls /help, /help/billing, /help/returns, /pricing, and /about.
• Depth 3 also crawls /help/refunds, /enterprise, and every other page linked from the depth 2 pages.

• Depth 1 doesn’t follow any links. To include the pages your start URL links to, use depth 2.
• Depth isn’t the number of folders in a URL. https://www.example.com/docs/guides/setup isn’t automatically “depth 3”. If your start page links straight to it, it’s depth 2.
• Menus, headers, and footers count as links. Any page in the site’s main menu is at depth 2 from a page that shows the menu. Starting at /help doesn’t keep the crawl inside /help.
• Every start URL is depth 1. Adding a page as a start URL is often a better way to reach it than raising the depth.
• The number of pages grows fast. If each page has 30 links, depth 3 can reach almost 1,000 pages. A crawl stops at maxDocuments (200 at most per source). The crawler finishes one level before starting the next, so the deepest level is the one that gets cut short.

​

Which Pages Get Crawled

• Each page is loaded in a headless browser, so JavaScript-heavy sites and single-page apps work.
• Linked documents such as PDFs, Word files, and PowerPoint decks are downloaded and ingested too, if they’re on an allowed host. They count toward maxDocuments.
• Every start URL’s host is allowed for the whole source, so the crawler follows links between all of them.

​

Websites Are Not Always Built Like They Look

• A help center on support.example.com or example.zendesk.com
• Developer docs on docs.example.com
• A blog on blog.example.com
• A store on shop.example.com or example.myshopify.com
• PDFs and downloads on a CDN or storage bucket, like cdn.example.com or example.s3.amazonaws.com
• Both example.com and www.example.com, which are two different hosts

• Redirects to another host. If https://example.com redirects to https://www.example.com, the links on the page you land on point to www.example.com, which isn’t an allowed host. Use the URL your browser ends up on as the start URL.
• Links that aren’t real links. The crawler follows <a href> links. It doesn’t hover or click, so menus that load their items only on hover or click, or buttons that navigate with JavaScript, aren’t followed.
• Pages nothing links to. The crawler doesn’t read sitemaps or use site search. Add pages like these as start URLs.
• Pages behind a login. The crawler can’t fill in login forms or use SSO. Sites protected by HTTP basic auth can be crawled (see Crawling Sites with Basic Auth). For other logins, download the content and add it as static documents instead.

​

Crawling Sites with Basic Auth

https://username:password@docs.example.com

• Add the credentials to every start URL on the protected host
• Percent-encode special characters in the username or password, such as @ (%40), : (%3A), / (%2F), and # (%23). For example, p@ss becomes p%40ss
• This works in both the web app and the API

​

Crawl Settings

​

Recrawling

​

Limits