Web Scraping

Last updated: October 28, 2026

Some of what Iris needs to know lives on a website: a trust center, a help center, a security page. Add Web Content brings those pages into your Knowledge Map as files, so Iris can cite them. Sites that change can refresh themselves every week.

Add a web page

  1. Open Knowledge Map and click Add Web Content, between Upload Files and Add from Integrations. A window titled Upload A Web Page opens.

  2. Enter the URL. It has to start with http:// or https://.

  3. Open Tags and pick from the list. Tags are required, just like the URL, because they decide which answers can use the page. See Tags.

  4. Optionally set an Expiration Date (Optional). It applies to every page captured.

  5. Under Capture scope, choose how much to bring in, then click Submit.

Both URL and Tags carry an asterisk. This URL ends in /blog/, so the second scope reads Entire section. With no path it would read Entire site.

Both URL and Tags carry an asterisk. This URL ends in /blog/, so the second scope reads Entire section. With no path it would read Entire site.

Choose a capture scope

The URL you enter is the boundary. Iris never goes above it, so to capture one section of a site, enter that section's address.

Choice

What Iris captures

Just this page

"Captures only the URL above. No links are followed."

Entire site

Shown when the URL has no path, such as https://example.com. "Captures pages from across the whole site."

Entire section

Shown when the URL has a path, such as /blog/. "Captures pages from across /blog/ and nothing outside it."

There are no depth or page-count settings. Once you pick a multi-page scope, a note called How Iris chooses pages appears. On a multi-page capture, Iris skips duplicate and translated copies, leaves out utility pages such as search and error pages, and spreads its page budget across sections so one big section can't crowd out the rest. It stays on the site you entered, and skips pages with very little text.

Pick Entire site or Entire section and two more options appear:

  • Also import documents linked on this page. Iris imports up to 100 PDF and Word files linked from the page you entered. It does not look for documents on the other pages it captures.

  • Set up weekly auto-sync. Iris re-crawls the URL every week. This needs Manage Integrations. If you don't have it, the page still uploads, but the sync isn't created.

After you click Submit

Single pages usually finish within a minute, multi-page captures in a few. A Crawling row appears in your files with a progress message above the list. Dismiss it and keep working.

Each page becomes a file named after the page title, with a small link to the original under the name. Web shows in the Source column. To update one page by hand, open the menu at the end of its row and choose Refresh Content. That needs Upload Files and Refresh Files. If the page can't be fetched, the existing file stays as it was.

Weekly auto-sync

Iris runs web syncs on Sundays. Each run re-crawls up to 200 pages under the site's host and path. It updates pages that changed and adds new ones. It never deletes pages that disappeared from the site. Delete those files yourself. Deleting a file does not stop the sync, so a page still on the site may come back.

Manage syncs under Settings > Integrations > Sync rules. Web syncs share that list with integration rules, and a web row shows an Auto-sync label. Click the Open menu button at the end of the row. The only action on a web row is Remove, which stops future crawls and keeps the files already added.

To add a site later, click Add web sync. The Add sync rule button beside it is for integration folders. In Add Site to Web Sync, choose Existing crawled site or New URL, add tags, and click Add to Sync. Syncs added this way don't include linked documents. For integration rules, see Sync Rules.

Web syncs live in the same Sync rules list as your integration rules. Add web sync sits right beside Add sync rule.

Web syncs live in the same Sync rules list as your integration rules. Add web sync sits right beside Add sync rule.

When a page doesn't come in

  • "This URL cannot be imported. Only publicly reachable http(s) web addresses are supported." The address is private, internal, or not http or https. Intranet pages can't be reached from outside your network.

  • "This URL could not be validated right now." A temporary lookup problem. Try again later.

  • A file marked Failed, or a multi-page message such as Nothing scraped. The site blocked scraping or timed out.

  • Nothing appears at all. For a single page this is the common case. Iris quietly skips pages behind a login, paywall or bot check, pages that return an error, pages that load only through scripts, and pages with almost no text. No file is created and no error shows.

Each web file shows the page title, the original address under it, and Web in the Source column.

Each web file shows the page title, the original address under it, and Web in the Source column.

Here is what to do instead:

  • Open the page yourself, save it as a PDF, and upload it as a file.

  • If the content lives in a system you already connect, such as SharePoint or Confluence, use that integration.

  • Ask the site owner to allow access, or to publish the content somewhere public.

Who can do what

Permission

What it allows

Upload Files

Add web content.

Refresh Files

With Upload Files, use Refresh Content.

Delete Files

Delete a web file.

Manage Integrations

Weekly auto-sync and web syncs.

Related articles