Where do I get the API token?

Log in to app.oncrawl.com, click your profile icon then Account Settings, scroll to API Access Tokens and generate or copy a token. Paste it into the sidebar; it is used as a bearer token for every request.

Why does my crawl not appear in the dropdown?

Data Extraction, Aggregations, Crawl Comparison and Link Export only list crawls with status "done". Running, paused or failed crawls can be managed in the Crawl Management module but cannot be queried. Lookups are also cached for 5 minutes, so a crawl that just finished may take a few minutes to show up.

Do preset queries cover every crawled URL?

No. Every preset filters to fetched pages, and most content, canonical and link presets additionally require status 200, so for example Pages Without Title or Orphan Pages only report 200 pages and exclude redirects and errors. All Fetched Pages is the preset without a status filter.

Are there export size limits?

Full exports call OnCrawl's export endpoint with a 300 second timeout, so very large result sets can time out. Use the optional URL filter (a contains match on the URL) to split big extractions into smaller chunks. OnCrawl returns semicolon separated CSV; the app detects semicolon versus comma automatically before parsing.

What can aggregations do?

One grouping field per query (status code, depth, URL segments, content type, canonical evaluation and a few others) with a URL count per bucket, optionally filtered by the same OQL as the presets. It is a count only aggregation; there are no sums or averages.

OnCrawl API Suite: free Technical SEO tool

#about

OnCrawl API v2 with bearer token authentication.

7 modules: Data Extraction (35+ preset OQL queries), Crawl Management (launch/pause/resume/cancel), Aggregations (groupBy queries), Crawl Comparison (new/lost/changed pages), Project CRUD, Scheduling (daily/weekly/monthly), Link Export.

60+ data fields in 14 categories.

TTL caching.

Streamlit app

Platform: Browser-based (no installation required)
Input: OnCrawl API token (bearer authentication)Workspace and project selectionCrawl selection for queries
Output: Crawl data with intelligent CSV parsing. Aggregation results with count sorting. Comparison exports with page status (new/lost/changed).
Category: Technical SEO

Use cases

Technical SEO audits - 35+ issue presets
Crawl automation - launch and schedule crawls
Change tracking - compare crawls over time
Link analysis - export internal/external links
Project management - create projects via API

#features

35+ preset OQL queries with custom builder
Crawl lifecycle: launch, pause, resume, cancel
Crawl comparison: new, lost, changed pages
Aggregation normalisation (cols/rows to buckets)
Schedule creation: daily/weekly/monthly
60+ data fields across 14 categories

#how-to-use

Enter your OnCrawl API token
Select workspace and project from dropdowns
Choose a module: Data Extraction, Crawl Management, Aggregations, Comparison, Projects, Scheduling, or Link Export
Use presets or build a custom OQL query
Download CSV (auto delimiter detection) or Excel

#faq

Where do I get the API token?: Log in to app.oncrawl.com, click your profile icon then Account Settings, scroll to API Access Tokens and generate or copy a token. Paste it into the sidebar; it is used as a bearer token for every request.
Why does my crawl not appear in the dropdown?: Data Extraction, Aggregations, Crawl Comparison and Link Export only list crawls with status "done". Running, paused or failed crawls can be managed in the Crawl Management module but cannot be queried. Lookups are also cached for 5 minutes, so a crawl that just finished may take a few minutes to show up.
Do preset queries cover every crawled URL?: No. Every preset filters to fetched pages, and most content, canonical and link presets additionally require status 200, so for example Pages Without Title or Orphan Pages only report 200 pages and exclude redirects and errors. All Fetched Pages is the preset without a status filter.
Are there export size limits?: Full exports call OnCrawl's export endpoint with a 300 second timeout, so very large result sets can time out. Use the optional URL filter (a contains match on the URL) to split big extractions into smaller chunks. OnCrawl returns semicolon separated CSV; the app detects semicolon versus comma automatically before parsing.
What can aggregations do?: One grouping field per query (status code, depth, URL segments, content type, canonical evaluation and a few others) with a URL count per bucket, optionally filtered by the same OQL as the presets. It is a count only aggregation; there are no sums or averages.

/contact

Want me to run this for you? I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Send me a message →