AI crawler index

What is DuckDuckBot?

DuckDuckBot is DuckDuckGo's search engine crawler. DuckDuckGo's web crawler, which works to improve its search results.

At a glance

Operator	DuckDuckGo
Purpose	Search engine
User agent	`DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)`
Obeys robots.txt	Yes, according to the operator
Published IP ranges	https://duckduckgo.com/duckduckbot.json
How to verify it	DuckDuckGo publishes the IP addresses it crawls from; match the source IP against them.
Official documentation	https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot

Should you block DuckDuckBot?

It is a classic search engine crawler. Blocking it removes you from that search engine and from the products built on its index, so most sites should allow it.

DuckDuckGo says it respects robots.txt (WWW::RobotRules). Its AI-assisted answers use a separate crawler, DuckAssistBot.

How to block or allow DuckDuckBot in robots.txt

Block it everywhere:

User-agent: DuckDuckBot
Disallow: /

Allow it everywhere:

User-agent: DuckDuckBot
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

How to verify that a visit is really DuckDuckBot

Anyone can write DuckDuckBot in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

DuckDuckGo publishes its IP ranges as JSON. This command downloads the list and prints True if the IP is in it. Replace 203.0.113.7 with the IP from your logs; Python 3 comes with macOS and most Linux systems.

curl -s https://duckduckgo.com/duckduckbot.json \
  | python3 -c 'import sys, json, ipaddress as ip
a = ip.ip_address(sys.argv[1])
ranges = json.load(sys.stdin)["prefixes"]
print(any(a in ip.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))' 203.0.113.7

Other DuckDuckGo crawlers

DuckAssistBot: AI search

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.