Which AI crawlers actually ignore robots.txt: Bytespider, DeepSeekBot and LAIONDownloader checked

A community registry lists all three of these as not respecting robots.txt. That is accurate for two of them in a way you can actually act on, and it hides something different, and more useful to know, about the third.

Where the claim actually comes from

Search for whether any of these three respects robots.txt and you land on a crowd of crawler-directory blog posts, most of them repeating each other without saying where the claim originated. The source most of them are quoting, whether they say so or not, is the same community-maintained registry this site's own census work already draws from: ai-robots-txt/ai.robots.txt on GitHub. Its own comparison table lists Respects robots.txt as No for Bytespider (operator ByteDance), DeepSeekBot (operator DeepSeek) and LAIONDownloader (operator LAION).

Bytespider and DeepSeekBot: real companies, no published policy to check the claim against

For a crawler like GPTBot or ClaudeBot, you can weigh a claim about robots.txt behavior against the operator's own published crawler documentation. ByteDance and DeepSeek do not give you that option. A direct check of bytedance.com turns up no crawler or robots.txt policy page anywhere on the site. DeepSeekBot's own user-agent string names its identifying URL as deepseek.com/about; fetched directly, that page returns a 404, and deepseek.com's current navigation has no crawling or bot policy page either. So for these two, the registry's classification is not something you can weigh against an official statement that says otherwise, because no official statement exists to weigh it against. What is left is the registry's classification and your own server logs. Treat robots.txt for either one the same way this site's guide on Cloudflare's AI crawler defaults already treats an unenforced preference: worth writing, not worth trusting on its own. A server or CDN-level block on the user agent is the lever that actually holds.

LAIONDownloader is a different problem: there may be no single operator to comply

LAION's own FAQ says plainly that LAION does not do the crawling itself. Common Crawl did that in the past, and Common Crawl respected robots.txt. LAION's own description of its role is analyzing data Common Crawl already collected and evaluating the images referenced in it against their alt text. Actually fetching those images, when someone wants to reconstruct a dataset locally from LAION's published URL list, happens through a separate tool, most commonly img2dataset, an open-source downloader built for exactly that. That tool's own documentation does not mention robots.txt at all, checking it or ignoring it. What it does check is a set of HTTP response headers, X-Robots-Tag: noai, noindex, noimageai and noimageindex, skipping images that serve any of those by default.

That makes the registry's LAIONDownloader entry true in a narrow sense and misleading in a broader one. It does not describe one continuously operating bot with a company behind it that could choose to start honoring your file. It describes a name attached to whatever script an individual researcher runs against a public list of image URLs, and that script sets its own user agent and decides its own behavior. A Disallow rule aimed at that name only reaches a script that both identifies itself honestly and chooses to check your file, and nothing requires either.

What actually reaches each one

• Bytespider and DeepSeekBot: robots.txt documents your intent and costs nothing to add, but treat it as unenforced. A server or CDN-level block on the user agent, the same distinction this site's Content-Signal guide draws between writing a preference and enforcing one, is what actually holds.
• LAIONDownloader, and anything using img2dataset's defaults: robots.txt will not reach it, since there may be no live requester checking your file at fetch time at all. The header-level opt-outs img2dataset already checks, noai and noimageai on the images themselves, are the closer thing to a working lever, and even those only bind tools that choose to check them, the same limit this site's guide on noindex and AI training opt-out found for the noai meta tag.
• Either way, check what is actually being served rather than what you intended to write, with the checker on this site.

Related: Cloudflare's AI crawler defaults covers the write-versus-enforce distinction in more depth, does noindex block AI training covers the header and meta-tag opt-outs that behave the same way LAIONDownloader's problem does, and the crawler census has the numbers on how many sites already block Bytespider by name.