This documentation provides guidance on how to prevent specific bots, such as GPTBot and CCBot, from crawling your website's content.
Blocking Specific Crawlers
To block specific crawlers, such as GPTBot and CCBot, from accessing your content, you can leverage the robots.txt file on your site. This file provides directives to web crawlers about which pages or sections should not be crawled.
Here's an example of how you can add directives for GPTBot and CCBot:
User-agent: GPTBot\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /
By including these lines in your robots.txt file, you instruct the specified crawlers not to crawl any part of your site. Most well-behaved crawlers, including GPTBot and CCBot, respect the directives in the robots.txt file.
Considerations
-
Respecting Directives: While most crawlers follow the rules set in the
robots.txtfile, it's important to note that these rules are not foolproof. Some crawlers may not adhere to the directives, and for guaranteed blocking, a server-side solution may be necessary. -
Server-Side Paywall: For an absolute block of crawlers, consider implementing a server-side paywall. This ensures that crawlers are unable to access content even if they do not adhere to the directives. More information about the different types of paywall implementation is available here.