A dilemma most publishers and website owners face now is whether to allow AI search to use your writing in search results.
It would be a much easier decision if AI summaries were more generous with citing and linking to your original content.
However, that’s not the reality at present. Unlike the traditional blue links in search engine results, AI answers may or may not include a link. Even if a link is included, it’s not always prominent.
It’s still early days yet, but AI is rapidly becoming a fact of life and is affecting all publishers.
The difference between AI search and AI training
When you think of AI, ChatGPT, Gemini, and Claude are the big ones that instantly come to mind.
Even though they are dominant, the important distinction for website owners isn’t the AI tool itself, but how it accesses and uses your content.
The internet today is exploding with bots and crawlers, all scooping up vast amounts of data from websites to train new AI models and tools.
If you use Cloudflare, your security analytics are probably full of them. You have the option to block many AI training bots and crawlers if you don’t want your content used for training.
Cloudflare has a security listing of the most common crawlers, which can be blocked with a simple toggle.
For the rest, it’s a game of cat and mouse. If you discover a new crawler pulling data from your site, you need to create a blocking rule in Cloudflare or your “.htaccess” file.
But AI search crawlers pose a different dilemma. Search engines such as Google, Bing, and DuckDuckGo increasingly use crawled web content to provide AI-generated answers and summaries.
If you allow these crawlers to access your site, they may use your writing as a source for AI-powered search results. If you don’t want them to do this, you can use the controls each search engine provides to limit or block access.
Keep in mind that blocking the wrong crawler can also affect your standard search visibility, depending on the search engine and the control you use.
Google has recently given site owners the choice to allow or block AI search results. But in the fine print it says, “This control doesn’t affect AI training,” which adds another layer of complexity to the problem.
What’s the best approach to take with AI crawlers?
Every publisher or website owner needs to make decisions based on what is best for their potential discovery, organic traffic, and site security.
That means there is no overall best answer to the question of whether to block or allow AI search to use your writing.
Closing off access to every AI search and training crawler is difficult to do effectively. But where possible, blocking access to AI training bots and crawlers can be a logical move.
However, that becomes complicated when search engines use different bots for search, AI search, and training. Google is a good example of this complication.
Another consideration is that restricting access to AI crawlers now might be pointless if your site has already been crawled extensively by hundreds of bots for AI purposes.
Perhaps one aspect that is often overlooked is the load on site servers.
For small site owners on affordable shared hosting, high crawl rates can use up the bandwidth allocated to a hosting account. This can cause a site to exceed its limits or even face extra fees.
In this case, blocking or limiting crawlers and bots can be an appropriate measure to avoid possible downtime or extra charges.
With my site, I initially tried to block every crawler other than the main search engines.
But it was a waste of time and effort because as soon as I blocked one, a new one would appear. For months, I played the whack-a-mole game before giving up on the idea.
With so many new bots and crawlers becoming active, it was an impossible task.
Finally, I opted for a more balanced approach by creating firewall rules based on bot behavior while allowing full access to the major search engines.
It’s not a perfect solution by any means, but blocking or challenging bots and crawlers by paths and endpoints has considerably reduced bot activity on my server.
All you can do right now is use the options and tools you have at hand to establish some measure of control over access to your site.
What are the downsides of blocking bots?
If you’re determined and don’t want to allow AI search to use your writing under any circumstances, blocking is the only option.
But be careful, because you might risk losing traffic from search engines, or even possibly from social platforms.
That’s why the issue of AI use and permission doesn’t have a clear yes or no answer.
AI is now an everyday tool for internet users, which makes it an important reality for site owners.
Traditional search and the ten blue links aren’t disappearing, but AI discovery is becoming an increasingly important part of the search landscape. The two are working side-by-side for now, which means that trying to cut off one avenue might negatively affect the other.
We can all see crawler and bot activity on our sites, but that only tells us that they came, they copied, and they left.
It doesn’t tell us what they will do with our site content. And that is the biggest downside.
We just don’t know.
Summary
The answer to the question, should you allow AI search to use your writing, is at best, messy.
The only thing we know for certain is that there are many unknowns. That makes it impossible to know whether you should allow them or not.
As with any developing technology, change happens fast. And that’s doubly true for the rapid pace of updates and improvements to AI tools and search.
It doesn’t stand still. It’s a moving target. It’s a mystery.
These are three good reasons why it’s so difficult to decide what to do with your site.
Related Reading: Do You Publish For Readers, Search Engines, And AI?
English teacher, writer, and author. Passionate about grammar, creative writing, and self-publishing. Always exploring new tools and tech to help writers succeed.
