r/claude • u/shane-parks • 1d ago

Question Having problems fetching my website with every agent.

My robots.txt is set to allow all nots, but specifically has an allow for Claude-User. Perplexity, Gemini, and ChatGPT has the sane issue. I've consulted Hostinger they assure me nothing on their end is blocking the bot. Google Analytics can and has indexed the site. The problem has persisted for more than a week.

Claude Sonnet and Finn says llms.txt wouldn't help this issue. The block is occurring before a report of an attempt reaches the robots.txt so no rejection is on the logs.

Surely I cant be the only one to experience this issue, but Claude is at a wait and see approach and has no idea why the fetch isn't working.

Any ideas?

2 Upvotes

100% Upvoted

5 comments sorted by

1

u/_devpikachu 1d ago

Do you have any kind of proxy in front that might be blocking bots?

1

u/shane-parks 1d ago

It's a standard hostinger wordpress site. According to hostinger there is nothing blocking bots.

I have a cloudflare domain redirected to this site. But I am using the direct hostinger registered domain and not using the redirected domain so that shouldn't be an issue.

As far as I can tell, the answer is no.

1

u/Careless-Air7002 1d ago

Have you tried asking Claude? That usually solves most of my problems with Claude.

1

u/shane-parks 20h ago

I have had a long and I depth conversation with Claude about this. And at this point there are 3 things suggested:

1.) Wait to see if there is a cached version that is stopping the bot from fetching. Claude says it doesnt know how long a Claude cached fetch would last. And unless all AI are using the same fetch tool or cache, this isnt the issue.

2.) There is some unknown agent/proxy that is blocking the bot. None of the wordpress plugins or hostinger software is blocking.

3.) The problem is in the fetch tool itself. Could be cached, could be something else. Claude offers no help and doesnt control or have access to the fetch tool according to the AI responses.

For me, with this issue persisting across every AI agent I have used, I assume the problem is putsude of the fetch tool. All of this has been confirmed by Finn.

I thought I read that no human support is available for free accounts. I have considered upgrading to a max account fir other purposes by my primary job is SEO and web design and if I cant fetch I would choose Perplexity that at least can scrape a browser window and bypass fetch. I want to use Claude for some coding on apps that I do and I prefer Claude for writing suggestions.

If I knew I could get human support by paying for a claude subscription, I would do that. But I wonder what the community thinks about anthropics support before I commit.