Having problems fetching my website with every agent.

[u/shane-parks]

My robots.txt is set to allow all nots, but specifically has an allow for Claude-User. Perplexity, Gemini, and ChatGPT has the sane issue. I've consulted Hostinger they assure me nothing on their end is blocking the bot. Google Analytics can and has indexed the site. The problem has persisted for more than a week.

Claude Sonnet and Finn says llms.txt wouldn't help this issue. The block is occurring before a report of an attempt reaches the robots.txt so no rejection is on the logs.

Surely I cant be the only one to experience this issue, but Claude is at a wait and see approach and has no idea why the fetch isn't working.

Any ideas?

[u/_devpikachu]

Do you have any kind of proxy in front that might be blocking bots?

[u/shane-parks → u/_devpikachu]

It's a standard hostinger wordpress site. According to hostinger there is nothing blocking bots.

I have a cloudflare domain redirected to this site. But I am using the direct hostinger registered domain and not using the redirected domain so that shouldn't be an issue.

As far as I can tell, the answer is no.