Troubleshooting Chatbot Website Crawling Issues and Errors
Hi, I'm just trying to create my first website support chatbot, with our website (https://www.doorsonlineuk.co.uk) as the Knowledge Base. I want the whole site to be crawled, but after a few pages I get a ton of "Network Error RESPONSE HEADERS: Undefined" errors listing up the right hand side of the page. I assume this is our server or Cloudflare detecting an unknown bot/over-zealous crawling and blocking Lindy's crawler. Can someone tell me if I'm correct and if so, what IP/user-agent do I need to whitelist to get the whole thing crawled? (I intend to crawl by XML sitemap rather than web pages as otherwise there will be a ton of query parameter URLs getting crawled and I can't see a way to set rules for that kind of stuff in Lindy's knowledge base crawl).
