Ian L.

Commented on Troubleshooting Chatbot Website Crawling Issues an...·Posted in02 Support

Thanks for helping Alon, fingers crossed for a response. 🤞

Commented on Troubleshooting Chatbot Website Crawling Issues an...·Posted in02 Support

Hi Alon J., any luck with this? I have tried crawling other sites to see if the problem was specific to one, but they all have the same issue. I can't make use of Lindy until I can get a full site crawled (c.3500 pages)!

Commented on Troubleshooting Chatbot Website Crawling Issues an...·Posted in02 Support

Ian L.

OK, thanks for your help. 🙂

Commented on Troubleshooting Chatbot Website Crawling Issues an...·Posted in02 Support

Ian L.

Sure, no problem, here's a video of the errors flying up the page (I couldn't get a screenshot while they were rolling through!): https://www.awesomescreenshot.com/video/44811261?key=ad4695069c4efe3e509e39808d212489 The Lindy is https://chat.lindy.ai/doors-more/lindy/website-customer-support-68dc0da494bf5717c7cf5b41/editor The sitemap I'm trying to crawl is here: https://www.doorsonlineuk.co.uk/sitemap_index.xml

Posted in 02 Support·

Ian L.

Troubleshooting Chatbot Website Crawling Issues and Errors

Hi, I'm just trying to create my first website support chatbot, with our website (https://www.doorsonlineuk.co.uk) as the Knowledge Base. I want the whole site to be crawled, but after a few pages I get a ton of "Network Error RESPONSE HEADERS: Undefined" errors listing up the right hand side of the page. I assume this is our server or Cloudflare detecting an unknown bot/over-zealous crawling and blocking Lindy's crawler. Can someone tell me if I'm correct and if so, what IP/user-agent do I need to whitelist to get the whole thing crawled? (I intend to crawl by XML sitemap rather than web pages as otherwise there will be a ton of query parameter URLs getting crawled and I can't see a way to set rules for that kind of stuff in Lindy's knowledge base crawl).

7Comments