If you’ve received a message that our crawler was banned from your site, either through your robots.txt file or by a X-Robots-Tag HTTP header on your page, there are few things you can check