Robot.txt is a file to the direction to the web crawler how to crawl their page or website. Robot.txt is one of the protocol to regulate how the crawler, crawling your content, pages, and shows the users. This also instructed the robots how search engine treats the links.
Robot.txt files instruct where the user agent can or can not crawl the website. This indicates allowing and disallowing to crawl your webpage.
-The file must be placed your top-level directory
-This file named robot.txt
-The robot.txt files must be publicly available to see.
Watch this video and learn how to robot.txt files place in a right place.
robots.txt- রোবটস টেক্সট ফাইলটি এসইও এর জন্যে কাজে লাগানোর ক্ষেত্রে এই ভিডিও টিউটোরিয়াল থেকে আপনারা যা যা শিখতে পারবেন ঃ
– How to write perfect robots.txt file.
– How to allow or disallow specific folder or a directory of a website.
– How to block similar type of pages to search engines.
– What is crawl delay and how to write it down on a robots.txt file.
Subscribe me on YouTube for latest Bangla tutorials. Choose HD in YT settings for better viewing. Thank You. 🙂