{"id":25969,"date":"2020-12-21T03:07:43","date_gmt":"2020-12-20T19:07:43","guid":{"rendered":"https:\/\/web.mwwsb.com.my\/pjci\/?post_type=kb&p=25969"},"modified":"2022-09-08T21:41:36","modified_gmt":"2022-09-08T13:41:36","slug":"create-a-robots-txt-file","status":"publish","type":"kb","link":"https:\/\/www.casbay.com\/guide\/kb\/create-a-robots-txt-file","title":{"rendered":"Create a Robots.txt File"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

A web robot\u2019s primary job is to scan websites and pages for information. They work tirelessly to collect data on behalf of search engines and other applications. For some, there is good reason to keep pages away from search engines. \u00a0Whether you want to fine-tune access to your site. Or want to work on a development site without showing up Google results. Once implemented the robots.txt file lets web crawlers know which parts they can collect information.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t

Creating a Robots.txt File<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

As being one of the first aspects analyzed by crawlers, the robots.txt file can be implemented on a page(s) or an entire site. To discourage search engines from showing details about your site. Through this article, we will be providing insight into how to use the robots.txt file as well as the syntax needed to keep these bots at bay.<\/p>

User-agent: *
Disallow: \/<\/code><\/p>

Let\u2019s break down the code below \u201cuser-agent\u201d pertains to the web crawlers and the * sign means all web crawlers. Consequently, the first line grabs attention by saying \u201cListen up all web crawlers!\u201d. We move onto our second line which lets the web crawler know its direction. The forward-slash (\/) stops the bots from searching all the pages on your site. You can also discourage information collected for one specific page, in this case. it is a map of our building layout. Since the design of our building does not need to searchable, with the command below. I can tell all bots to leave out the index of the buildinglayout.png photo while keeping it viewable to any guest that wants to view it.<\/p>

User-agent: *
Disallow: \/buildinglayout.png<\/code><\/p>

Contrary, if you would like for all search engines to collect information on all the pages in your site you can leave the Disallow section blank.<\/p>

User-agent: *
Disallow:<\/code><\/p>

There are many types of web crawlers (aka user-agents) that can be specified. Below is a chart of the most popular web crawlers followed by their associations. Furthermore, you can also instruct these bots to index a certain page by using Allow. shown in the example below. You can implement these web crawlers within your robots.txt file like so:<\/p>

User-agent:Googlebot
Allow: \/parkinglotmap.png
Disallow: \/buildinglayout.png<\/code><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

Crawlers table<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t

Mostly, sites don\u2019t automatically come with a robots.txt file (and isn\u2019t required) so you can create one using a text editor and upload the file to your root directory or any other directory. \u00a0Luckily, if you use the popular CMS, WordPress, and its helpful SEO plugin Yoast, you\u2019ll see a section within the admin window to create a robots.txt file.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t

Robots.txt File In WordPress<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

After logging into your WordPress backend (yourdomain.com\/wp-login.php) locate the SEO section and select Tools.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
Select File editor under \u201cTools\u201d<\/figcaption><\/figure>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

After clicking on the file editor link, you see a page that looks similar to the code used in the first of our article.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
Edit of content robot.txt<\/figcaption><\/figure>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

Our example keeps web bots from the WordPress login page, including the wp-includes directory while still allowing users and bots to see other pages of our site. Take note of the necessary ending slashes after the directory (but not needed when disallowing pages). After editing select the \u201csave changes to robots.txt\u201d button to activate the robots.txt file.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t

Click to read the article: Common HTTP errors and how to troubleshoot them<\/a><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

A web robot\u2019s primary job is to scan websites and pages for information. They work tirelessly to collect data on behalf of search engines and other applications. For some, there is good reason to keep pages away from search engines. \u00a0Whether you want to fine-tune access to your site. Or want to work on a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"site-sidebar-layout":"no-sidebar","site-content-layout":"page-builder","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"disabled","ast-breadcrumbs-content":"","ast-featured-img":"disabled","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}}},"kbtopic":[117],"kbtag":[106],"mkb_version":[],"_links":{"self":[{"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kb\/25969"}],"collection":[{"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/comments?post=25969"}],"version-history":[{"count":15,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kb\/25969\/revisions"}],"predecessor-version":[{"id":37267,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kb\/25969\/revisions\/37267"}],"wp:attachment":[{"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/media?parent=25969"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=25969"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/kbtag?post=25969"},{"taxonomy":"mkb_version","embeddable":true,"href":"https:\/\/www.casbay.com\/guide\/wp-json\/wp\/v2\/mkb_version?post=25969"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}