robots.txt Reference

robots.txt Guide

All Search Engines

User-agent: *

Specific Search Engine

User-agent: Googlebot

Disallow: /google-blocked/

User-agent: bingbot

Disallow:

All Search Engines

User-agent: *

Block all URLs

User-agent: *

Disallow: /

Allow all URLs

User-agent: *

Disallow:

Block Directory

User-agent: *

Disallow: /folder

Allow File in Blocked Directory

Allow: /folder/file.pdf

Disallow: /folder/

Wild Card Blocks

User-agent: *

Disallow: /*.php

Disallow: /folder/*.jpg

Sitemaps

Sitemap: https://sabali.co/sitemaps/sitemap.xml

Sitemap: https://sabali.co/sitemaps/sitemap-images.xml

Google Robots.txt Support