User-agent: Specifies which search engine bots the rules apply to. Use "*" for all bots.
Allow/Disallow: Control which pages and directories search engines can crawl.
Crawl-delay: Sets the minimum time (in seconds) between requests from the same bot.
Sitemap: Points to your XML sitemap location for better indexing.
Host: Specifies the preferred domain (www or non-www) for your website.