If you own a website for your business, you’ve probably heard about search engine crawlers and the importance of files like robots.txt. But lately, there’s a new file making waves: llms.txt.
This simple text file gives you more control over how large language model (LLM) crawlers—think ChatGPT, Gemini, and others—access and use your website’s content.
Whether you run a small online store, a blog, or a large corporate site, understanding and using llms.txt can help protect your content, manage your online presence, and even drive more traffic.
This guide will walk you through everything you need to know, step by step, without confusing jargon—just clear advice you can use today.
What Is llms.txt and Why Should You Care?
Just like robots.txt tells search engines which pages they can and can’t crawl, llms.txt is a file you put on your website to tell LLM crawlers (the bots behind AI tools) what they can and cannot access or use from your site.
Why does this matter? Because LLMs often “learn” from public web content. If you don’t want your blog posts, product descriptions, or private information to end up in an AI chatbot’s answers, you need a way to say “no thanks.” That’s where llms.txt comes in.
Benefits of Using llms.txt
- Protect your content: Stop LLMs from scraping or using your website’s text.
- Control your brand: Decide what information is available to AI tools and what stays private.
- Stay ahead: Show your customers and partners that you value privacy and digital rights.
How Does llms.txt Work?
The llms.txt file is a plain text document you add to your website’s root directory (that’s the main folder where your homepage lives). LLM crawlers are starting to look for this file, just like search engines look for robots.txt.
Inside llms.txt, you list rules for different LLM crawlers. You can allow or disallow them from accessing certain parts of your site, or even block them entirely.
Step-by-Step: Creating Your Own llms.txt
Ready to take control? Here’s how you can create and set up your own llms.txt file, even if you’re not a tech expert.
1. Decide What You Want to Allow or Block
First, think about what you want LLM crawlers to do. Here are some common options:
- Block all LLM crawlers from your site
- Allow some, block others
- Block access to specific folders or pages (like your blog or customer area)
- Let them crawl everything
Take a few minutes to decide what feels right for your business.
2. List the LLM Crawlers You Want to Address
Each LLM crawler has a unique name, called a “user-agent.” Here are some common ones:
- OpenAI-UserAgent (for ChatGPT and related tools)
- Google-Extended (for Google’s Gemini)
- Anthropic-UserAgent (for Claude)
- PerplexityBot (for Perplexity AI)
- MetaAI (for Meta’s Llama)
You can find a more complete list online, but these are a good start.
3. Write Your llms.txt Rules
The format is simple. Here’s what each rule looks like:
| User-agent: [crawler name] Disallow: [URL path you want to block] Allow: [URL path you want to allow] |
If you want to block everything for a crawler, use:
| User-agent: OpenAI-UserAgent Disallow: / |
If you want to allow everything:
| User-agent: OpenAI-UserAgent Allow: / |
You can repeat this for each crawler.
Example: Block All LLMs From Your Entire Site
| User-agent: * Disallow: / |
The asterisk (*) means “all crawlers.”
Example: Allow Google-Extended, Block OpenAI-UserAgent
| User-agent: Google-Extended Allow: / User-agent: OpenAI-UserAgent Disallow: / |
Example: Block Only Your Blog Section
| User-agent: * Disallow: /blog/ |
This blocks all LLMs from accessing anything in your /blog/ folder.
4. Create the llms.txt File
Open a plain text editor (like Notepad on Windows, TextEdit on Mac, or any code editor). Copy and paste your rules into a new file.
Save the file as llms.txt (all lowercase).
5. Upload llms.txt to Your Website
You need to put llms.txt in your site’s root directory, which is the same place as your homepage (usually called index.html or index.php).
How you do this depends on your website setup:
- If you use WordPress: Use an FTP client (like FileZilla) or your hosting provider’s file manager.
- If you use a website builder (Wix, Squarespace, Shopify): check if they allow you to upload files to your root directory. Some platforms may not support this yet, so ask their support team.
- If you have a developer: Ask them to upload it for you.
Once uploaded, you should be able to visit https://yourwebsite.com/llms.txt in your browser and see your rules.
6. Test Your llms.txt File
After uploading, double-check that it’s working:
- Go to https://yourwebsite.com/llms.txt in your browser. You should see your file.
- Use online tools or LLM crawler documentation to see if your rules are recognized.
- Periodically check your server logs (or use analytics tools) to see if LLM bots respect your wishes.
Tips and Best Practices
- Keep it simple: Only include the rules you need. Don’t overcomplicate things.
- Update as needed: If new LLMs appear or your preferences change, edit your llms.txt and re-upload.
- Tell your team: Make sure anyone else who manages your website knows about llms.txt.
- Monitor your content: Occasionally, search for your content in AI tools to see if your rules are being followed.
Generate llms.txt Easily
If you want a shortcut, there are online tools that help you generate llms.txt files with just a few clicks. Simply enter your preferences, and the tool will create the file for you. Look for reputable options by searching for Generate llms.txt—this can save you time and help avoid mistakes.
Conclusion
Adding an llms.txt file to your website is a smart, simple step to take control of how your content is used by large language models. Whether you want to block, allow, or customize access, this file puts the power in your hands.
It’s easy to create, quick to upload, and shows your commitment to protecting your business and your customers.
Don’t wait for someone else to decide how your content is used. Take a few minutes today to set up your llms.txt and stay in charge.
Try Gushwork’s AI Visibility Suite today to see how AI models view your business, uncover where you’re missing out, and get clear steps to boost your visibility.
