Free SEO tool · Technical
Robots.txt Generator
Choose what search engines and AI crawlers can visit, add your sitemap, then copy or download your robots.txt.
robots.txt
Upload the file to the main folder of your website so it opens at yourdomain.com/robots.txt.
What does robots.txt do?
The robots.txt file tells crawlers which parts of your site they may visit. It lives at the root of your domain, for example example.com/robots.txt.
Common mistakes
- Blocking CSS or JavaScript files that Google needs to render the page.
- Leaving
Disallow: /in place after a site launch. - Using robots.txt to hide pages from Google. Blocked pages can still be indexed; use a noindex tag instead.
After uploading, test your rules with our robots.txt tester.
Frequently asked questions
Where do I upload robots.txt?
In the main (root) folder of your website, so it opens at yourdomain.com/robots.txt. WordPress creates a virtual one if no file exists.
Should I block AI crawlers?
Blocking training bots like GPTBot and CCBot stops your content being used to train models. Keep search bots like OAI-SearchBot and PerplexityBot allowed if you want to appear in AI answers.
Does robots.txt remove a page from Google?
No. Use a noindex tag or remove the page. Robots.txt only controls crawling.
More free tools
Meta Tag Generator
Write title and description tags that fit Google limits.
ON-PAGESERP Preview Tool
See how your page will look in Google results.
ON-PAGESchema Markup Generator
Create FAQ, LocalBusiness, Article and other JSON-LD schema.
ON-PAGESlug Generator
Turn titles into clean, SEO-friendly URL slugs.
CONTENTWord Counter
Count words, characters and reading time.
CONTENTCharacter Counter
Count characters and check limits for Google, X, Instagram and more.
CONTENTKeyword Density Checker
Spot over-used and missing keywords on a page.
CONTENTCase Converter
Switch text to UPPER, lower, Title or Sentence case.
CONTENTLorem Ipsum Generator
Generate placeholder text in paragraphs, sentences or words.
TECHNICALWebsite SEO Checker
Paste a URL, get the top issues holding it back.
TECHNICALPage Speed Checker
Test speed and Core Web Vitals on mobile and desktop.
TECHNICALBroken Link Checker
Find dead links on any page, internal and external.
TECHNICALHTTP Status Code Checker
Check status codes for up to 100 URLs at once.
TECHNICALRedirect Checker
Trace every redirect hop from a URL to its final page.
TECHNICALDomain Age Checker
Find a domain’s age, expiry date, registrar and WHOIS details.
TECHNICALRobots.txt Tester
Check if a URL is allowed or blocked for any crawler.
TECHNICALXML Sitemap Generator
Crawl a site and build a sitemap.xml file.
LOCAL SEOGoogle Review Link Generator
Make a direct Google review link and QR code.
LOCAL SEONAP Consistency Checker
Check your name, address and phone match across listings.
LOCAL SEOLocal Keyword Generator
Combine services and cities into local search keywords.
AI & PROMPTSAI Crawler Checker
See if GPTBot, ClaudeBot and PerplexityBot can read your site.
AI & PROMPTSllms.txt Generator
Create an llms.txt file that guides AI tools to your best pages.
SOCIAL MEDIAInstagram Hashtag Generator
Get niche, keyword and local hashtags for your next post.
SOCIAL MEDIAInstagram Bio Generator
Write a clear Instagram bio for your business in 150 characters.
SOCIAL MEDIAInstagram Caption Generator
Ready captions with hooks, calls to action and hashtags.
AI & PROMPTSAI Prompt Generator for Marketers
Build ready ChatGPT prompts for SEO, ads, emails and social.
AI & PROMPTSChatGPT Prompt Library
Copy-ready ChatGPT prompts for SEO, content, ads and email.
CONTENTContent Brief Generator
Enter a keyword, get titles, headings, questions and a full brief.
UTILITYImage Converter (JPG, PNG, WebP)
Convert images between JPG, PNG and WebP in your browser.
UTILITYFavicon Generator
Make a favicon set from an image, text or emoji.
UTILITYInvoice Generator
Make GST-ready invoices for freelancers and small businesses.
UTILITYPrivacy Policy Generator
Create a clear privacy policy for your website or app.
UTILITYTerms and Conditions Generator
Create terms and conditions for your website or online store.
UTILITYBulk URL Opener
Open many links or websites at once, in batches.
UTILITYImage Compressor
Shrink JPG, PNG and WebP images for a faster site.
UTILITYSocial Bookmarking Sites List
35+ high DA bookmarking sites. Filter, open in bulk or suggest one.
Tools show the problems. We fix them.
Send us your URL and we will reply with your three biggest issues, free.