Robots.txt for Small Blogs: What to Block and What to Allow

Robots.txt for blogs is a crawl-control file at your site root. It tells cooperating crawlers which paths they may fetch—and which to skip. It does not replace a sitemap, and it is a weak tool if your real goal is “keep this URL out of Google.”

This guide is robots-specific. URL lists and hygiene belong in XML sitemaps for small blogs. Broader “why isn’t this indexed?” work lives in why Google isn’t indexing your pages. Crawl/render tooling sits in the technical SEO tools checklist.

Official baselines: Introduction to robots.txt and How to write and submit a robots.txt.

Table of contents

1. What robots.txt is (and is not)
2. The Allow-Block Decision Card
3. Safe allow and block examples for small blogs
4. Never starve rendering: CSS and JS
5. The optional Sitemap: line
6. A minimal starter file
7. Test before you tighten
8. FAQ

What robots.txt is (and is not)

Robots.txt is a public text file crawlers request first. Respectful bots read User-agent, Allow, and Disallow rules, then decide what to fetch.

It is not:

• An XML sitemap (include list of important URLs)
• A noindex tag or password wall
• A security lock—anyone can still open a public URL in a browser
• A ranking boost by itself

Google’s intro is blunt: use robots.txt mainly to manage crawl traffic, not to hide pages from Search. Linked URLs that are disallowed can still show as bare results. When you need a page out of the index, fix indexing controls—not a hopeful Disallow.

The Allow-Block Decision Card

This is the original framework for this article. For each path, pick one column. If you hesitate between columns, default to Allow on a small blog.

        ALLOW-BLOCK DECISION CARD
  ┌──────────────────┬───────────┬───────────┬──────────────┐
  │ Path / pattern   │ ALLOW     │ BLOCK     │ NOT robots   │
  ├──────────────────┼───────────┼───────────┼──────────────┤
  │ Public posts     │ ✓ default │           │              │
  │ Theme CSS/JS     │ ✓ always  │ ✗ never*  │              │
  │ /wp-admin/ etc.  │           │ ✓ usual   │              │
  │ Thank-you / cart │           │ ✓ usual   │ or noindex   │
  │ Private docs     │           │ weak alone│ password /   │
  │                  │           │           │ auth instead │
  │ “Hide from Google│           │ weak alone│ noindex or   │
  │  forever”        │           │           │ remove       │
  └──────────────────┴───────────┴───────────┴──────────────┘
  * unless the resource truly does not affect how the page reads

Run the card before pasting a “SEO robots.txt” you found in a forum. Most damage on small blogs comes from blocking too much, not too little.

Safe allow and block examples for small blogs

Usually allow (do not Disallow)

• / for public HTML posts and pages
• Theme and plugin asset folders that supply CSS/JS for the front end
• Your sitemap URL if you reference it elsewhere

Usually block (when those URLs exist)

• Admin and login paths (/wp-admin/, /wp-login.php, equivalent CMS routes)
• Account, cart, checkout, and thank-you flows you do not want crawled
• Internal site-search result URLs that create infinite parameter junk
• Staging subfolders accidentally left on the live host

Usually leave alone

• Broad Disallow: / “while redesigning” left on after launch
• Blanket Disallow: /wp-content/ that also kills CSS/JS
• Blocking feeds or author archives without a clear reason

Example mindset: a 40-post niche blog rarely needs a clever robots file. A short, boring file that allows content and blocks admin noise is enough.

Never starve rendering: CSS and JS

Google’s robots guidance calls out resource files. You may block unimportant assets if pages still make sense without them. If missing CSS or JS makes the page harder to understand, do not block those paths.

One mistake beginners often make: copying an old “block /wp-content/ for security” snippet. That folder often holds the stylesheets and scripts the renderer needs. HTML may still fetch; the rendered page looks incomplete.

After any robots edit, spot-check a post template in Search Console’s URL Inspection and look for blocked resources. If the rendered preview looks broken, open the Allow column on those assets again.

The optional Sitemap: line

You can add:

Sitemap: https://example.com/sitemap.xml

That line is optional. Google can learn your sitemap from Search Console alone. The robots reference mainly helps discovery for crawlers that look there. Keep the URL absolute, HTTPS, and matching the live sitemap—not a staging copy.

If the sitemap itself is disallowed, you create noise: Search Console expects a fetchable file. Fix allow rules before you chase “sitemap submitted” theater.

A minimal starter file

Many WordPress and static hosts already ship something close to:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Adjust for your CMS. Static blogs may only need User-agent: * with an empty disallow set, plus an optional Sitemap line. Prefer fewer rules you understand over a long borrowed list.

Syntax notes that bite small sites:

• One rule group at a time; know which User-agent you mean
• Trailing slashes and prefixes matter—/blog and /blog/ are not identical patterns
• Comments start with #; keep them short

Test before you tighten

1. Open https://yoursite.com/robots.txt signed out
2. Confirm it is plain text, not an HTML 404 theme page
3. Walk the Allow-Block Decision Card for every new Disallow
4. Re-inspect two important URLs after deploy
5. If indexing stalls on paths you meant to keep public, check disallow rules first—then return to the indexing guide rather than adding more blocks

Crawl control is a scalpel. On a small blog, less cutting is usually safer.

FAQ

What is robots.txt used for on a small blog?

It is a crawl-control file that suggests which paths crawlers may fetch. It manages crawl traffic and resource access—it is not a reliable way to hide pages from Google Search results.

Is robots.txt the same as an XML sitemap?

No. Robots.txt guides what may be crawled. A sitemap lists URLs you want discovered. Keep sitemap hygiene in the XML sitemap guide; keep crawl rules here.

Should I block CSS and JavaScript in robots.txt?

Usually no. Google needs CSS and JS to render pages. Blocking theme or plugin assets can leave crawlers with a broken view of your layout and content.

Can a Disallow rule keep a page out of Google?

Not reliably. Disallowed URLs can still appear in results without a snippet if other sites link to them. Use noindex or access control when you need the URL out of Search—see indexing troubleshooting when the symptom is “URL still shows.”

Do I need a Sitemap line in robots.txt?

Optional. Many small blogs submit the sitemap only in Search Console. A Sitemap: line can help other crawlers find the file, but it is not required for Google if you already submitted it.

Where should robots.txt live?

At the site root: https://example.com/robots.txt. Subfolder copies are ignored for that host. Confirm the live file loads for a signed-out browser.

What should small blogs usually block?

Admin, account, cart, thank-you, internal search, and staging paths when those URLs should not be crawled. Keep public posts and render-critical assets allowed.

How do I test robots.txt changes?

Fetch the file live, check syntax, then use Google Search Console URL Inspection or robots testing tools Google documents. Watch for blocked resources on key templates—pair with the technical SEO tools checklist when you need a broader crawl/render pass.

Closing

For robots.txt for blogs, run the Allow-Block Decision Card: allow public content and render assets, block admin and utility noise, and move true “keep out of Search” jobs to noindex or auth. Keep the file short, test after edits, and treat the Sitemap line as optional—not a substitute for a clean sitemap or solid indexing controls.