Sitemap and Robots.txt

Search engines discover your content through sitemaps and respect crawling rules in robots.txt. Next.js makes it easy to generate dynamic sitemaps from your database and configure robots.txt programmatically. Proper sitemap and robots configuration ensures search engines can efficiently crawl and index your site. Let's optimize for discoverability!

Why Sitemaps and Robots.txt Matter

📄 Sitemap Benefits

Helps search engines discover all your pages
Indicates page priority and update frequency
Speeds up indexing of new content
Critical for large sites with many pages
Required for Google Search Console

🤖 Robots.txt Benefits

Controls which pages crawlers can access
Prevents indexing of admin/private pages
Saves crawl budget on large sites
Points crawlers to sitemap location
Manages crawler-specific rules

How Search Engines Use These Files

Crawler visits your site
Reads /robots.txt for crawling instructions
Finds sitemap URL in robots.txt
Downloads and parses sitemap.xml
Crawls URLs listed in sitemap (respecting robots.txt rules)
Indexes allowed pages

Static Sitemap

Simple Static Sitemap

app/sitemap.ts

import { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://example.com',
      lastModified: new Date(),
      changeFrequency: 'yearly',
      priority: 1,
    },
    {
      url: 'https://example.com/about',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: 'https://example.com/blog',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.5,
    },
  ];
}

// Generates /sitemap.xml automatically
// Available at: https://example.com/sitemap.xml

// ✅ Type-safe with MetadataRoute.Sitemap
// ✅ Automatic XML generation
// ✅ Priority and change frequency hints

Sitemap Properties

Sitemap Entry Fields

url (required): Full absolute URL
lastModified: Last modification date (Date or string)
changeFrequency: How often page changes ('always', 'hourly', 'daily', 'weekly', 'monthly', 'yearly', 'never')
priority: Priority 0.0 to 1.0 (1.0 is highest)
alternates: Language/region alternates (optional)

Dynamic Sitemap from Database

Blog Sitemap Example

app/sitemap.ts

import { MetadataRoute } from 'next';

async function getAllPosts() {
  const posts = await db.posts.findMany({
    where: { published: true },
    select: {
      slug: true,
      updatedAt: true,
    },
    orderBy: { updatedAt: 'desc' },
  });
  return posts;
}

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://myblog.com';
  
  // Static pages
  const staticPages = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily' as const,
      priority: 1,
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly' as const,
      priority: 0.8,
    },
    {
      url: `${baseUrl}/blog`,
      lastModified: new Date(),
      changeFrequency: 'daily' as const,
      priority: 0.9,
    },
  ];
  
  // Dynamic blog posts
  const posts = await getAllPosts();
  const postPages = posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }));
  
  return [...staticPages, ...postPages];
}

// ✅ Fetches posts from database
// ✅ Includes only published posts
// ✅ Uses actual update dates
// ✅ Combines static and dynamic pages

E-commerce Sitemap Example

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://mystore.com';
  
  // Fetch products
  const products = await db.products.findMany({
    where: { published: true },
    select: {
      id: true,
      updatedAt: true,
      category: true,
    },
  });
  
  // Fetch categories
  const categories = await db.categories.findMany({
    select: {
      slug: true,
      updatedAt: true,
    },
  });
  
  // Static pages
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${baseUrl}/products`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
  ];
  
  // Category pages
  const categoryPages: MetadataRoute.Sitemap = categories.map(category => ({
    url: `${baseUrl}/category/${category.slug}`,
    lastModified: category.updatedAt,
    changeFrequency: 'daily',
    priority: 0.8,
  }));
  
  // Product pages
  const productPages: MetadataRoute.Sitemap = products.map(product => ({
    url: `${baseUrl}/products/${product.id}`,
    lastModified: product.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.6,
  }));
  
  return [...staticPages, ...categoryPages, ...productPages];
}

// ✅ Multiple content types
// ✅ Categories and products
// ✅ Appropriate priorities
// ✅ Dynamic from database

Large Site with Sitemap Index

app/sitemap.ts

import { MetadataRoute } from 'next';

// Main sitemap - static pages only
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      priority: 1,
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      priority: 0.8,
    },
    {
      url: `${baseUrl}/contact`,
      lastModified: new Date(),
      priority: 0.8,
    },
  ];
}

// ✅ Main sitemap for static pages
// ✅ Separate sitemaps for large sections

app/blog/sitemap.ts

import { MetadataRoute } from 'next';

// Blog-specific sitemap
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  const posts = await db.posts.findMany({
    where: { published: true },
    select: { slug: true, updatedAt: true },
  });
  
  return posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
  }));
}

// Available at: /blog/sitemap.xml
// ✅ Nested sitemap for blog section
// ✅ Keeps main sitemap small
// ✅ Better for sites with 1000+ pages

Multilingual Sitemaps

Sitemap with Alternates

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  const posts = await getAllPosts();
  
  return posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
    
    // Language/region alternates
    alternates: {
      languages: {
        es: `${baseUrl}/es/blog/${post.slug}`,
        de: `${baseUrl}/de/blog/${post.slug}`,
        fr: `${baseUrl}/fr/blog/${post.slug}`,
      },
    },
  }));
}

// Generates:
// <url>
//   <loc>https://example.com/blog/post</loc>
//   <xhtml:link rel="alternate" hreflang="es" href=".../es/blog/post"/>
//   <xhtml:link rel="alternate" hreflang="de" href=".../de/blog/post"/>
// </url>

// ✅ Multi-language support
// ✅ Search engines understand alternates
// ✅ Better international SEO

Robots.txt Configuration

Static Robots.txt

public/robots.txt

# Allow all crawlers
User-agent: *
Allow: /

# Disallow specific paths
Disallow: /admin
Disallow: /api
Disallow: /dashboard
Disallow: /_next/

# Sitemap location
Sitemap: https://example.com/sitemap.xml

# ✅ Simple static configuration
# ✅ Place in public/robots.txt
# ✅ Accessible at /robots.txt

Dynamic Robots.txt

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin', '/api', '/dashboard'],
    },
    sitemap: 'https://example.com/sitemap.xml',
  };
}

// Generates /robots.txt:
// User-agent: *
// Allow: /
// Disallow: /admin
// Disallow: /api
// Disallow: /dashboard
// Sitemap: https://example.com/sitemap.xml

// ✅ Type-safe configuration
// ✅ Dynamic generation
// ✅ Environment-based rules possible

Multiple User-Agent Rules

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Default for all crawlers
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin', '/api'],
        crawlDelay: 10,
      },
      // Specific rules for Googlebot
      {
        userAgent: 'Googlebot',
        allow: '/',
        disallow: ['/admin'],
        // No crawl delay for Google
      },
      // Block bad bots
      {
        userAgent: 'BadBot',
        disallow: '/',
      },
    ],
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/blog/sitemap.xml',
    ],
  };
}

// ✅ Multiple user-agent rules
// ✅ Crawler-specific settings
// ✅ Multiple sitemaps
// ✅ Crawl delay for non-Google bots

Environment-Based Robots

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_BASE_URL || 'https://example.com';
  const isProduction = process.env.NODE_ENV === 'production';
  
  if (!isProduction) {
    // Block all crawlers in non-production
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }
  
  // Production rules
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin', '/api', '/dashboard'],
    },
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

// ✅ Environment-aware
// ✅ Block staging/dev from indexing
// ✅ Production-only sitemaps
// ✅ Dynamic base URL

Complete Sitemap Examples

Example 1: Multi-Section Site

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  // Fetch all content
  const [posts, products, docs] = await Promise.all([
    db.posts.findMany({ where: { published: true } }),
    db.products.findMany({ where: { published: true } }),
    db.docs.findMany({ where: { published: true } }),
  ]);
  
  // Homepage and main sections
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${baseUrl}/blog`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
    {
      url: `${baseUrl}/products`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
    {
      url: `${baseUrl}/docs`,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.8,
    },
  ];
  
  // Blog posts
  const blogPages: MetadataRoute.Sitemap = posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
  }));
  
  // Products
  const productPages: MetadataRoute.Sitemap = products.map(product => ({
    url: `${baseUrl}/products/${product.id}`,
    lastModified: product.updatedAt,
    changeFrequency: 'daily',
    priority: 0.8,
  }));
  
  // Documentation
  const docPages: MetadataRoute.Sitemap = docs.map(doc => ({
    url: `${baseUrl}/docs/${doc.slug}`,
    lastModified: doc.updatedAt,
    changeFrequency: 'monthly',
    priority: 0.6,
  }));
  
  return [...staticPages, ...blogPages, ...productPages, ...docPages];
}

// ✅ Parallel fetching for speed
// ✅ Multiple content types
// ✅ Appropriate priorities
// ✅ Change frequencies match update patterns

Example 2: Site with User-Generated Content

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  // Only include public user profiles (opted-in)
  const publicProfiles = await db.users.findMany({
    where: { 
      profilePublic: true,
      verified: true,
    },
    select: {
      username: true,
      profileUpdatedAt: true,
    },
  });
  
  // User-generated posts (public only)
  const publicPosts = await db.posts.findMany({
    where: { 
      visibility: 'public',
      flagged: false,
    },
    select: {
      id: true,
      updatedAt: true,
    },
  });
  
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      priority: 1,
    },
  ];
  
  const profilePages: MetadataRoute.Sitemap = publicProfiles.map(user => ({
    url: `${baseUrl}/u/${user.username}`,
    lastModified: user.profileUpdatedAt,
    changeFrequency: 'weekly',
    priority: 0.6,
  }));
  
  const postPages: MetadataRoute.Sitemap = publicPosts.map(post => ({
    url: `${baseUrl}/posts/${post.id}`,
    lastModified: post.updatedAt,
    changeFrequency: 'daily',
    priority: 0.5,
  }));
  
  return [...staticPages, ...profilePages, ...postPages];
}

// ✅ Only public content
// ✅ Privacy-respecting
// ✅ Filters flagged content
// ✅ Verified users only

Sitemap and Robots Structure

Organization of sitemap and robots files

appImportant

public

Select a file or folder to see details

Sitemap and Robots Best Practices

1. Keep Sitemaps Under 50,000 URLs

TYPESCRIPT

// ✅ GOOD: Split large sites into multiple sitemaps
// app/sitemap.ts - Main sitemap (static pages)
// app/blog/sitemap.ts - Blog posts
// app/products/sitemap.ts - Products

// Google limit: 50,000 URLs per sitemap
// If you have more, use sitemap index

// ❌ BAD: Single sitemap with 100,000 URLs
// Will be rejected by search engines

2. Only Include Indexable Pages

TYPESCRIPT

// ✅ GOOD: Only published, public content
const posts = await db.posts.findMany({
  where: { 
    published: true,
    noIndex: false,  // Respect noindex flag
  },
});

// ❌ BAD: Include everything
const posts = await db.posts.findMany();  // Draft posts, private content

// Don't include:
// - Admin pages
// - Thank you pages
// - Paginated pages (only page 1)
// - Pages with noindex
// - 404/error pages

3. Use Accurate Last Modified Dates

TYPESCRIPT

// ✅ GOOD: Actual update dates from database
return posts.map(post => ({
  url: `${baseUrl}/blog/${post.slug}`,
  lastModified: post.updatedAt,  // Real date
}));

// ❌ BAD: Current date for everything
return posts.map(post => ({
  url: `${baseUrl}/blog/${post.slug}`,
  lastModified: new Date(),  // Inaccurate
}));

// Accurate dates help crawlers prioritize

4. Set Realistic Change Frequencies

TYPESCRIPT

// ✅ GOOD: Match actual update patterns
{
  url: '/',
  changeFrequency: 'daily',  // Homepage updated daily
}
{
  url: '/blog/post',
  changeFrequency: 'weekly',  // Posts edited occasionally
}
{
  url: '/about',
  changeFrequency: 'yearly',  // Rarely changes
}

// ❌ BAD: Everything "always"
changeFrequency: 'always'  // Reserved for real-time data

// Be honest with change frequency

5. Use Priority Thoughtfully

TYPESCRIPT

// ✅ GOOD: Relative importance
{
  url: '/',
  priority: 1.0,  // Homepage most important
}
{
  url: '/blog',
  priority: 0.9,  // Main sections
}
{
  url: '/blog/post',
  priority: 0.7,  // Individual posts
}
{
  url: '/blog/tag/react',
  priority: 0.5,  // Tag pages
}

// ❌ BAD: Everything priority 1.0
// Priority is relative to YOUR site, not the web

6. Block Sensitive Pages in Robots.txt

TYPESCRIPT

// ✅ GOOD: Block admin and sensitive areas
{
  userAgent: '*',
  allow: '/',
  disallow: [
    '/admin',
    '/dashboard',
    '/api',
    '/checkout',  // Sensitive user flow
    '/auth',      // Login pages
    '/_next',     // Next.js internals
  ],
}

// Also use meta robots for page-level control:
export const metadata = {
  robots: {
    index: false,
    follow: false,
  },
};

7. Always Include Sitemap in Robots.txt

TYPESCRIPT

// ✅ GOOD: Point to sitemap
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
    },
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/blog/sitemap.xml',
    ],
  };
}

// Helps crawlers find your content faster

Testing and Verification

Testing Your Sitemap

Verification Steps

Visit your sitemap URL: https://yoursite.com/sitemap.xml
Verify XML is well-formed (no errors in browser)
Check all URLs are absolute (https://...)
Verify URLs are canonical versions
Submit to Google Search Console: Search Console → Sitemaps → Add sitemap
Monitor for errors in Search Console

Testing Robots.txt

Verification Steps

Visit: https://yoursite.com/robots.txt
Verify syntax is correct
Check sitemap URL is present
Test in Google Search Console: Settings → robots.txt tester
Verify blocked URLs are actually blocked
Ensure allowed URLs are allowed

Common Issues

TYPESCRIPT

// ❌ PROBLEM: Relative URLs in sitemap
{
  url: '/blog/post',  // Wrong!
}

// ✅ SOLUTION: Use absolute URLs
{
  url: 'https://example.com/blog/post',
}

// ❌ PROBLEM: Mixed http/https URLs
{
  url: 'http://example.com/page',  // Inconsistent
}
{
  url: 'https://example.com/other',
}

// ✅ SOLUTION: All URLs use same protocol (https)
const baseUrl = 'https://example.com';

// ❌ PROBLEM: Sitemap too large
// 60,000 URLs in one sitemap

// ✅ SOLUTION: Split into multiple sitemaps
// Main sitemap + section sitemaps

Key Takeaways

app/sitemap.ts - dynamic sitemap generation
app/robots.ts - dynamic robots.txt
50,000 URL limit - split large sites
Absolute URLs - always use full https:// URLs
Accurate dates - use real lastModified dates
Realistic priorities - relative to your site
Block sensitive areas - admin, API, checkout
Submit to Search Console - monitor and verify

🎉 Metadata and SEO Section Complete!

You've completed the Metadata and SEO section! You've mastered:

✅ Static Metadata Configuration
✅ Dynamic Metadata Generation
✅ Open Graph and Social Media Cards
✅ Sitemap and Robots.txt

You now have complete mastery of SEO in Next.js! You can configure static and dynamic metadata, generate stunning social media cards, create dynamic sitemaps from your database, and control search engine crawling with robots.txt. These skills ensure your Next.js apps are discoverable, shareable, and optimized for search engines and social platforms.

🚀 SEO is a Journey

Great SEO requires ongoing effort. Monitor Search Console, analyze performance, update metadata as content changes, and continuously improve. Good metadata and sitemaps are the foundation—quality content and user experience are what keep visitors coming back!

Final Quiz: Sitemap and Robots Mastery

Question 1 of 4

What is a sitemap.xml file?

Why Sitemaps and Robots.txt Matter

📄 Sitemap Benefits

Helps search engines discover all your pages
Indicates page priority and update frequency
Speeds up indexing of new content
Critical for large sites with many pages
Required for Google Search Console

🤖 Robots.txt Benefits

Controls which pages crawlers can access
Prevents indexing of admin/private pages
Saves crawl budget on large sites
Points crawlers to sitemap location
Manages crawler-specific rules

How Search Engines Use These Files

Crawler visits your site
Reads /robots.txt for crawling instructions
Finds sitemap URL in robots.txt
Downloads and parses sitemap.xml
Crawls URLs listed in sitemap (respecting robots.txt rules)
Indexes allowed pages

Static Sitemap

Simple Static Sitemap

app/sitemap.ts

import { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://example.com',
      lastModified: new Date(),
      changeFrequency: 'yearly',
      priority: 1,
    },
    {
      url: 'https://example.com/about',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: 'https://example.com/blog',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.5,
    },
  ];
}

// Generates /sitemap.xml automatically
// Available at: https://example.com/sitemap.xml

// ✅ Type-safe with MetadataRoute.Sitemap
// ✅ Automatic XML generation
// ✅ Priority and change frequency hints

Sitemap Properties

Sitemap Entry Fields

url (required): Full absolute URL
lastModified: Last modification date (Date or string)
changeFrequency: How often page changes ('always', 'hourly', 'daily', 'weekly', 'monthly', 'yearly', 'never')
priority: Priority 0.0 to 1.0 (1.0 is highest)
alternates: Language/region alternates (optional)

Dynamic Sitemap from Database

Blog Sitemap Example

app/sitemap.ts

import { MetadataRoute } from 'next';

async function getAllPosts() {
  const posts = await db.posts.findMany({
    where: { published: true },
    select: {
      slug: true,
      updatedAt: true,
    },
    orderBy: { updatedAt: 'desc' },
  });
  return posts;
}

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://myblog.com';
  
  // Static pages
  const staticPages = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily' as const,
      priority: 1,
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly' as const,
      priority: 0.8,
    },
    {
      url: `${baseUrl}/blog`,
      lastModified: new Date(),
      changeFrequency: 'daily' as const,
      priority: 0.9,
    },
  ];
  
  // Dynamic blog posts
  const posts = await getAllPosts();
  const postPages = posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }));
  
  return [...staticPages, ...postPages];
}

// ✅ Fetches posts from database
// ✅ Includes only published posts
// ✅ Uses actual update dates
// ✅ Combines static and dynamic pages

E-commerce Sitemap Example

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://mystore.com';
  
  // Fetch products
  const products = await db.products.findMany({
    where: { published: true },
    select: {
      id: true,
      updatedAt: true,
      category: true,
    },
  });
  
  // Fetch categories
  const categories = await db.categories.findMany({
    select: {
      slug: true,
      updatedAt: true,
    },
  });
  
  // Static pages
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${baseUrl}/products`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
  ];
  
  // Category pages
  const categoryPages: MetadataRoute.Sitemap = categories.map(category => ({
    url: `${baseUrl}/category/${category.slug}`,
    lastModified: category.updatedAt,
    changeFrequency: 'daily',
    priority: 0.8,
  }));
  
  // Product pages
  const productPages: MetadataRoute.Sitemap = products.map(product => ({
    url: `${baseUrl}/products/${product.id}`,
    lastModified: product.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.6,
  }));
  
  return [...staticPages, ...categoryPages, ...productPages];
}

// ✅ Multiple content types
// ✅ Categories and products
// ✅ Appropriate priorities
// ✅ Dynamic from database

Large Site with Sitemap Index

app/sitemap.ts

import { MetadataRoute } from 'next';

// Main sitemap - static pages only
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      priority: 1,
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      priority: 0.8,
    },
    {
      url: `${baseUrl}/contact`,
      lastModified: new Date(),
      priority: 0.8,
    },
  ];
}

// ✅ Main sitemap for static pages
// ✅ Separate sitemaps for large sections

app/blog/sitemap.ts

import { MetadataRoute } from 'next';

// Blog-specific sitemap
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  const posts = await db.posts.findMany({
    where: { published: true },
    select: { slug: true, updatedAt: true },
  });
  
  return posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
  }));
}

// Available at: /blog/sitemap.xml
// ✅ Nested sitemap for blog section
// ✅ Keeps main sitemap small
// ✅ Better for sites with 1000+ pages

Multilingual Sitemaps

Sitemap with Alternates

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  const posts = await getAllPosts();
  
  return posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
    
    // Language/region alternates
    alternates: {
      languages: {
        es: `${baseUrl}/es/blog/${post.slug}`,
        de: `${baseUrl}/de/blog/${post.slug}`,
        fr: `${baseUrl}/fr/blog/${post.slug}`,
      },
    },
  }));
}

// Generates:
// <url>
//   <loc>https://example.com/blog/post</loc>
//   <xhtml:link rel="alternate" hreflang="es" href=".../es/blog/post"/>
//   <xhtml:link rel="alternate" hreflang="de" href=".../de/blog/post"/>
// </url>

// ✅ Multi-language support
// ✅ Search engines understand alternates
// ✅ Better international SEO

Robots.txt Configuration

Static Robots.txt

public/robots.txt

# Allow all crawlers
User-agent: *
Allow: /

# Disallow specific paths
Disallow: /admin
Disallow: /api
Disallow: /dashboard
Disallow: /_next/

# Sitemap location
Sitemap: https://example.com/sitemap.xml

# ✅ Simple static configuration
# ✅ Place in public/robots.txt
# ✅ Accessible at /robots.txt

Dynamic Robots.txt

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin', '/api', '/dashboard'],
    },
    sitemap: 'https://example.com/sitemap.xml',
  };
}

// Generates /robots.txt:
// User-agent: *
// Allow: /
// Disallow: /admin
// Disallow: /api
// Disallow: /dashboard
// Sitemap: https://example.com/sitemap.xml

// ✅ Type-safe configuration
// ✅ Dynamic generation
// ✅ Environment-based rules possible

Multiple User-Agent Rules

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Default for all crawlers
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin', '/api'],
        crawlDelay: 10,
      },
      // Specific rules for Googlebot
      {
        userAgent: 'Googlebot',
        allow: '/',
        disallow: ['/admin'],
        // No crawl delay for Google
      },
      // Block bad bots
      {
        userAgent: 'BadBot',
        disallow: '/',
      },
    ],
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/blog/sitemap.xml',
    ],
  };
}

// ✅ Multiple user-agent rules
// ✅ Crawler-specific settings
// ✅ Multiple sitemaps
// ✅ Crawl delay for non-Google bots

Environment-Based Robots

app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_BASE_URL || 'https://example.com';
  const isProduction = process.env.NODE_ENV === 'production';
  
  if (!isProduction) {
    // Block all crawlers in non-production
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }
  
  // Production rules
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin', '/api', '/dashboard'],
    },
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

// ✅ Environment-aware
// ✅ Block staging/dev from indexing
// ✅ Production-only sitemaps
// ✅ Dynamic base URL

Complete Sitemap Examples

Example 1: Multi-Section Site

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  // Fetch all content
  const [posts, products, docs] = await Promise.all([
    db.posts.findMany({ where: { published: true } }),
    db.products.findMany({ where: { published: true } }),
    db.docs.findMany({ where: { published: true } }),
  ]);
  
  // Homepage and main sections
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${baseUrl}/blog`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
    {
      url: `${baseUrl}/products`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
    {
      url: `${baseUrl}/docs`,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.8,
    },
  ];
  
  // Blog posts
  const blogPages: MetadataRoute.Sitemap = posts.map(post => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly',
    priority: 0.7,
  }));
  
  // Products
  const productPages: MetadataRoute.Sitemap = products.map(product => ({
    url: `${baseUrl}/products/${product.id}`,
    lastModified: product.updatedAt,
    changeFrequency: 'daily',
    priority: 0.8,
  }));
  
  // Documentation
  const docPages: MetadataRoute.Sitemap = docs.map(doc => ({
    url: `${baseUrl}/docs/${doc.slug}`,
    lastModified: doc.updatedAt,
    changeFrequency: 'monthly',
    priority: 0.6,
  }));
  
  return [...staticPages, ...blogPages, ...productPages, ...docPages];
}

// ✅ Parallel fetching for speed
// ✅ Multiple content types
// ✅ Appropriate priorities
// ✅ Change frequencies match update patterns

Example 2: Site with User-Generated Content

app/sitemap.ts

import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com';
  
  // Only include public user profiles (opted-in)
  const publicProfiles = await db.users.findMany({
    where: { 
      profilePublic: true,
      verified: true,
    },
    select: {
      username: true,
      profileUpdatedAt: true,
    },
  });
  
  // User-generated posts (public only)
  const publicPosts = await db.posts.findMany({
    where: { 
      visibility: 'public',
      flagged: false,
    },
    select: {
      id: true,
      updatedAt: true,
    },
  });
  
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: baseUrl,
      lastModified: new Date(),
      priority: 1,
    },
  ];
  
  const profilePages: MetadataRoute.Sitemap = publicProfiles.map(user => ({
    url: `${baseUrl}/u/${user.username}`,
    lastModified: user.profileUpdatedAt,
    changeFrequency: 'weekly',
    priority: 0.6,
  }));
  
  const postPages: MetadataRoute.Sitemap = publicPosts.map(post => ({
    url: `${baseUrl}/posts/${post.id}`,
    lastModified: post.updatedAt,
    changeFrequency: 'daily',
    priority: 0.5,
  }));
  
  return [...staticPages, ...profilePages, ...postPages];
}

// ✅ Only public content
// ✅ Privacy-respecting
// ✅ Filters flagged content
// ✅ Verified users only

Sitemap and Robots Structure

Organization of sitemap and robots files

appImportant

public

Select a file or folder to see details

Sitemap and Robots Best Practices

1. Keep Sitemaps Under 50,000 URLs

TYPESCRIPT

// ✅ GOOD: Split large sites into multiple sitemaps
// app/sitemap.ts - Main sitemap (static pages)
// app/blog/sitemap.ts - Blog posts
// app/products/sitemap.ts - Products

// Google limit: 50,000 URLs per sitemap
// If you have more, use sitemap index

// ❌ BAD: Single sitemap with 100,000 URLs
// Will be rejected by search engines

2. Only Include Indexable Pages

TYPESCRIPT

// ✅ GOOD: Only published, public content
const posts = await db.posts.findMany({
  where: { 
    published: true,
    noIndex: false,  // Respect noindex flag
  },
});

// ❌ BAD: Include everything
const posts = await db.posts.findMany();  // Draft posts, private content

// Don't include:
// - Admin pages
// - Thank you pages
// - Paginated pages (only page 1)
// - Pages with noindex
// - 404/error pages

3. Use Accurate Last Modified Dates

TYPESCRIPT

// ✅ GOOD: Actual update dates from database
return posts.map(post => ({
  url: `${baseUrl}/blog/${post.slug}`,
  lastModified: post.updatedAt,  // Real date
}));

// ❌ BAD: Current date for everything
return posts.map(post => ({
  url: `${baseUrl}/blog/${post.slug}`,
  lastModified: new Date(),  // Inaccurate
}));

// Accurate dates help crawlers prioritize

4. Set Realistic Change Frequencies

TYPESCRIPT

// ✅ GOOD: Match actual update patterns
{
  url: '/',
  changeFrequency: 'daily',  // Homepage updated daily
}
{
  url: '/blog/post',
  changeFrequency: 'weekly',  // Posts edited occasionally
}
{
  url: '/about',
  changeFrequency: 'yearly',  // Rarely changes
}

// ❌ BAD: Everything "always"
changeFrequency: 'always'  // Reserved for real-time data

// Be honest with change frequency

5. Use Priority Thoughtfully

TYPESCRIPT

// ✅ GOOD: Relative importance
{
  url: '/',
  priority: 1.0,  // Homepage most important
}
{
  url: '/blog',
  priority: 0.9,  // Main sections
}
{
  url: '/blog/post',
  priority: 0.7,  // Individual posts
}
{
  url: '/blog/tag/react',
  priority: 0.5,  // Tag pages
}

// ❌ BAD: Everything priority 1.0
// Priority is relative to YOUR site, not the web

6. Block Sensitive Pages in Robots.txt

TYPESCRIPT

// ✅ GOOD: Block admin and sensitive areas
{
  userAgent: '*',
  allow: '/',
  disallow: [
    '/admin',
    '/dashboard',
    '/api',
    '/checkout',  // Sensitive user flow
    '/auth',      // Login pages
    '/_next',     // Next.js internals
  ],
}

// Also use meta robots for page-level control:
export const metadata = {
  robots: {
    index: false,
    follow: false,
  },
};

7. Always Include Sitemap in Robots.txt

TYPESCRIPT

// ✅ GOOD: Point to sitemap
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
    },
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/blog/sitemap.xml',
    ],
  };
}

// Helps crawlers find your content faster

Testing and Verification

Testing Your Sitemap

Verification Steps

Visit your sitemap URL: https://yoursite.com/sitemap.xml
Verify XML is well-formed (no errors in browser)
Check all URLs are absolute (https://...)
Verify URLs are canonical versions
Submit to Google Search Console: Search Console → Sitemaps → Add sitemap
Monitor for errors in Search Console

Testing Robots.txt

Verification Steps

Visit: https://yoursite.com/robots.txt
Verify syntax is correct
Check sitemap URL is present
Test in Google Search Console: Settings → robots.txt tester
Verify blocked URLs are actually blocked
Ensure allowed URLs are allowed

Common Issues

TYPESCRIPT

// ❌ PROBLEM: Relative URLs in sitemap
{
  url: '/blog/post',  // Wrong!
}

// ✅ SOLUTION: Use absolute URLs
{
  url: 'https://example.com/blog/post',
}

// ❌ PROBLEM: Mixed http/https URLs
{
  url: 'http://example.com/page',  // Inconsistent
}
{
  url: 'https://example.com/other',
}

// ✅ SOLUTION: All URLs use same protocol (https)
const baseUrl = 'https://example.com';

// ❌ PROBLEM: Sitemap too large
// 60,000 URLs in one sitemap

// ✅ SOLUTION: Split into multiple sitemaps
// Main sitemap + section sitemaps

Key Takeaways

app/sitemap.ts - dynamic sitemap generation
app/robots.ts - dynamic robots.txt
50,000 URL limit - split large sites
Absolute URLs - always use full https:// URLs
Accurate dates - use real lastModified dates
Realistic priorities - relative to your site
Block sensitive areas - admin, API, checkout
Submit to Search Console - monitor and verify

🎉 Metadata and SEO Section Complete!

You've completed the Metadata and SEO section! You've mastered:

✅ Static Metadata Configuration
✅ Dynamic Metadata Generation
✅ Open Graph and Social Media Cards
✅ Sitemap and Robots.txt

🚀 SEO is a Journey

Final Quiz: Sitemap and Robots Mastery

Question 1 of 4

Why Sitemaps and Robots.txt Matter

📄 Sitemap Benefits

🤖 Robots.txt Benefits

How Search Engines Use These Files

Static Sitemap

Simple Static Sitemap

Sitemap Properties

Sitemap Entry Fields

Dynamic Sitemap from Database

Blog Sitemap Example

E-commerce Sitemap Example

Large Site with Sitemap Index

Multilingual Sitemaps

Sitemap with Alternates

Robots.txt Configuration

Static Robots.txt

Dynamic Robots.txt

Multiple User-Agent Rules

Environment-Based Robots

Complete Sitemap Examples

Example 1: Multi-Section Site

Example 2: Site with User-Generated Content

Sitemap and Robots Structure

Sitemap and Robots Best Practices

1. Keep Sitemaps Under 50,000 URLs

2. Only Include Indexable Pages

3. Use Accurate Last Modified Dates

4. Set Realistic Change Frequencies

5. Use Priority Thoughtfully

6. Block Sensitive Pages in Robots.txt

7. Always Include Sitemap in Robots.txt

Testing and Verification

Testing Your Sitemap

Verification Steps

Testing Robots.txt

Verification Steps

Common Issues

Key Takeaways

🎉 Metadata and SEO Section Complete!

Final Quiz: Sitemap and Robots Mastery

What is a sitemap.xml file?

Master sitemap.xml and robots.txt in Next.js! Learn dynamic sitemap generation and search engine optimization.

Continue Your Next.js Journey

You might also like

UI Components

Developer Tools

HTML Game

Join Our Web Development Training Program

Join Our Social Communities

NextJS Tutorials

Sitemap and Robots.txt

Why Sitemaps and Robots.txt Matter

📄 Sitemap Benefits

🤖 Robots.txt Benefits

How Search Engines Use These Files

Static Sitemap

Simple Static Sitemap

Sitemap Properties

Sitemap Entry Fields

Dynamic Sitemap from Database

Blog Sitemap Example

E-commerce Sitemap Example

Large Site with Sitemap Index

Multilingual Sitemaps

Sitemap with Alternates

Robots.txt Configuration

Static Robots.txt

Dynamic Robots.txt

Multiple User-Agent Rules

Environment-Based Robots

Complete Sitemap Examples

Example 1: Multi-Section Site

Example 2: Site with User-Generated Content

Sitemap and Robots Structure

Sitemap and Robots Best Practices

1. Keep Sitemaps Under 50,000 URLs

2. Only Include Indexable Pages

3. Use Accurate Last Modified Dates

4. Set Realistic Change Frequencies