Code Beyond Limits
Back to blog
9 min read

Automating Sitemap Generation with Node.js: The Step-by-Step Guide

Automating Sitemap Generation with Node.js: The Step-by-Step Guide

A sitemap is a file that contains information about the pages, videos, and other files on your website, as well as their relationships. Search engines such as Google use this file to crawl your site more effectively.

By providing a sitemap, you can indicate to search engines which pages and files are important on your site, and also share useful details about them, such as when they were last updated and whether there are alternative language versions available.

In this guide, I’ll walk you through the process of automating the creation of a sitemap using Node.js. The guide will cover the creation of both sitemap index and sitemaps, along with handling multi-language content. Furthermore, I will show you how to automate the entire process, resulting in a fully-automated Node.js server for sitemap generation.

Sitemap Index

It is important to note that each sitemap has a size limit of 50MB (uncompressed) or 50,000 URLs. In case your sitemap exceeds these limits, it will need to be divided into multiple sitemaps. These individual sitemaps can then be linked together using a sitemap index file.

The sitemap index file serves as a centralized location for search engines to access all of your sitemaps. Just as with the standard sitemaps, all sitemap requirements also apply to the sitemap index file.

Sitemap has a size limit of 50MB (uncompressed) or 50,000 URLs.

By properly segmenting large sitemaps into smaller ones and utilizing a sitemap index file, you can ensure that search engines can efficiently crawl and index all of the content on your website.

Setting Up the Server

To begin, let’s create an empty folder for our application, navigate inside it, and run the npm init command. This will prompt you with some basic questions, and after filling them out, the package.json file will be created.

Now, let’s create the app.js file, which will be the main file of our application and where we will build the server.

const express = require('express');
const http = require('http');

const app = express();

// Catch 404 and forward to error handler
app.use((req, res, next) => {
  console.log(`Requested endpoint "${req.originalUrl}" not found.`);
  const error = new Error('Endpoint Not Found');
  error.status = 404;
  next(error);
});

// Error handler
app.use((error, req, res, next) => {
  console.log('Error: ', error);
  res.status(error.status || 500).json({ success: false });
});

const server = http.createServer(app);

server.listen(3000, () => {
  console.log(`Server is listening on port 3000.`);
});

With the code above, we have completed our server setup. We used Express framework to create a server, we added a middleware that will catch all 404 errors and pass them to the global error handler, and we configured a global error handler. After that, we created and built the server, which is now listening on port 3000. To start the server, you can run the npm start command in the terminal.

To install the express package in your app, you can run npm i express in the terminal.

Creating Sitemaps

Consider this scenario: you have an e-commerce website with a large number of product pages, as well as common pages such as the home page, login, register, and about us.

To help search engines easily discover and index all of these pages, you need to create a sitemap that lists the URLs of all the pages you want to be indexed. Additionally, if your website supports multiple languages, you will need to include the language-specific version of each page in the sitemap. Also, you need to ensure that the size of your sitemap does not exceed 50MB or contain more than 50,000 URLs.

Since an e-commerce website can easily exceed 50,000 URLs, it is necessary to implement a logic that will divide all URLs into multiple sitemaps. Each sitemap should have a maximum of 50,000 URLs, and all individual sitemaps should be linked together using a sitemap index file.

To implement this logic, create a new folder called controllers in the root directory, and then create a new file called seo-controller.js inside the folder with the following content:

const { simpleSitemapAndIndex } = require('sitemap');

const products = [
  { title: 'Product 1', display_url: 'product_1', updated_at: '2023-03-01T10:00:00.000Z' },
  { title: 'Product 2', display_url: 'product_2', updated_at: '2023-03-02T10:00:00.000Z' },
  { title: 'Product 3', display_url: 'product_3', updated_at: '2023-03-03T10:00:00.000Z' },
  { title: 'Product 4', display_url: 'product_4', updated_at: '2023-03-04T10:00:00.000Z' },
  { title: 'Product 5', display_url: 'product_5', updated_at: '2023-03-05T10:00:00.000Z' },
];

exports.generateSitemaps = async () => {
  try {
    const sitemap_items = [
      { url: '', priority: 1 },
      { url: 'login' },
      { url: 'register' },
      { url: 'about-us' },
      { url: 'terms-and-conditions' },
    ];

    // Adding all product pages to the sitemap
    sitemap_items.push(
      ...products.map((Product) => ({ url: `products/${Product.display_url}`, lastmod: Product.updated_at }))
    );

    // Adding alternative language versions (Chinese and Spanish)
    const language_specific_items = [];
    language_specific_items.push(
      ...sitemap_items.map((item) => ({ ...item, url: item.url === '' ? `zh` : `zh/${item.url}` }))
    );
    language_specific_items.push(
      ...sitemap_items.map((item) => ({ ...item, url: item.url === '' ? `es` : `es/${item.url}` }))
    );
    sitemap_items.push(...language_specific_items);

    // Generating sitemaps and sitemap index
    await simpleSitemapAndIndex({
      hostname: 'https://example.com',
      destinationDir: './sitemaps/',
      sitemapHostname: `https://example.com/api/v1/sitemaps/`,
      sourceData: sitemap_items,
    });
  } catch (error) {
    throw new Error(error);
  }
};

In the above code, we simulated a database by creating a products array. Each product has its title, display_url and updated_at properties. Later, we created and exported a function called generateSimtemaps, which duty is the create sitemaps and sitemap index. For sitemap generation, we used a popular third-party package called sitemap.

To install the sitemap package in your app, you can run npm i sitemap in the terminal.

We started by creating sitemap_items array with the list of common pages (home, login, register, about us, and terms and conditions). Note that we added priority property for the home page, which indicates the importance of a home page relative to other URLs on your site.

The priority property in the sitemap indicates the importance of a particular URL relative to other URLs on your site.

Later, we added all product pages to sitemap_items array by iterating over the list of products. Note that we added lastmod property for each product page, which indicates the last time the page was modified.

The lastmod property in the sitemap indicates the last time the page was modified.

After that, we added alternative language-specific versions of each page to the sitemap_items array for Chinese and Spanish languages.

In the end, we used simpleSitemapAndIndex function that we imported from sitemap package to generate our sitemaps and sitemap index. We 4 parameters to this function:

  • hostname - The host for which the sitemap will be created.
  • destinationDir - The folder where the generated sitemaps will be added.
  • sitemapHostname - The URL where the sitemaps will be accessible.
  • sourceData - The list of all the URLs that the sitemap should have.

In case of more than 50,000 URLs, simpleSitemapAndIndex function will automatically divide all the URLs into multiple sitemaps, it will create a sitemap index file, and will link all individual sitemaps together using a sitemap index file.

Automating the Process

To automate the creation of sitemaps, we’ll use a popular third-party package called cron. It enables us to schedule the creation of sitemaps at specific times and repeat the task on a regular basis.

To install the cron package in your app, you can run npm i cron in the terminal.

Let’s create a new file called crons-controller.js in the controllers folder, with the following content:

const { CronJob } = require('cron');
const { generateSitemaps } = require('./seo-controller');

new CronJob(
  '*/5 * * * *',
  async () => {
    try {
      await generateSitemaps();
      console.log(`Sitemaps generated successfully.`);
    } catch (error) {
      console.log(`Error when generating sitemaps: `, error);
    }
  },
  null,
  true,
  null,
  null,
  true
);

The above code creates a cron job that automatically runs the generateSitemaps function every five minutes. This ensures that the sitemaps are always up-to-date with the latest content. If new products are added to our e-commerce website, they will be included in the next cycle.

To complete the automation of sitemaps creation, all that’s left to do is to import the crons-controller.js in the main app.js file, and we can do it just after the initialization of the app variable:

const express = require('express');
const http = require('http');

const app = express();

// Start all cron jobs
require('./controllers/crons-controller');

With this, we have successfully automated the creation of sitemaps, ensuring that they are consistently updated every 5 minutes with the latest content.

Serving Sitemaps

All that is left for us to do is to add the logic that will serve the generated sitemaps when someone requests them. We can do this by creating an endpoint with the same URL as specified in the sitemapHostname config within the simpleSitemapAndIndex function.

We can create this endpoint in the main app.js file, right after the import of the crons.controller.js:

const express = require('express');
const http = require('http');

const app = express();

// Start all cron jobs
require('./controllers/crons-controller');

app.get('/api/v1/sitemaps/:sitemap', (req, res) => {
  const { sitemap } = req.params;
  return res.sendFile(path.join(__dirname, 'sitemaps', sitemap));
});

As all generated sitemaps are created within the sitemaps folder, we have created a dynamic endpoint that accepts the sitemap file name as the last part of the endpoint URL. Upon receiving a request for a specific sitemap, we extract the sitemap name from the URL params and serve that particular sitemap from the sitemaps folder.

Conclusion

By creating and providing a sitemap, you can help search engines like Google to crawl and index your website more efficiently, which greatly benefits your website’s SEO.

With the step-by-step guide provided in this article, you can easily set up a Node.js server for sitemap generation, as well as handle multi-language content and create sitemaps for large websites with ease. By implementing these best practices and ensuring that your sitemap is always up to date, you can improve your website’s search engine visibility and attract more visitors to your website.

Resources

If you want to use this setup for your own projects, you can find the source code on my GitHub repository.