Pagination is a feature that is present in most applications, as it allows for the efficient retrieval of large sets of data. It is particularly useful when dealing with lists of products, users, articles, etc. In order to achieve this, it is necessary to fetch both the total number of items and the requested batch of data at the same time.
When it comes to MongoDB, there is a correct way to implement pagination that avoids common mistakes and improves performance. Unfortunately, many articles, tutorials, and courses do not implement pagination correctly, leading to issues such as data inconsistency and decreased performance.
Incorrect Approach - Common Mistakes
A common mistake in implementing pagination is fetching the total number of items from the database first and then fetching the paginated batch. This approach has two critical issues:
-
Two database calls are made, resulting in reduced server performance and an increase in the number of unnecessary calls to the database.
-
Data inconsistency can occur, as items can be deleted or added between the two database calls. This can result in the user seeing a discrepancy in the total number of items and the number of items they are able to view.
An example of the data inconsistency that can occur as a result of this approach is when a user requests to view items on page 3, with a page size of 10. The application first retrieves the total number of items from the database, let’s say it is 25. However, between the time the application retrieves the total number of items and the time it retrieves the items for page 3, an item has been deleted. This results in the user seeing a total number of items as 25, but only being able to view 4 items on page 3, instead of the expected 5. This discrepancy can cause confusion for the user, as they are not able to view the number of items they expect to see.
An example of code that demonstrates the common mistake in implementing pagination is as follows:
const getArticles = async (req, res) => {
let { page, pageSize } = req.query;
try {
// If "page" and "pageSize" are not sent, we will default them to 1 and 50.
page = parseInt(page, 10) || 1;
pageSize = parseInt(pageSize, 10) || 50;
const totalArticles = await Articles.count({});
const articles = await Articles.find({})
.limit(pageSize)
.skip((page - 1) * pageSize);
return res.status(200).json({
success: true,
articles: {
metadata: { totalCount: totalArticles, page, pageSize },
data: articles,
},
});
} catch (error) {
return res.status(500).json({ success: false });
}
};As demonstrated in the example, the total number of items and the paginated batch are fetched separately, utilizing two separate database calls with the count() and find() queries.
The Right Way to Do it - MongoDB Aggregation Framework
The only right way to implement pagination in MongoDB is through the use of the MongoDB Aggregation Framework. The Aggregation Framework is a powerful feature of the MongoDB, though it is more complex than the basic find() query and may require more time to master. It enables the processing of data through multiple stages, with the output of each stage serving as the input for the next.
In the context of pagination, the $facet stage is of particular importance. This stage allows for the simultaneous processing of the same set of data, resulting in the retrieval of both the total number of documents and the paginated batch in a single, efficient database query.
$facet processes multiple aggregation pipelines within a single stage on the same set of input documents. Each sub-pipeline has its own field in the output document where its results are stored as an array of documents.
Now, we can refactor our code to use single aggregate() query instead of two separate queries.
exports.getArticles = async (req, res) => {
let { page, pageSize } = req.query;
try {
// If "page" and "pageSize" are not sent we will default them to 1 and 50.
page = parseInt(page, 10) || 1;
pageSize = parseInt(pageSize, 10) || 50;
const articles = await Articles.aggregate([
{
$facet: {
metadata: [{ $count: 'totalCount' }],
data: [{ $skip: (page - 1) * pageSize }, { $limit: pageSize }],
},
},
]);
return res.status(200).json({
success: true,
articles: {
metadata: { totalCount: articles[0].metadata[0].totalCount, page, pageSize },
data: articles[0].data,
},
});
} catch (error) {
return res.status(500).json({ success: false });
}
};In the above example, we have implemented a pagination solution utilizing the Aggregation Framework, which eliminates the risk of data inconsistencies and enhances performance by reducing the number of necessary database calls.
The example output of the above solution would appear in the following format:
{
metadata: {
totalCount: 100,
page: 1,
pageSize: 50,
},
data: [
{
_id: 1,
title: 'Article 1',
},
{
_id: 2,
title: 'Article 2',
},
...
],
};Pipeline Structure for Optimal Performance
The order of stages in the aggregation pipeline is important. If you start the pipeline with the $facet stage, MongoDB will not use indexes, which can make your query slower. To avoid this, place the $facet stage at the end of the pipeline for optimal pagination.
If the $facet stage is the first stage in a pipeline, the stage will perform a COLLSCAN. The $facet stage does not make use of indexes if it is the first stage in the pipeline.
If the $facet stage comes later in the pipeline and earlier stages have used indexes, $facet will not trigger a COLLSCAN during execution.
In general, the order of stages in the pipeline should be as follows:
- $match
- $sort
- $facet
An example of code that demonstrates the aggregation pipeline structure for the optimal pagination is as follows:
exports.getArticles = async (req, res) => {
let { page, pageSize } = req.query;
try {
// If "page" and "pageSize" are not sent we will default them to 1 and 50.
page = parseInt(page, 10) || 1;
pageSize = parseInt(pageSize, 10) || 50;
const articles = await Articles.aggregate([
{
$match: {
// Your filtering criteria
}
},
{
$sort: {
// Your sorting criteria
}
},
{
$facet: {
metadata: [{ $count: 'totalCount' }],
data: [{ $skip: (page - 1) * pageSize }, { $limit: pageSize }],
},
},
]);
return res.status(200).json({
success: true,
articles: {
metadata: { totalCount: articles[0].metadata[0].totalCount, page, pageSize },
data: articles[0].data,
},
});
} catch (error) {
return res.status(500).json({ success: false });
}
};Conclusion
Pagination is an important feature in most applications as it allows for the efficient retrieval of large sets of data.
The only right way to implement pagination in MongoDB is through the use of the MongoDB Aggregation Framework. In the context of pagination, the $facet stage is of particular importance. This stage allows for the simultaneous processing of the same set of data, resulting in the retrieval of both the total number of documents and the paginated batch in a single, efficient database query.
By using the $facet stage, developers can avoid the common mistake of fetching the total number of items and the paginated batch separately, which can lead to data inconsistency and decreased performance.
By mastering the MongoDB Aggregation Framework and utilizing it for pagination, developers can improve the performance and user experience of their applications.