Pagination is the process of splitting data into discrete pages, and you should implement it when you are building RESTful Web APIs that potentially serve huge amount of data. Imagine you have thousands or maybe millions of records in your database and your API endpoint try to return all of them at once. That would definitely take down the entire performance of your application and worst, it can make your application unusable and inefficient.

In this post, we are going to look at how we can implement the standard offset pagination and generate Hypermedia as the Engine of Application State (a.k.a HATEOAS) links to enable users easily consume our APIs.

Getting Started

Let's go ahead and fire up Visual Studio 2019 and create a new ASP.NET Core Web Application project.

You can name your project to whatever you want. But for this exercise, we'll just name the project as PaginationDemo.

Now, select ASP.NET Core Web API from the project template selection and create it.

To make this exercise more practical, we'll setup some test data using an in-memory data store instead of using the static weatherforecast data provided by the default template.

Configuring In-memory Data Store

Let's add the required NuGet package into our project. As you may already known, there are a few ways to add NuGet package dependencies in Visual Studio; you could either use the Package Manager Console(PMC), via NuGet Package Manager(NPM) or even adding them directly into your project file (.csproj). Choice is yours 😉.  

Go ahead and install the latest versions of the following packages:

  • Microsoft.EntityFrameworkCore
  • Microsoft.EntityFrameworkCore.InMemory

Next, we'll create a simple data access layer where we can implement code related to data operations. Depending on your project and the architecture you follow, your data access or persistence layer might be in a separate project to value a clean separation of concerns. But for the simplicity of this exercise, we will contain all the code within the Web API project itself.

Now, create a couple of new folders at the root of the project with the following structure:

PaginationDemo (root)
└───Data (folder)
    └───Entities (folder)

Add the following class within the Entities folder:

using System;

namespace PaginationDemo.Data.Entities
{
    public class Hobby
    {
        public Guid Id { get; set; }
        public string Name { get; set; }
        public DateTime CreatedAt { get; set; }
    }
}

Nothing fancy there. The preceding code defines a Hobby class that house some properties. We will use this class later to populate each property with test data.

Let's move on and create the following class within the Data folder:

using Microsoft.EntityFrameworkCore;
using PaginationDemo.Data.Entities;

namespace PaginationDemo.Data
{
    public class HobbyDbContext : DbContext
    {
        public HobbyDbContext(DbContextOptions<HobbyDbContext> options)
        : base(options) { }
        public DbSet<Hobby> Hobbies { get; set; }
    }
}

The preceding code defines a HobbyDbContext class and a Hobbiesproperty that represents a DbSet. Entity Framework Core requires a DbContext for us to query the data store. This is typically done by creating a class that inherits from the DbContext class as shown in the preceding code.

Now that we have configured a DBContext to be able to work with in-memory data, let's continue and setup some tests data. The following is the code:

namespace PaginationDemo.Data
{
    public class FakeDataSeeder
    {
        public static void Seed(IServiceProvider serviceProvider)
        {
            using var context = new HobbyDbContext(
                      serviceProvider
                      .GetRequiredService<DbContextOptions<HobbyDbContext>>());

            if (context.Hobbies.Any()) { return; }

            var hobbies = new List<Hobby>
            {
                new Hobby{ Name = "Cooking", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Listening to Music", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Drinking Beer", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Playing Guitar", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Blogging", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Vlogging", CreatedAt = DateTime.Now },
                new Hobby{ Name = "Travelling", CreatedAt = DateTime.Now },
            }

           context.Hobbies.AddRange(hobbies);

           context.SaveChanges();
        }
    }
}

The preceding code defines a Seed() method which will initialize a few Hobby data set when the application starts. This is done by adding the data into the Hobbies DbSet of the HobbyDbContext.

At this point, we now have a DbContext and a data seeder helper class in place. What we need to do next is to wire them into the Startup.cs class to get our data populated.

Add the following code within ConfigureServices() method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddDbContext<HobbyDbContext>(options => options.UseInMemoryDatabase("HobbyDb"));

    //removed other code for brevity
}

The preceding code registers a DbContext with an in-memory database called "HobbyDb".

Now, add the following code within Configure() method:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        //removed other code for brevity
        
        //initialize data seeds
        using var serviceScope = app.ApplicationServices
                                    .GetRequiredService<IServiceScopeFactory>()
                                    .CreateScope();

        var service = serviceScope.ServiceProvider;

        FakeDataSeeder.Seed(service);

    }

    //removed other code for brevity
}

Defining Models / DTOs

As a quick overview, Data Transfer Objects (DTOs) are classes that defines a Model with sometimes predefined validation in place for HTTP responses and request. You can think of DTOs as ViewModels in MVC where you only want to expose relevant data to the View. The basic idea about having a DTO is to decouple them from the actual Entity classes that are used by the data access layer to populate the data. This way, when a requirement changes or if your Entity properties are changed, they won't be affected and won’t break your API. In simple terms, your Entity classes should only be used for database related process, and your DTOs should only be used for taking request inputs, response output and only expose properties that you want your users to see. Depending on what architectural pattern you follow for structuring your API project, there could be cases that an Entity is mapped to a Model, and a Model is mapped to the DTO and vice versa.

For this exercise, we will just map directly our Entity to a DTO. But before we do that, let's first create a skeleton Model that will hold some properties for pagination. The following is the code:

namespace PaginationDemo.Models
{
    public class PagedModel<TModel>
    {
        const int MaxPageSize = 500;
        private int _pageSize;
        public int PageSize
        {
            get => _pageSize;
            set => _pageSize = (value > MaxPageSize) ? MaxPageSize : value;
        }

        public int CurrentPage { get; set; }
        public int TotalItems { get; set; }
        public int TotalPages { get; set; }
        public IList<TModel> Items { get; set; }

        public PagedModel()
        {
            Items = new List<TModel>();
        }
    }
}

The preceding code is a generic class that takes in a Model of type TModel. We did that so any Models that require pagination can use it.

Now, let's create an extension method that handles the actual pagination against the database. The following is the code:

namespace PaginationDemo.Extensions
{
    public static class DataPagerExtension
    {
        public static async Task<PagedModel<TModel>> PaginateAsync<TModel>(
            this IQueryable<TModel> query,
            int page,
            int limit,
            CancellationToken cancellationToken)
            where TModel : class
        {

            var paged = new PagedModel<TModel>();

            page = (page < 0) ? 1 : page;

            paged.CurrentPage = page;
            paged.PageSize = limit;

            var totalItemsCountTask = query.CountAsync(cancellationToken);

            var startRow = (page - 1) * limit;
            paged.Items = await query
                       .Skip(startRow)
                       .Take(limit)
                       .ToListAsync(cancellationToken);

            paged.TotalItems = await totalItemsCountTask;
            paged.TotalPages = (int)Math.Ceiling(paged.TotalItems / (double)limit);

            return paged;
        }
    }
}

The preceding code calculates and sets the metadata for pagination inlcuding the CurrentPage, TotalPages, TotalItems and the actual rows returned. It uses the LINQ Skip() and Take() operators to perform offset pagination against the data. The actual result is stored in the Items property of the PagedModel class.

Now, let's create a couple of DTOs for getting the list of paginated data. Let's start by creating the GetHobbyResponseDto. The following is the code:

namespace PaginationDemo.Dtos
{
    public record GetHobbyResponseDto
    {
        public Guid Id { get; init; }
        public string Name { get; init; }
        public DateTime CreatedAt { get; init; }
    }
}

The preceding code uses a record as a type instead of a class. Record type is one of the great features added in C# 9 which is by default integrated when you are already targetting to .NET 5 runtime. We won't be covering the details of record types in this post and other new features in C# 9. For more information, I would recommend reading this article here.

Here's the code for the GetHobbyListResponseDto:

using System.Collections.Generic;

namespace PaginationDemo.Dtos
{
    public record GetHobbyListResponseDto
    {
        public int CurrentPage { get; init; }

        public int TotalItems { get; init; }

        public int TotalPages { get; init; }

        public List<GetHobbyResponseDto> Items { get; init; }
    }
}

The preceding code also uses a recod type and defines a few propeprties that will be exposed to the users as an Http response.

At this point, we now have the peices that we need to perform pagination. Let's continue and create a service that will fetch data from our in-memory data store.

Creating a Service

In real application, we don't want our data access layer to be exposed within the Controller level. Instead, we will abstract our DbContext call into a service class so that our Controllers won't depend directly to it. Abstracting the code logic in a separate service is a way on making your Controller thin and simple however, we don’t want the Controller to directly depend on the actual service implementation as it can leads to tightly-coupled dependencies. To avoid tight-coupling, we will create an interface abstraction to decouple the actual service dependency. This makes your code more testable, extensible and easier to manage.

Let's go a head and create an interface with the following definition:

namespace PaginationDemo.Interfaces
{
    public interface IHobbyService
    {
        Task<GetHobbyListResponseDto> GetByPageAsync(int limit, int page, CancellationToken cancellationToken);
    }
}

The following is the code for the service that implements the interface:

namespace PaginationDemo.Services
{
    public class HobbyService: IHobbyService
    { 
        private readonly HobbyDbContext _dbContext;
        public HobbyService(HobbyDbContext dbContext)
        {
            _dbContext = dbContext;
        }

        public async Task<GetHobbyListResponseDto> GetByPageAsync(int limit, int page, CancellationToken cancellationToken)
        {

            var hobbies = await _dbContext.Hobbies
                           .AsNoTracking()
                           .OrderBy(p => p.CreatedAt)
                           .PaginateAsync(page, limit, cancellationToken);

            return new GetHobbyListResponseDto {
                CurrentPage = hobbies.CurrentPage,
                TotalPages = hobbies.TotalPages,
                TotalItems = hobbies.TotalItems,
                Items = hobbies.Items.Select(p => new GetHobbyResponseDto
                {
                    Id = p.Id,
                    Name = p.Name,
                    CreatedAt = p.CreatedAt
                }).ToList()
            };
        }
    }
}

The preceding code is responsible for querying data against the conceptual data Model. We’ve used the AsNoTracking() method built-in to EF Core to improve the query performance. No tracking queries are much quicker because it eliminates the need to setup change tracking information for the entity, thus they are quicker to execute and improve query performance for read-only data. It also uses the OrderBy() LINQ extension method to order the result based on CreatedAt property. Ordering the data first ensures consistency when the data is requested. It then invoke the PaginateAsync() extension method that we’ve implemented earlier to chunk the data based on Page and Limit values. Finally, we construct the Items data using LINQ Method-Based Query.

If you want to see the actual SQL script generated by EF Core or if you prefer to use raw SQL script to query the data, checkout this article here.

In order to resolved the service dependency at runtime, we need to register the service mapping into the DI container. Add the following code within the ConfigureServices() method of Startup.cs file:

services.AddTransient<IHobbyService, HobbyService>();

The preceding code tells the DI framework to resolve the required dependency that have been injected into a class constructor at run time.

To format and serialized the JSON response properly into a camel-case structure, then set the following configuration:

services.AddControllers()
        .AddJsonOptions(ops =>
        {
            ops.JsonSerializerOptions.IgnoreNullValues = true;
            ops.JsonSerializerOptions.WriteIndented = true;
            ops.JsonSerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase;
            ops.JsonSerializerOptions.DictionaryKeyPolicy = JsonNamingPolicy.CamelCase;
            ops.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter());
        });

The preceding code is I think self-explanatory and there's not much to talk about it. Let's move on to the next steps.

Creating a Controller

Create an empty API Controller and name it as HobbiesController. The following is the code:

namespace PaginationDemo.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class HobbiesController : ControllerBase
    {
        private readonly IHobbyService _service;

        public HobbiesController(IHobbyService service)
        {
            _service = service;
        }

        [HttpGet(Name = nameof(GetHobbyListAsync))]
        [ProducesResponseType(typeof(GetHobbyListResponseDto), Status200OK)]
        [ProducesResponseType(typeof(ProblemDetails), Status400BadRequest)]
        public async Task<IActionResult> GetHobbyListAsync(
            [FromQuery] UrlQueryParameters urlQueryParameters, 
            CancellationToken cancellationToken)
        {

            if (!ModelState.IsValid)
            {
                return BadRequest();
            }

            var hobbies = await _service.GetByPageAsync(
                                    urlQueryParameters.Limit,
                                    urlQueryParameters.Page,
                                    cancellationToken);

            return Ok(hobbies);
        }

    }

    public record UrlQueryParameters(int Limit = 50, int Page = 1);
}

The preceding code injects the IHobbyService as dependency to the Controller class constructor. This decouples your data access implementation from the Controller class allowing more flexibility when you decide to change the underlying implementation of the service, as well as making your Controller thin and clean.

Keep in mind though that there are various options that you can opt to for making your Controller classes as thin as possible. One popular example is using Mediator/MediatR.

The HobbiesController class has contains a GET endpoint called GetHobbyListAsync which takes in a UrlQueryParameters as QueryString parameters. The method is responsible for delegating the request down to the service and returns the response. Clean and simple. :wink:

Test Output

Now let's doa  quick test. Run your application and nagivate to the following endpoint:

https://localhost:44342/api/hobbies?limit=2&page=2

The response output should look similar to the following:

{
  "currentPage": 2,
  "totalItems": 7,
  "totalPages": 4,
  "items": [
    {
      "id": "bf663ca6-312d-4c74-af47-6ee0b0649477",
      "name": "Drinking Beer",
      "createdAt": "2021-02-09T18:51:46.6286203-06:00"
    },
    {
      "id": "f56feffd-8945-4d0c-a27f-4c012fd220b1",
      "name": "Playing Guitar",
      "createdAt": "2021-02-09T18:51:46.6286207-06:00"
    }
  ]
}

If the limit and page QueryString parameters are not supplied, the API will return all the records based on the predefined limit.

Cool! We now have a pagination in place. Let's move on to the next step and implement HATEOAS links to the existing response.

Integrating Resource Links (HATEOAS)

To get started implementing this feature, we need to define a couple of common references. Create the following record and enum:

public record LinkedResource(string Href);

public enum LinkedResourceType
{
    None,
    Prev,
    Next
}

The LinkedResource contains an Href property which will hold the route Url for the resource. Of course, you can add whatever properties that you want to include. The LinkedResource holds the possibile resource types. For example, your resource links will have metadata for Create, Update Get and Delete operations. Depending on your needs, this is where you add the type of resource that you want to tie to the response. For this exercise, we will just be focusing on the Next and Prev resource links to make our existing paginated response easily to consume.

Now, let's go ahead and create the following interface:

namespace PaginationDemo.Interfaces
{
    public interface ILinkedResource
    {
        IDictionary<LinkedResourceType, LinkedResource> Links { get; set; }
    }
}

The preceding code is an interface that defines a property called Links. This enable us to create dynamic links based on LinkedResourceType and on top of that, allowing us to easily create an extension method for adding links.

Let's go ahead and create an extension method for adding link resources. The following is the code:

namespace PaginationDemo.Extensions
{
    public static class LinkedResourceExtension
    {
        public static void AddResourceLink(this ILinkedResource resources,
                  LinkedResourceType resourceType, 
                  string routeUrl)
        {
            resources.Links ??= new Dictionary<LinkedResourceType, LinkedResource>();
            resources.Links[resourceType] = new LinkedResource(routeUrl);
        }
    }
}

The preceding code adds an entry to the Links collection based on the LinkedResourceType. The this keyword in the method argument denotes that the method is an extension method of the type ILinkedResource.

Now, let's update our GetHobbyListResponseDto class to implement the ILinkedResource interface. The following is the updated code:

namespace PaginationDemo.Dtos
{
    public record GetHobbyListResponseDto: ILinkedResource
    {
        public int CurrentPage { get; init; }

        public int TotalItems { get; init; }

        public int TotalPages { get; init; }

        public List<GetHobbyResponseDto> Items { get; init; }

        public IDictionary<LinkedResourceType, LinkedResource> Links { get; set; }
    }
}

Finally, let's create the method to generate the links. The following is the code that I added direclty to the Controller class as a private member:

private GetHobbyListResponseDto GeneratePageLinks(UrlQueryParameters 
                     queryParameters, 
                     GetHobbyListResponseDto response)
{

    if (response.CurrentPage > 1)
    {
        var prevRoute = Url.RouteUrl(nameof(GetHobbyListAsync), new { limit = queryParameters.Limit, page = queryParameters.Page - 1 });

        response.AddResourceLink(LinkedResourceType.Prev, prevRoute);

    }

    if (response.CurrentPage < response.TotalPages)
    {
        var nextRoute = Url.RouteUrl(nameof(GetHobbyListAsync), new { limit = queryParameters.Limit, page = queryParameters.Page + 1 });

        response.AddResourceLink(LinkedResourceType.Next, nextRoute);
    }

    return response;
}

The GeneratePageLinks method from the preceding code takes a UrlQueryParameters and GetHobbyListResponseDto as parameters. It then checks if the current response has next (response.CurrentPage < response.TotalPages) or has previous (response.CurrentPage > 1) data sets and create the corresponding resource link. Since the  GetHobbyListResponseDto implements the ILinkedResource interface, it can then automatically invoke the AddResourceLink() extension method that we have created earlier. This makes our code becomes fluent and clean.

Now, we can update our GetHobbyListAsync() method to return the following response:

 return Ok(GeneratePageLinks(urlQueryParameters,hobbies));

And we're done!

Testing the Response

Now, when you run the application again and navigate to: https://localhost:44342/api/hobbies?limit=2&page=2

You should be presented with the following response output:

{
  "currentPage": 2,
  "totalItems": 7,
  "totalPages": 4,
  "items": [
    {
      "id": "bf663ca6-312d-4c74-af47-6ee0b0649477",
      "name": "Drinking Beer",
      "createdAt": "2021-02-09T18:51:46.6286203-06:00"
    },
    {
      "id": "f56feffd-8945-4d0c-a27f-4c012fd220b1",
      "name": "Playing Guitar",
      "createdAt": "2021-02-09T18:51:46.6286207-06:00"
    }
  ],
  "links": {
    "Prev": {
      "href": "/api/Hobbies?limit=2&page=1"
    },
    "Next": {
      "href": "/api/Hobbies?limit=2&page=3"
    }
  }
}

Sweet! Notice the links attribute from the response now where you can see the Next and Prev routes for the page.

That's it! I hope you find this post helpful. Feel free to share this to fellow developers and provide feedback. I'd be happy to adjust the code for improvements. Cheers! 🍻

Source Code

You can fork or download the source code here:

proudmonkey/Blog.PaginationHateoasDemo
Contribute to proudmonkey/Blog.PaginationHateoasDemo development by creating an account on GitHub.

References

Dependency injection in ASP.NET Core
Learn how ASP.NET Core implements dependency injection and how to use it.
Use record types - C# tutorial
Learn about how to use record types, build hierarchies of records, and when to choose records over classes.
Method-Based Query Syntax Examples: Projection (LINQ to DataSet) - ADO.NET
Learn more about: Method-Based Query Syntax Examples: Projection (LINQ to DataSet)
Raw SQL Queries - EF Core
Using raw SQL for queries in Entity Framework Core