How to get them all?

Sometimes you need more from your API than what you get from the first set of returned results.

Welcome to the world of pagination — spreading out results over a number of pages or sections. I’ll provide an overview into some basic pagination concepts, and dive deep into how we do things at Square.

The concept of pagination originally comes from books, but I think it’s easier to describe in terms of lists. You see pagination all the time on blogs, news sites, image sharing sites, etc. Whenever there isn’t enough room on the page to show all of the posts at once, the big list of posts is broken up into different “pages”. You are probably familiar with the UI components similar to the one below that help you navigate between pages.

Pagination with APIs works a little bit differently but has the same underlying concept. When you try to retrieve a list of objects that is too large, either for the computational costs associated with retrieval (or consumption) or any of the plethora of networking concerns (waiting for a few gigs of data from an API call can have unexpected effects on your application), the service will generally respond with information about how to access the next “page”, or section, of results.

I’m going to refer to this tidbit of information as a “cursor”, but it could have many names and forms including:

To help explain the concept a little more I made an animation of what a cursor added to a response looks like for our v2 List Transactions endpoint. Your code would make a request to the List Transactions endpoint, and if you have more than 50 transactions, the API would return the first 50 and attach another field to the json response called cursor. This is a pagination token, and attaching that to your next request (in the form of a URL parameter) tells the API which transactions you have already seen, and where to start when returning the next 50 results. That response includes a different pagination cursor for you to get the next 50 results. That continues until you have “paged through” all the results that you requested, with the last page usually having fewer results than the expected page size. You can see a conceptual example of what that looks like in the animation below.

In most cases, when you experience pagination, you likely want all of the items in the list, and aren’t very happy about the extra work required to page through the results. Here are some best practices to make it a little less painful to get all of your results:

Depending on your programming language of choice, pagination can be a wonderful use case for while. The basic workflow is that while you are getting a pagination token in your response, keep making subsequent requests. In pseudo code that might look like this:

As soon as your API stops responding with a pagination cursor, then you can stop looping and continue execution with the complete set of data. There are a couple important points to remember with an approach like this:

Related posts:

I dive deep into my favorite ideas that stirred in my mind because of one of the best TV shows of all time, Game of Thrones.

The sun was beaming down on the desert our city lies on. Sweltering in the heat, it was midday. Everyone and their neighbors were thirsty and hungry after the drought. Our living conditions were far…

Dedico esse texto aos experts e co-fundadores do Astella Expert Network — Alexandre Bazzi, Alexandre Tarifa, Nara Vaz e Theo Orosco. Os últimos 4 anos foram de extrema reflexão e evolução na Astella…