Pagination
Any app where you want to get records from can have a lot of data. If all this data is accessed in a single call, it can overwhelm our systems. The backend engine’s memory can be exhausted as it maintains all that information. As a solution to this problem, many APIs provide a method to get a specific amount of data on a page-by-page basis. This process of pagination is generally used in actions such as bulk import of contacts.
Implementing pagination in Integry
You can create pagination based endpoints in our app. These are generally used in a query. A paginated query fetches a page of records and runs all its child steps (in the template) one by one. It keeps fetching the data until all the pages are exhausted. Let’s go through an example to get a better understanding of this concept.
Consider we want to get contacts from ActiveCampaign in a paginated form. We’ll make the query for that by going into the Queries option inside Activities of Connectors. You can select Query and start setting it up.
Add the name, description & activity fields first. Then connect your account from where you want to fetch your contacts from.
See the details on how to make a query here. Next comes the Endpoint step. Once you set up the above configurations, the next step is to configure the API call.
Since this is a query so the HTTP method GET is pre-selected, because we will be requesting data from the API. You will have to provide the base URL & the endpoint from where we can fetch contacts (these details can be found under the respective app's API documentation).
For the running example, we have filled in the details of URL as well as Headers as per the Active Campaign's API documentation. Also, note the Pagination toggle is turned off for now and we will send the request by clicking the Test Query button.
Once the call is sent, you can view the result under the Iteration.
As we have not enabled the pagination, we will be returned only one iteration. You can also view the Response that we received and the Request that we made.
Request
Response
If you turn on the pagination toggle in the API Call Section & then send the Test Call, the query will return results on multiple pages. As soon as you turn on the toggle, you will see a number of pagination configurations supported, depending on the API of your app, you can set the configurations up. For this example, we will enable the pagination and provide the configurations in the URL as can be seen below.
In this case, you will receive ten contacts per page. All contacts can be viewed across various iterations. You can see the "Next Page" button in this case. To retrieve the next ten contacts, click Next Page
You can view the next iteration
If you expand the iteration you will see the request URL for each iteration as well
Pagination Fields
Limit Parameter: This is the parameter that will pass the count/page size value to the URL. The name of this variable that exists in the API goes in the blank field here.
Offset Parameter: This is the key that will pass the number of records to skip from the overall dataset so we can get the subsequent records. I
Total Items Variable: An API usually tells us the total number of items or records in the response of every request. The name of that parameter can be given here. This is read by our execution engine. We use this to figure out the number of iterations to run. This is an optional field.
You can click on the Show Advanced Options buttons to get more optional parameters as shown below.
API Page Size: APIs sometimes return the page size in the response body. We use this as the value of the limit parameter.
API Offset: APIs may also return the next offset value. We use this to specify the offset parameter.
Next Link Link: Some APIs don’t have a Total Items Parameter and they return the URL for the next page instead. We can specify the parameter of that URL here.
More Items Parameter: APIs can have a Boolean parameter that specifies that there are more records. We can give that parameter here.
Increment Offset: Typically we increment the offset by the page size that is mentioned in the Limit Parameter field. You can set this to a constant such as 1 if your API uses page-based pagination.
Reading Variables:
While setting the pagination fields, you can specify where to read the variables from by selecting one of the options in the dropdown menu.
A variable can be sent via URL, Header, or Body by selecting an option from the dropdown as shown below.
