Pagination
Any app where you want to get records from can have a lot of data. If all this data is accessed in a single call, it can overwhelm our systems. The backend engine’s memory can be exhausted as it maintains all that information. As a solution to this problem, many APIs provide a method to get a specific amount of data on a page-by-page basis. This process of pagination is generally used in actions such as bulk import of contacts.
Implementing pagination in Integry
You can create pagination based endpoints in our app. These are generally used in a query. A paginated query fetches a page of records and runs all its child steps (in the template) one by one. It keeps fetching the data until all the pages are exhausted. Let’s go through an example to get a better understanding of this concept.
Consider we want to get a list of subscribers from MailChimp in a paginated form. We’ll make the query for that by going into the Queries tab in the application settings menu. The page shows the existing queries and a button “Create Query” as shown below. Create a new query by clicking this button.
Add the basic descriptions, activity fields, and select authorization type. See the details on how to make a query here. Next comes the Endpoint step. Select the Create new endpoint option from the Endpoint dropdown menu as shown below.
The endpoint creation form will appear. You can follow the basic procedure of filling an endpoint form here. It is the same for all endpoint types. As in our current example, we are making a call to an external API and requesting data, so this is the Outgoing Endpoint Type.
Request URL
We can see from the MailChimp Api documentation, for getting information about members in a list, a request has to be sent to the following URL:
The {list_id} variable in the above URL is the unique id for the list that the user provides in the activity field. We know that the values of the activity fields are available under the requestBody Twig object. So we can create the field for the list ID and give it the machine name ‘mailchimp_list’. To get the value of this field, we’ll use the variable requestBody.data.mailchimp_list. The above API call URL will then become:
Right below the Request URL section, you can see the Enable Pagination Toggle button.
Once you toggle the above button on, several Pagination Fields will appear.
Integry supports all types of pagination, including offset based, cursor-based, page number based, URL based and custom.
Here are the details of the pagination fields:
Pagination Fields
Limit Parameter: This is the parameter that will pass the count/page size value to the URL. The name of this variable that exists in the API goes in the blank field here. In the running example of Mailchimp, it is "count". The actual value of count i.e. how many records/contacts should be returned per page in one request goes in the second blank field. If it is left blank, the count is set to the default value of the API. For Mailchimp’s API, it is 10 by default.
Offset Parameter: This is the key that will pass the number of records to skip from the overall dataset so we can get the subsequent records. In the running example of Mailchimp, it is "offset". In our app, the backend engine is configured to automatically determine its value. For example, if the count is 2 and it is the first call, the offset will be zero. For the next request, it will skip the first two records, so its value will be 2. Next value will be 4 and so on.
Total Items Variable: An API usually tells us the total number of items or records in the response of every request. The name of that parameter can be given here. This is read by our execution engine. We use this to figure out the number of iterations to run. This is an optional field.
You can click on the Show Advanced Options buttons to get more optional parameters as shown below.
API Page Size: APIs sometimes return the page size in the response body. We use this as the value of the limit parameter.
API Offset: APIs may also return the next offset value. We use this to specify the offset parameter.
Next Variable Link: Some APIs don’t have a Total Items Parameter and they return the URL for the next page instead. We can specify the parameter of that URL here.
More Items Parameter: APIs can have a Boolean parameter that specifies that there are more records. We can give that parameter here.
Increment Offset: Typically we increment the offset by the page size that is mentioned in the Limit Parameter field. You can set this to a constant such as 1 if your API uses page-based pagination.
Reading Variables:
While setting the pagination fields, you can specify where to read the variables from by selecting one of the options in the dropdown menu.
A variable can be sent via Query Parameter, Header, or Json body by selecting an option from the dropdown as shown below.
For the running example, we can send the limit parameter and offset in the query parameters.
Response template
If we give the value of the limit variable as 2 for a provided list ID and also set the URL to only get a few specific variables like ID and email address, the resulting request URL sent will be:
The response sent by Mailchimp is shown below:
{
"members":[
{
"id":"3bfab1fd390ff98ce9589ae3470f5b21",
"email_address":"user50@example.com",
"merge_fields":{
"FNAME":"User",
"LNAME":"Fifty",
"PHONENUMBER":""
}
},
{
"id":"b62dcbb04081981c3bda423ab3df2a62",
"email_address":"user51@example.com",
"merge_fields":{
"FNAME":"Fifty",
"LNAME":"One",
"PHONENUMBER":""
}
}
],
"total_items":12
}
This remote response we received contains an array of “members” on the top level. This is just the response that we need for our paginated bulk import endpoint. So we will not change this response. Our response template will be:
{{ response.contacts | json_encode | raw }}
Next, you need to set Object Output for your trigger. This is followed by the header details.
Header Details
The parameters that are sent along with the URL as headers can be specified here. These can be given as shown below.
After adding the headers, click on the Create Endpoint button at the bottom. The example above is specific to Mailchimp. The details of templates, endpoint URL and headers vary from app to app.