Using Forced Update Order in your Flows

The problem - Lack of order in multiple updates

In integrations, updates are reflected in response to a Trigger from one app to the other. A Trigger notifies Integry of specific events by sending webhooks, and those webhooks are utilized in the Actions that are performed in response to that Trigger.

Many apps send a high number of webhooks in small timeframes. These webhooks are processed by Integry, and the corresponding Actions are performed on them. In an ideal world, the events would always be sent to us in the right order. However, because of a number of reasons like network congestion, service load levels, messages being sent of different machines, parallelism, etc, these events tend to arrive out-of-order. This can lead to the Integry engine performing Actions in the incorrect order. So, the latest webhook not being processed at the end causes the integration to miss important updates. Again, this might not matter if your integrations are quiet and not as noisy or if the sending system guarantees ordering (which is rare).

Let’s understand this problem better with an example. Say that you have an integration of Asana with Google Tasks. So the tasks between both apps are synced, and updating a task’s description or title in one app will reflect in the other. If you just change the title of a task in Asana using a few words, it will send several webhooks to Integry. After changing every few characters, a webhook will be received on our back end.

For example, if you have a task titled “My First Task,” and you change it to:

“My First Task - edited by John."

This will provide us with several webhooks for this title update. The titles in each webhook will be something like this:

1. My first task - edi
2. My first task - edited
3. My first task - edited by 
4. My first task - edited by John

Our back-end engine starts processing the above webhooks in parallel. But the order for their processing is not guaranteed. The 4th webhook can be processed first, then the 2nd, then 1st, and finally the 3rd. So this will result in the corresponding title in Google Tasks being updated to:

“My First Task - edited by”

The Force Update Order feature

Having order in updates from one app to the other is important to counter the above problem. We provide you with the Force Update Order feature in the Flow builder as a solution to this. You can give your updates a sequence with a single click and ensure no updates are missed.

You can enable Update Order for your Flow with the toggle button found in the Flow User Interface configuration section, as seen below.

The corresponding back-end algorithms for Update Order become available in Flows when the above toggle is on. The following are the detailed prerequisites of this feature’s functionality. 

Requirements to enable Forced Update Order

1. Toggle for Forced Update Order in your Flow should be on.
2. The Objects should be properly defined and mapped in the incoming endpoints for the Triggers used in the Flow for both apps. 
3. Your app needs to provide a Last Updated timestamp. This should be an accurate time and should not overlap with any other webhook so that it can be identified and sorted by our back-end engine. In the Object output for the endpoint, this timestamp should be mapped with the “last_updated” property of the object. So, the “last_updated” property should always be mapped. From Asana API documentation, we can see that the timestamp associated with updating a task is found in the “created_at” field. 

So the “last_updated” property will be mapped in the object similar to this:

Doing the above will provide our back-end engine with sufficient data to handle the ordering of updates.

How it works

Once you turn on the toggle for Forced Order, it is handled automatically. Let’s look at how it will work for the current example.

Whenever Asana sends an update through a webhook, it also sends an associated timestamp, mapped with the Object property named “last_updated.” This represents the time of the update very accurately (up to microseconds). So it will be different for each of the above example webhooks. Let’s see those webhooks with the associated timestamp now. Please note that this has been simplified for your understanding.

1. My first task - edi                          (“Last updated” : “09:49:43.2488”)
2. My first task - edited                     (“Last updated” : “09:49:43.8723”)
3. My first task - edited by                (“Last updated” : “09:49:44.3113”)
4. My first task - edited by John       (“Last Updated” : “09:49:45.0082”)

As soon as a webhook starts getting processed by our back-end execution engine, it divides the above four webhooks into two parts: one getting executed and the rest that are in the queue for execution. So let’s assume the 3rd webhook is processed first. The webhooks will now be:

In process:

My first task - edited by

In queue:

1. My first task - edi                             (“Last updated” : “09:49:43.2488”)
2. My first task - edited                        (“Last updated” : “09:49:43.8723”)
3. My first task - edited by John          (“Last Updated” : “09:49:45.0082”)

From the webhooks not being executed, it checks the associated timestamps. Those webhooks whose timestamping are more recent than those being executed are kept in the queue for processing afterward. And the rest of the webhooks with an older timestamp than the current one are discarded. This is because as a newer webhook has been processed, it is assumed that all the older changes have already been included in it. So now this will be the status of webhooks.

In process:

My first task - edited by

In queue:

My first task - edited by John                  (“Last Updated”: “09:49:45.0082”)

After the processing of this webhook, the next webhook in the queue starts being processed. The same process will be followed if there is still more than one webhook in the queue. One of the webhooks is taken up for processing, and the timestamp for the remaining webhooks is checked. Similarly, the ones with an older timestamp are discarded, and the remaining ones with newer timestamps are again added to the queue for processing. This is how this process will carry on until no further webhooks remain in the queue.

Note that there will always be one webhook under process, and the rest are either in the queue or discarded. This rids us from losing data due to the uncertainty of order associated with processing multiple webhooks at a time. Because we drop older updates, we always convert partial updates to full updates by fetching the entire object with a Get call. Because of this reason, each update translates to 3 tasks and is more expensive to run.

Object and Integration combination

Forced Updated Order also ensures that the combination of Object ID and the Integration ID is maintained. This means that your updates for the same Object, but in different integrations, can be processed in parallel. Similarly, updates for two different Objects in the same integration can be processed simultaneously.

Learn more about creating your Flow from scratch.