Have you seen a change in the time taken to trigger a WebHook?

Lee Joramo

I use WebHook in a number of places in our Smartsheet system. In the past after a sheet was saved, WebHooks would fire and be received within a few seconds by my servers.

Now I am seeing a consistent 60 second delay after save for WebHooks to contact my servers.

I think this change has happened in the last 6 months.

For most of my use cases, this doesn't matter. WebHooks trigger events that update other systems, and my users never notice a delay.

However some of the WebHooks trigger updates to the sheet the user is currently working in. In the past the user would see their sheet be updated within 10 seconds of a save. Now it is around 70 seconds, that is a HUGE difference in user experience.

The impact of this to up-to-now has not been that big. The users who I know have experienced it, trigger the webhooks so infrequently that they assume it is a momentary glitch.

However, of some of these sheets are seasonal. and I will soon be having many more users doing lots of work on affected sheets.

What is your experience with how long it takes WebHooks to fire?

I am wondering if I should move to brute force API queries of the sheets every few seconds and not use WebHooks for these time sensitive tasks.

Paul Newcome

I've noticed a bit of a slowdown in Bridge the past few months as well. Previously, saves to a sheet were triggering my workflows within a few seconds, sometimes even before I could navigate back to Bridge and open op the run log. Now it is consistently right around 60 seconds (plus or minus one or two seconds) to the point where I just pull up a timer and wait to refresh my run logs. It gets much worse from around 11:00 - 11:30ish AM EST.

Lee Joramo

Paul, thanks for the info. I remember that talking Smartsheet coders at Engaged, they indicated that Bridge heavily used Webhooks behind the scenes. So this makes sense.

Paul Newcome

I wonder if you have noticed a substantial difference around the same times as well whether that particular issue is Bridge specific or Webhooks in general.

Lee Joramo

I haven't monitored this issue related to time of day. I will start to make note of that, and let you know.

Lee Joramo

Paul, I have setup logging to collect the time from when the sheet is saved to when my server receives the webhook call. It is very early, but it looks like I see some patterns:

Row Created webhooks take 60 seconds

Row Updated or Deleted webhooks take 30 or 60 seconds

If there is a mixture or Created, Updated or Deleted, it looks like it is run 60 seconds.

I have only been running this for about an hour Friday afternoon, so this is limited data. I will do a deep analysis later next week.

Paul Newcome

All of my Bridge triggers are when column values are updated (row updated in specific column). I will have to set some other triggers up to see how they run, but that may not be until later this week.

Lee Joramo

@Paul Newcome I read the API docs quite often, and just noticed that there is a Change Log.

There is an item for last October that says they added a "one-minute debounce" to WebHooks. So there is confirmation.

https://smartsheet.redoc.ly/#section/Changelog/2024-10-14

Paul Newcome

@Lee Joramo I saw the change log there, but the last time I looked at it, it wasn't in that same format and didn't go back quite that far. At least now we have an answer… Thanks!

prime_nathaniel

I almost exclusively work on Bridge and Integrations. It is lucky if you guys are seeing sub 1 minute, it is anything up to 10 minute sometimes now. What really kills me is the logging changes they made, you don't get logs for most runs until they finish, especially when you have numerous child or worst case recursive solutions.

If I had to guess the new architecture has moved all webhook processing to batches not event based. As you can imagine large scale event based sequential processing is probably part of the slow down issues we've seen in the past. Probably all webhooks just go into the giant queue and batches out now. The real question is whether all trigger events are in one giant queue or if those are fragmented in their entity model.

Reminder all automations, formulas, cell links, notifications, etc would also be competing for processing time / window. Traditionally, these triggers would also be hooked into the entity update post database so they only work on confirmed changes. I'd also be more than willing to be some of the oldest "core" code was calling other models directly in a synchronous fashion (and may still to give it preferential treatment)

.
If everything used to be on their own queues and webhooks were real time then that would explain why they used to be 5 seconds. Then moving to batches atleast upped it to X timeframe interval. But if they also unified event triggers into a single queue then rather than being behind 1 or 2 items you might be behind 50+.

Lee Joramo

@prime_nathaniel thanks for you insights.

I have been moving my more time critical processes back to using the API to "pull" changes and away from the WebHook "push".

For most of my use cases, the delay in webhooks makes them not very useful since I want them to respond to users actively working on sheets.

silas.frantz

Wow, yes, supremely secure feature. I need to validate my receiver app? Impressive, must be gatekeeping some rich payload content. Oh, it's just a skinny payload and I still have to pester the API to get any content… hmm, that's a lot of API calls… sort of what I was looking to avoid. Ohhhh I see, it's a gatekept, stripped down payload, delayed by some unknown amount of time — it's a feature for which resources don't need to be prioritized because we maxed out the scale for useless and secure. 👍️