What is the best way to deal with HTTP 429 - Throttled - requests with PnP ? #62
-
|
I have a script where I iterate through 22k items in a CSV file and update document sets for each item. So I'm basically repeatedly calling
However, every now and then the operation fails with HTTP 429 - Too Many Requests. Is there a standard way of capturing this and identifying the Wait/Retry time which I understand would be in the response header. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 24 replies
-
|
In principle there is nothing you need to do to handle 429s. PnP PowerShell makes use of the PnP Framework behind the scenes where we override way CSOM work with it comes to executing requests. We implemented an incremental backoff flow which means that if a 429 comes in we initially wait 0.5 seconds. Then we try again. If it still fails, we wait 2x0.5 seconds, then 2, then 4, then 8, then 16, 32, 64, 128, 256 seconds. We try up to a maximum of 10 times before giving up. That means that maximum wait time before we give up is 512 seconds, meaning more than 8 minutes. This delay has shown to handle the throttling errors is almost every case. We have seen that this covers throttling nicely. Do you by any change in your script run other cmdlets not coming from PnP PowerShell, as all PnP PowerShell cmdlets use the throttling handling mechanism I just described? |
Beta Was this translation helpful? Give feedback.
-
|
Another short update: we did some additional tests here and we are strongly considering actually switching our approach over to using the retry-after header value. |
Beta Was this translation helpful? Give feedback.
-
|
If we do that the -UseRetryAfterHeader property will be removed again from Connect-PnPOnline. Nevertheless I would be interested to see the tracelog from your end using that property with 0.3.13-nightly. |
Beta Was this translation helpful? Give feedback.
-
|
You'll notice a false positive in the tracelog : duplicate log entries in the case of throttling. The second line will show the actual value reported back by the retry-after header, the first line shows our initial delay time, wrongly stated as seconds where it actually is milliseconds. We obviously will fix that in the next release. |
Beta Was this translation helpful? Give feedback.
-
|
We decided to flip over to using the retry-after header value in general. I removed the switch from PowerShell again as it is now the default behavior. So 0.3.13-night contains the -UseRetryAfterHeader, 0.3.14-night and later will not contain the switch. |
Beta Was this translation helpful? Give feedback.
-
|
Is there some way to write throttling information to logs/stdout? Edit: I see that you answered pretty much what I wanted in the other thread on this page. |
Beta Was this translation helpful? Give feedback.
-
|
Literally just "Set-PnPListItem" over and over .. I randomly get 429 errors ... |
Beta Was this translation helpful? Give feedback.
-
|
How do you connect to your tenant (e.g. what connection method/parameters do you use)? And which version of PnP PowerShell do you use? The 'legacy' version (SharePointPnPPowerShellOnline) or the new PnP.PowerShell version? |
Beta Was this translation helpful? Give feedback.
-
|
Yep I'm running it through again with the TraceLog set ... I was definitely seeing Exceptions thrown from ExecuteQuery telling the server response was HTTP 429 .. But I'm seeing those "CSOM Request Frequency .. " entries in the tracelog too .. We will see (sometimes it takes 20-30 minutes before I get a 429) |
Beta Was this translation helpful? Give feedback.
-
|
Ok ... and it seems even after posting the screenshot .. I didn't notice the extra little "ExecuteQuery" which had snuck in there ... |
Beta Was this translation helpful? Give feedback.
-
|
too many threads here :-) But as I wrote in the other thread: we flipped the default behavior of the PnP.Framework to use the retry-after header now. As of PowerShell 0.3.14-nightly it will use that. |
Beta Was this translation helpful? Give feedback.
-
|
As this is still beta, I guess removing switch parameters is in order, but in general wouldn't it be best to avoid changes that will break code? :) |
Beta Was this translation helpful? Give feedback.
-
|
@MartinHatch using service principals instead of a normal user will be more resilient towards throttling. We often use multiple service principals to avoid throttling when migrating high volume content |
Beta Was this translation helpful? Give feedback.
-
|
Apologize for opening an old thread :) |
Beta Was this translation helpful? Give feedback.
-
|
PnP.Framework for most requests should use PnPFramework RetryHandler that handles throttling by using Retry-After. There is currently no support for RateLimit though (would like that very much!) The exception that I am aware of is provisioning templates, that uses native SPWeb methods, so that is a "black box" with no reply when u get throttled What kind of request did you get the above with? And what version? That looks like the "old" way of doing things, where they incremented wait time (not recommended) that should have been fixed in PnP.Framework |
Beta Was this translation helpful? Give feedback.
-
|
It is Get-PnPGroup
Get-PnPSiteCollectionAdmin |
Beta Was this translation helpful? Give feedback.
In principle there is nothing you need to do to handle 429s. PnP PowerShell makes use of the PnP Framework behind the scenes where we override way CSOM work with it comes to executing requests. We implemented an incremental backoff flow which means that if a 429 comes in we initially wait 0.5 seconds. Then we try again. If it still fails, we wait 2x0.5 seconds, then 2, then 4, then 8, then 16, 32, 64, 128, 256 seconds. We try up to a maximum of 10 times before giving up. That means that maximum wait time before we give up is 512 seconds, meaning more than 8 minutes. This delay has shown to handle the throttling errors is almost every case. We have seen that this covers throttling nicely.
D…