Skip to content

Commit

Permalink
vfs: fix backends being Shutdown too early when startup takes a long …
Browse files Browse the repository at this point in the history
…time

Before this change if the VFS took more than 5 to initialise (which
can happen if there is a lot of files or a lot of files which need
uploading) the backend was dropped out of the cache before the VFS was
fully created.

This was noticeable in the dropbox backend where the batcher Shutdown
too soon and prevented further uploads.

This fixes the problem by Pinning backends before the VFS cache is
created.

https://forum.rclone.org/t/if-more-than-251-elements-in-the-que-to-upload-fails-with-batcher-is-shutting-down/38076/2
  • Loading branch information
ncw committed May 18, 2023
1 parent 3115ede commit fc798d8
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions vfs/vfs.go
Original file line number Diff line number Diff line change
Expand Up @@ -227,13 +227,14 @@ func New(f fs.Fs, opt *vfscommon.Options) *VFS {
fs.Logf(f, "--vfs-cache-mode writes or full is recommended for this remote as it can't stream")
}

vfs.SetCacheMode(vfs.Opt.CacheMode)

// Pin the Fs into the cache so that when we use cache.NewFs
// with the same remote string we get this one. The Pin is
// removed when the vfs is finalized
cache.PinUntilFinalized(f, vfs)

// This can take some time so do it after the Pin
vfs.SetCacheMode(vfs.Opt.CacheMode)

return vfs
}

Expand Down

0 comments on commit fc798d8

Please sign in to comment.