http: upload_fobj raises NotImplementedError

[skshetry]

dvc/dvc/fs/base.py:275: in upload
    return self.upload_fobj(wrapped, to_info, size=total)
dvc/dvc/fs/fsspec_wrapper.py:151: in upload_fobj
    with self.open(to_info, "wb") as fdest:
dvc/dvc/fs/fsspec_wrapper.py:87: in open
    return self.fs.open(self._with_bucket(path_info), mode=mode)
dvc/.env38/lib/python3.8/site-packages/fsspec/spec.py:978: in open
    f = self._open(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <fsspec.implementations.http.HTTPFileSystem object at 0x11200c460>
path = 'http://127.0.0.1:51993/34c8018b-3350-4f97-a44a-6f66fe6bde06/foo'
mode = 'wb', block_size = None, autocommit = True, cache_type = None
cache_options = None, size = None, kwargs = {}

    def _open(
        self,
        path,
        mode="rb",
        block_size=None,
        autocommit=None,  # XXX: This differs from the base class.
        cache_type=None,
        cache_options=None,
        size=None,
        **kwargs,
    ):
        """Make a file-like object

        Parameters
        ----------
        path: str
            Full URL with protocol
        mode: string
            must be "rb"
        block_size: int or None
            Bytes to download in one request; use instance value if None. If
            zero, will return a streaming Requests file-like instance.
        kwargs: key-value
            Any other parameters, passed to requests calls
        """
        if mode != "rb":
>           raise NotImplementedError
E           NotImplementedError

dvc/.env38/lib/python3.8/site-packages/fsspec/implementations/http.py:330: NotImplementedError

[efiop]

@skshetry What fsspec version do you have installed?

[skshetry]

pip show fsspec
Name: fsspec
Version: 2021.8.1
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Requires:
Required-by: sshfs, s3fs, ossfs, gcsfs, adlfs, dvc

[skshetry]

It came up when adding test for callbacks for http in iterative/dvc#6641.

[isidentical]

The HTTPFileSystem unfortunately does not support upload_fobj (or writing in any kind beside put_file). Is there are a use case for this? We can try to add buffering as we did with gdrive, and store everything in memory until we are done and upload it in one go.

[skshetry]

@isidentical, it's probably used for to_remote, I think?

[isidentical]

Ah, right. Will send a patch to the upstream.

[isidentical]

Awaiting fsspec/filesystem_spec#764

[skshetry]

@efiop, I don't think this is fixed on dvc's side. cc @isidentical