Blob.upload_from_file fails with non-seekable stream

[offsetcyan]

Environment details

• OS type and version: NixOS 23.05 (Stoat)
• Python version: Python 3.11.3
• pip version: pip 23.0
• google-cloud-storage version: 2.12.0

Steps to reproduce

1. Provide non-seekable stream to Blob.upload_from_file, i.e. Popen.stdout
2. Exception is raised

Code example

from subprocess import PIPE, Popen
from google.cloud.storage import Client

TARGET_BUCKET = <bucket name>
TARGET_BLOB = <blob name>

_client = Client()
_bucket = _client.bucket(TARGET_BUCKET)

with Popen(["echo", "some thing or whatever"], stdout=PIPE) as proc:
    blob = _bucket.blob(TARGET_BLOB)
    blob.upload_from_file(proc.stdout, content_type="text/plain")

Stack trace

Traceback (most recent call last):
  File "/home/dch/projects/fullfact/audio-transcriber/test.py", line 13, in <module>
    blob.upload_from_file(proc.stdout, content_type="text/plain")
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/cloud/storage/blob.py", line 2760, in upload_from_file
    self._prep_and_do_upload(
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/cloud/storage/blob.py", line 2601, in _prep_and_do_upload
    created_json = self._do_upload(
                   ^^^^^^^^^^^^^^^^
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/cloud/storage/blob.py", line 2424, in _do_upload
    response = self._do_resumable_upload(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/cloud/storage/blob.py", line 2242, in _do_resumable_upload
    upload, transport = self._initiate_resumable_upload(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/cloud/storage/blob.py", line 2116, in _initiate_resumable_upload
    upload.initiate(
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/resumable_media/requests/upload.py", line 402, in initiate
    method, url, payload, headers = self._prepare_initiate_request(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dch/projects/fullfact/audio-transcriber/.venv/lib/python3.11/site-packages/google/resumable_media/_upload.py", line 470, in _prepare_initiate_request
    if stream.tell() != 0:
       ^^^^^^^^^^^^^
OSError: [Errno 29] Illegal seek

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[andrewsg]

Apologies for the delay. Because uploads handle errors by rewinding to the last byte confirmed and sending the chunk over again, non-seekable streams are not supported by design. Special-casing non-seekable streams and disabling resumption of a resumable upload is possible and we will consider it as a feature request. In the meantime, I will update the API documentation in python-storage to make it clear that streams must be seekable for now.

[offsetcyan]

Would you like me to submit a separate issue labelled feature request or is this existing issue satisfactory?

[andrewsg]

No need to make a separate request as the work is already tracked on our side. Thanks!