Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Azure Blob Storage SDK to v12 #2573

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bf69e03
Upgrade Azure to v12
pamelafox Nov 21, 2024
e106bc8
Update changelog
pamelafox Nov 21, 2024
073b7e7
New lines
pamelafox Nov 21, 2024
e50a3b8
Remove comment
pamelafox Nov 21, 2024
11b1c7d
Why is test failing
pamelafox Nov 21, 2024
30a6304
Rename dev containers
pamelafox Nov 22, 2024
8c87d91
Fix dev container config
pamelafox Nov 22, 2024
ff6f3dc
More remote print debugging
pamelafox Nov 23, 2024
f78c728
Update Azurite emulator for tests
pamelafox Nov 23, 2024
95dd260
update minimum reqs for azure-storage-blob
pamelafox Nov 23, 2024
a8af6dc
Apply pre-commit
pamelafox Nov 23, 2024
70508d5
Change how import error is handled
pamelafox Nov 23, 2024
a7a8e5f
Update the min reqs for 3.9
pamelafox Nov 23, 2024
da835a8
Revert unneeded tox change
pamelafox Nov 23, 2024
cde5d70
Lower numpy version
pamelafox Nov 23, 2024
c526059
Update tests, readme, dev container
pamelafox Nov 25, 2024
96c6048
Update devcontainer
pamelafox Nov 25, 2024
d7167eb
Make devcontainer work for all services
pamelafox Nov 25, 2024
4d39865
Add ports to devcontainer
pamelafox Nov 25, 2024
c700069
Add link to the admin
pamelafox Nov 25, 2024
8dc71bc
Run pre-commit
pamelafox Nov 25, 2024
f22343e
Change constructor to receive client, change download to send_file in…
pamelafox Nov 26, 2024
18d1c33
Update changelog
pamelafox Nov 26, 2024
71e6889
Remove type annotations to match S3 client
pamelafox Nov 26, 2024
59c752a
Merge branch 'master' into azureupgrade
pamelafox Nov 30, 2024
4e611de
Add to latest version
pamelafox Nov 30, 2024
b989875
Show timestamp for directories
pamelafox Dec 12, 2024
129a245
File issues
pamelafox Dec 12, 2024
52311f5
Use requires_sync=False for compatibility with connection strings
pamelafox Jan 6, 2025
4f86809
Fix ruff issues
pamelafox Jan 6, 2025
9801c0a
Force precommit to run
pamelafox Jan 6, 2025
1955b03
Sort imports
pamelafox Jan 6, 2025
c66dfe3
Merge branch 'master' into azureupgrade
pamelafox Jan 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .devcontainer/azure/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ARG IMAGE=bullseye
FROM mcr.microsoft.com/devcontainers/${IMAGE}
14 changes: 14 additions & 0 deletions .devcontainer/azure/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// For format details, see https://aka.ms/devcontainer.json.
{
"name": "flask-admin (Python + Azurite)",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, I could add Postgres and Mongo to this dev container too, to have a single container that can run all the tests. It should be fairly easy given tests.yaml has the services setup, just copying that to docker-compose.yaml.

"dockerComposeFile": "docker-compose.yaml",
"service": "app",
"workspaceFolder": "/workspace",
"forwardPorts": [10000, 10001],
"portsAttributes": {
"10000": {"label": "Azurite Blob Storage Emulator", "onAutoForward": "silent"},
"10001": {"label": "Azurite Blob Storage Emulator HTTPS", "onAutoForward": "silent"}
},
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "vscode"
}
31 changes: 31 additions & 0 deletions .devcontainer/azure/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
version: '3'

services:
app:
build:
context: .
dockerfile: Dockerfile
args:
IMAGE: python:3.12

volumes:
- ../..:/workspace:cached

# Overrides default command so things don't shut down after the process ends.
command: sleep infinity

# Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
network_mode: service:storage

storage:
container_name: azurite
image: mcr.microsoft.com/azure-storage/azurite:latest
restart: unless-stopped
volumes:
- storage-data:/data

# Add "forwardPorts": ["10000", "10001"] to **devcontainer.json** to forward Azurite locally.
# (Adding the "ports" property to this file will not forward from a Codespace.)

volumes:
storage-data:
8 changes: 8 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// For format details, see https://aka.ms/devcontainer.json
{
"name": "flask-admin (Python 3.12)",
"image": "mcr.microsoft.com/devcontainers/python:3.12-bullseye",

// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "vscode"
}
2 changes: 1 addition & 1 deletion .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:
ports:
- 27017:27017
azurite:
image: arafato/azurite:2.6.5
image: mcr.microsoft.com/azure-storage/azurite:latest
env:
executable: blob
ports:
Expand Down
2 changes: 2 additions & 0 deletions doc/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Changelog
2.0.0a2
-------

* Azure Blob Storage SDK has been upgraded from the legacy version (v2) to the latest version (v12). All functionality remains the same, but the dependency is now `azure-storage-blob>=12.0.0`.

Breaking changes:

* Removed support for Python 3.8.
Expand Down
29 changes: 29 additions & 0 deletions examples/azure-blob-storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Azure Blob Storage Example

Flask-Admin example for an Azure Blob Storage account.

To run this example:

1. Clone the repository and navigate to this example::

git clone https://github.com/pallets-eco/flask-admin.git
cd flask-admin/examples/azure-storage

2. Create and activate a virtual environment::

python -m venv venv
source venv/bin/activate

3. Install requirements::

pip install -r requirements.txt

4. Either run the Azurite Blob Storage emulator or create an actual Azure Blob Storage account. Set this environment variable:

export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"

The value below is the default for the Azurite emulator.

4. Run the application::

python app.py
Empty file.
19 changes: 19 additions & 0 deletions examples/azure-blob-storage/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import os

from flask import Flask
from flask_admin import Admin
from flask_admin.contrib.fileadmin.azure import AzureFileAdmin
from flask_babel import Babel

app = Flask(__name__)
app.config["SECRET_KEY"] = "secret"
admin = Admin(app)
babel = Babel(app)
file_admin = AzureFileAdmin(
container_name="fileadmin-tests",
connection_string=os.getenv("AZURE_STORAGE_CONNECTION_STRING"),
)
admin.add_view(file_admin)

if __name__ == "__main__":
app.run(debug=True)
1 change: 1 addition & 0 deletions examples/azure-blob-storage/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../..[azure-blob-storage]
153 changes: 86 additions & 67 deletions flask_admin/contrib/fileadmin/azure.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
import os.path as op
from datetime import datetime
from datetime import timedelta
from time import sleep

try:
from azure.storage.blob import BlobPermissions
from azure.storage.blob import BlockBlobService
except ImportError:
BlobPermissions = BlockBlobService = None
from azure.core.exceptions import ResourceExistsError
from azure.storage.blob import BlobProperties
from azure.storage.blob import BlobSasPermissions
from azure.storage.blob import BlobServiceClient
from azure.storage.blob import generate_blob_sas
except ImportError as e:
raise Exception(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the handling of the importerror to make mypy happier. I looked at other optional modules and they all seemed to handle importerrors a bit differently. I found one that did it this way. This seems fine since you should import the module unless you're using it.

"Could not import `azure.storage.blob`. "
"Enable `azure-blob-storage` integration "
"by installing `flask-admin[azure-blob-storage]`"
) from e

from flask import redirect

Expand Down Expand Up @@ -47,28 +53,29 @@ def __init__(self, container_name, connection_string):
:param connection_string:
Azure Blob Storage Connection String
"""

if not BlockBlobService:
raise ValueError(
"Could not import `azure.storage.blob`. "
"Enable `azure-blob-storage` integration "
"by installing `flask-admin[azure-blob-storage]`"
)

self._container_name = container_name
self._connection_string = connection_string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the changes I made on the s3 admin side when bringing it up to date was to have __init__ take a client instance rather than parameters that get passed the client.

Do you think we should do something similar here and accept an instance of BlobServiceClient, or is it still fine to just use the connection string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I think thats nice, as I personally don't typically use connection strings (this was my first time using from_connection_string), so that gives developers more flexibility as to how they connect. I can make that change. That'd be breaking, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be, but this is scheduled to go out for the v2 release where we're making a bunch of breaking changes, so I'm ok with it.

If you'd be happy to, feel free :) 🙏

self.__client = None

@property
def _client(self):
if not self.__client:
self.__client = BlockBlobService(connection_string=self._connection_string)
self.__client.create_container(self._container_name, fail_on_exist=False)
self.__client = BlobServiceClient.from_connection_string(
self._connection_string
)
try:
self.__client.create_container(self._container_name)
except ResourceExistsError:
pass
return self.__client

@property
def _container_client(self):
return self._client.get_container_client(self._container_name)

@classmethod
def _get_blob_last_modified(cls, blob):
last_modified = blob.properties.last_modified
def _get_blob_last_modified(cls, blob: BlobProperties):
last_modified = blob.last_modified
tzinfo = last_modified.tzinfo
epoch = last_modified - datetime(1970, 1, 1, tzinfo=tzinfo)
return epoch.total_seconds()
Expand All @@ -93,7 +100,9 @@ def get_files(self, path, directory):
folders = set()
files = []

for blob in self._client.list_blobs(self._container_name, path):
container_client = self._client.get_container_client(self._container_name)

for blob in container_client.list_blobs(path):
blob_path_parts = blob.name.split(self.separator)
name = blob_path_parts.pop()

Expand All @@ -103,7 +112,7 @@ def get_files(self, path, directory):
if blob_is_file_at_current_level and not blob_is_directory_file:
rel_path = blob.name
is_dir = False
size = blob.properties.content_length
size = blob.size
last_modified = self._get_blob_last_modified(blob)
files.append((name, rel_path, is_dir, size, last_modified))
else:
Expand All @@ -125,18 +134,10 @@ def get_files(self, path, directory):
def is_dir(self, path):
path = self._ensure_blob_path(path)

num_blobs = 0
for blob in self._client.list_blobs(self._container_name, path):
blob_path_parts = blob.name.split(self.separator)
is_explicit_directory = blob_path_parts[-1] == self._fakedir
if is_explicit_directory:
blobs = self._container_client.list_blobs(name_starts_with=path)
for blob in blobs:
if blob.name != path:
return True

num_blobs += 1
path_cannot_be_leaf = num_blobs >= 2
if path_cannot_be_leaf:
return True

return False

def path_exists(self, path):
Expand All @@ -145,12 +146,13 @@ def path_exists(self, path):
if path == self.get_base_path():
return True

try:
next(iter(self._client.list_blobs(self._container_name, path)))
except StopIteration:
if path is None:
return False
else:

# Return true if it exists as either a directory or a file
for _ in self._container_client.list_blobs(name_starts_with=path):
return True
return False

def get_base_path(self):
return ""
Expand All @@ -160,80 +162,97 @@ def get_breadcrumbs(self, path):

accumulator = []
breadcrumbs = []
for folder in path.split(self.separator):
accumulator.append(folder)
breadcrumbs.append((folder, self.separator.join(accumulator)))
if path is not None:
for folder in path.split(self.separator):
accumulator.append(folder)
breadcrumbs.append((folder, self.separator.join(accumulator)))
return breadcrumbs

def send_file(self, file_path):
file_path = self._ensure_blob_path(file_path)

if not self._client.exists(self._container_name, file_path):
if file_path is None:
raise ValueError()
container_client = self._client.get_container_client(self._container_name)
if len(list(container_client.list_blobs(file_path))) != 1:
raise ValueError()

now = datetime.utcnow()
url = self._client.make_blob_url(self._container_name, file_path)
sas = self._client.generate_blob_shared_access_signature(
self._container_name,
file_path,
BlobPermissions.READ,

blob_client = self._client.get_blob_client(
container=self._container_name, blob=file_path
)
url = blob_client.url
account_name = self._connection_string.split(";")[1].split("=")[1]

delegation_key_start_time = now
delegation_key_expiry_time = delegation_key_start_time + timedelta(days=1)
user_delegation_key = self._client.get_user_delegation_key(
key_start_time=delegation_key_start_time,
key_expiry_time=delegation_key_expiry_time,
)
sas = generate_blob_sas(
account_name=account_name,
container_name=self._container_name,
blob_name=file_path,
user_delegation_key=user_delegation_key,
permission=BlobSasPermissions(read=True),
expiry=now + self._send_file_validity,
start=now - self._send_file_lookback,
)

return redirect(f"{url}?{sas}")

def read_file(self, path):
path = self._ensure_blob_path(path)

blob = self._client.get_blob_to_bytes(self._container_name, path)
return blob.content
if path is None:
raise ValueError("No path provided")
blob = self._container_client.get_blob_client(path).download_blob()
return blob.readall()

def write_file(self, path, content):
path = self._ensure_blob_path(path)

self._client.create_blob_from_text(self._container_name, path, content)
if path is None:
raise ValueError("No path provided")
self._container_client.upload_blob(path, content, overwrite=True)

def save_file(self, path, file_data):
path = self._ensure_blob_path(path)

self._client.create_blob_from_stream(
self._container_name, path, file_data.stream
)
if path is None:
raise ValueError("No path provided")
self._container_client.upload_blob(path, file_data.stream)

def delete_tree(self, directory):
directory = self._ensure_blob_path(directory)

for blob in self._client.list_blobs(self._container_name, directory):
self._client.delete_blob(self._container_name, blob.name)
for blob in self._container_client.list_blobs(directory):
self._container_client.delete_blob(blob.name)

def delete_file(self, file_path):
file_path = self._ensure_blob_path(file_path)

self._client.delete_blob(self._container_name, file_path)
if file_path is None:
raise ValueError("No path provided")
self._container_client.delete_blob(file_path)

def make_dir(self, path, directory):
path = self._ensure_blob_path(path)
directory = self._ensure_blob_path(directory)

if path is None or directory is None:
raise ValueError("No path provided")
blob = self.separator.join([path, directory, self._fakedir])
blob = blob.lstrip(self.separator)
self._client.create_blob_from_text(self._container_name, blob, "")
self._container_client.upload_blob(blob, b"")

def _copy_blob(self, src, dst):
src_url = self._client.make_blob_url(self._container_name, src)
copy = self._client.copy_blob(self._container_name, dst, src_url)
while copy.status != "success":
sleep(self._copy_poll_interval_seconds)
copy = self._client.get_blob_properties(
self._container_name, dst
).properties.copy
src_client = self._container_client.get_blob_client(src)
dst_blob = self._container_client.get_blob_client(dst)
dst_blob.start_copy_from_url(src_client.url, requires_sync=True)

def _rename_file(self, src, dst):
self._copy_blob(src, dst)
self.delete_file(src)

def _rename_directory(self, src, dst):
for blob in self._client.list_blobs(self._container_name, src):
for blob in self._container_client.list_blobs(src):
self._rename_file(blob.name, blob.name.replace(src, dst, 1))

def rename_path(self, src, dst):
Expand Down
Loading
Loading