-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade Azure Blob Storage SDK to v12 #2573
base: master
Are you sure you want to change the base?
Changes from 13 commits
bf69e03
e106bc8
073b7e7
e50a3b8
11b1c7d
30a6304
8c87d91
ff6f3dc
f78c728
95dd260
a8af6dc
70508d5
a7a8e5f
da835a8
cde5d70
c526059
96c6048
d7167eb
4d39865
c700069
8dc71bc
f22343e
18d1c33
71e6889
59c752a
4e611de
b989875
129a245
52311f5
4f86809
9801c0a
1955b03
c66dfe3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
ARG IMAGE=bullseye | ||
FROM mcr.microsoft.com/devcontainers/${IMAGE} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
// For format details, see https://aka.ms/devcontainer.json. | ||
{ | ||
"name": "flask-admin (Python + Azurite)", | ||
"dockerComposeFile": "docker-compose.yaml", | ||
"service": "app", | ||
"workspaceFolder": "/workspace", | ||
"forwardPorts": [10000, 10001], | ||
"portsAttributes": { | ||
"10000": {"label": "Azurite Blob Storage Emulator", "onAutoForward": "silent"}, | ||
"10001": {"label": "Azurite Blob Storage Emulator HTTPS", "onAutoForward": "silent"} | ||
}, | ||
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root. | ||
"remoteUser": "vscode" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
version: '3' | ||
|
||
services: | ||
app: | ||
build: | ||
context: . | ||
dockerfile: Dockerfile | ||
args: | ||
IMAGE: python:3.12 | ||
|
||
volumes: | ||
- ../..:/workspace:cached | ||
|
||
# Overrides default command so things don't shut down after the process ends. | ||
command: sleep infinity | ||
|
||
# Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function. | ||
network_mode: service:storage | ||
|
||
storage: | ||
container_name: azurite | ||
image: mcr.microsoft.com/azure-storage/azurite:latest | ||
restart: unless-stopped | ||
volumes: | ||
- storage-data:/data | ||
|
||
# Add "forwardPorts": ["10000", "10001"] to **devcontainer.json** to forward Azurite locally. | ||
# (Adding the "ports" property to this file will not forward from a Codespace.) | ||
|
||
volumes: | ||
storage-data: |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
// For format details, see https://aka.ms/devcontainer.json | ||
{ | ||
"name": "flask-admin (Python 3.12)", | ||
"image": "mcr.microsoft.com/devcontainers/python:3.12-bullseye", | ||
|
||
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root. | ||
"remoteUser": "vscode" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Azure Blob Storage Example | ||
|
||
Flask-Admin example for an Azure Blob Storage account. | ||
|
||
To run this example: | ||
|
||
1. Clone the repository and navigate to this example:: | ||
|
||
git clone https://github.com/pallets-eco/flask-admin.git | ||
cd flask-admin/examples/azure-storage | ||
|
||
2. Create and activate a virtual environment:: | ||
|
||
python -m venv venv | ||
source venv/bin/activate | ||
|
||
3. Install requirements:: | ||
|
||
pip install -r requirements.txt | ||
|
||
4. Either run the Azurite Blob Storage emulator or create an actual Azure Blob Storage account. Set this environment variable: | ||
|
||
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;" | ||
|
||
The value below is the default for the Azurite emulator. | ||
|
||
4. Run the application:: | ||
|
||
python app.py |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
import os | ||
|
||
from flask import Flask | ||
from flask_admin import Admin | ||
from flask_admin.contrib.fileadmin.azure import AzureFileAdmin | ||
from flask_babel import Babel | ||
|
||
app = Flask(__name__) | ||
app.config["SECRET_KEY"] = "secret" | ||
admin = Admin(app) | ||
babel = Babel(app) | ||
file_admin = AzureFileAdmin( | ||
container_name="fileadmin-tests", | ||
connection_string=os.getenv("AZURE_STORAGE_CONNECTION_STRING"), | ||
) | ||
admin.add_view(file_admin) | ||
|
||
if __name__ == "__main__": | ||
app.run(debug=True) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../..[azure-blob-storage] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,19 @@ | ||
import os.path as op | ||
from datetime import datetime | ||
from datetime import timedelta | ||
from time import sleep | ||
|
||
try: | ||
from azure.storage.blob import BlobPermissions | ||
from azure.storage.blob import BlockBlobService | ||
except ImportError: | ||
BlobPermissions = BlockBlobService = None | ||
from azure.core.exceptions import ResourceExistsError | ||
from azure.storage.blob import BlobProperties | ||
from azure.storage.blob import BlobSasPermissions | ||
from azure.storage.blob import BlobServiceClient | ||
from azure.storage.blob import generate_blob_sas | ||
except ImportError as e: | ||
raise Exception( | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I changed the handling of the importerror to make mypy happier. I looked at other optional modules and they all seemed to handle importerrors a bit differently. I found one that did it this way. This seems fine since you should import the module unless you're using it. |
||
"Could not import `azure.storage.blob`. " | ||
"Enable `azure-blob-storage` integration " | ||
"by installing `flask-admin[azure-blob-storage]`" | ||
) from e | ||
|
||
from flask import redirect | ||
|
||
|
@@ -47,28 +53,29 @@ def __init__(self, container_name, connection_string): | |
:param connection_string: | ||
Azure Blob Storage Connection String | ||
""" | ||
|
||
if not BlockBlobService: | ||
raise ValueError( | ||
"Could not import `azure.storage.blob`. " | ||
"Enable `azure-blob-storage` integration " | ||
"by installing `flask-admin[azure-blob-storage]`" | ||
) | ||
|
||
self._container_name = container_name | ||
self._connection_string = connection_string | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One of the changes I made on the s3 admin side when bringing it up to date was to have Do you think we should do something similar here and accept an instance of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh I think thats nice, as I personally don't typically use connection strings (this was my first time using from_connection_string), so that gives developers more flexibility as to how they connect. I can make that change. That'd be breaking, right? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be, but this is scheduled to go out for the v2 release where we're making a bunch of breaking changes, so I'm ok with it. If you'd be happy to, feel free :) 🙏 |
||
self.__client = None | ||
|
||
@property | ||
def _client(self): | ||
if not self.__client: | ||
self.__client = BlockBlobService(connection_string=self._connection_string) | ||
self.__client.create_container(self._container_name, fail_on_exist=False) | ||
self.__client = BlobServiceClient.from_connection_string( | ||
self._connection_string | ||
) | ||
try: | ||
self.__client.create_container(self._container_name) | ||
except ResourceExistsError: | ||
pass | ||
return self.__client | ||
|
||
@property | ||
def _container_client(self): | ||
return self._client.get_container_client(self._container_name) | ||
|
||
@classmethod | ||
def _get_blob_last_modified(cls, blob): | ||
last_modified = blob.properties.last_modified | ||
def _get_blob_last_modified(cls, blob: BlobProperties): | ||
last_modified = blob.last_modified | ||
tzinfo = last_modified.tzinfo | ||
epoch = last_modified - datetime(1970, 1, 1, tzinfo=tzinfo) | ||
return epoch.total_seconds() | ||
|
@@ -93,7 +100,9 @@ def get_files(self, path, directory): | |
folders = set() | ||
files = [] | ||
|
||
for blob in self._client.list_blobs(self._container_name, path): | ||
container_client = self._client.get_container_client(self._container_name) | ||
|
||
for blob in container_client.list_blobs(path): | ||
blob_path_parts = blob.name.split(self.separator) | ||
name = blob_path_parts.pop() | ||
|
||
|
@@ -103,7 +112,7 @@ def get_files(self, path, directory): | |
if blob_is_file_at_current_level and not blob_is_directory_file: | ||
rel_path = blob.name | ||
is_dir = False | ||
size = blob.properties.content_length | ||
size = blob.size | ||
last_modified = self._get_blob_last_modified(blob) | ||
files.append((name, rel_path, is_dir, size, last_modified)) | ||
else: | ||
|
@@ -125,18 +134,10 @@ def get_files(self, path, directory): | |
def is_dir(self, path): | ||
path = self._ensure_blob_path(path) | ||
|
||
num_blobs = 0 | ||
for blob in self._client.list_blobs(self._container_name, path): | ||
blob_path_parts = blob.name.split(self.separator) | ||
is_explicit_directory = blob_path_parts[-1] == self._fakedir | ||
if is_explicit_directory: | ||
blobs = self._container_client.list_blobs(name_starts_with=path) | ||
for blob in blobs: | ||
if blob.name != path: | ||
return True | ||
|
||
num_blobs += 1 | ||
path_cannot_be_leaf = num_blobs >= 2 | ||
if path_cannot_be_leaf: | ||
return True | ||
|
||
return False | ||
|
||
def path_exists(self, path): | ||
|
@@ -145,12 +146,13 @@ def path_exists(self, path): | |
if path == self.get_base_path(): | ||
return True | ||
|
||
try: | ||
next(iter(self._client.list_blobs(self._container_name, path))) | ||
except StopIteration: | ||
if path is None: | ||
return False | ||
else: | ||
|
||
# Return true if it exists as either a directory or a file | ||
for _ in self._container_client.list_blobs(name_starts_with=path): | ||
return True | ||
return False | ||
|
||
def get_base_path(self): | ||
return "" | ||
|
@@ -160,80 +162,97 @@ def get_breadcrumbs(self, path): | |
|
||
accumulator = [] | ||
breadcrumbs = [] | ||
for folder in path.split(self.separator): | ||
accumulator.append(folder) | ||
breadcrumbs.append((folder, self.separator.join(accumulator))) | ||
if path is not None: | ||
for folder in path.split(self.separator): | ||
accumulator.append(folder) | ||
breadcrumbs.append((folder, self.separator.join(accumulator))) | ||
return breadcrumbs | ||
|
||
def send_file(self, file_path): | ||
file_path = self._ensure_blob_path(file_path) | ||
|
||
if not self._client.exists(self._container_name, file_path): | ||
if file_path is None: | ||
raise ValueError() | ||
container_client = self._client.get_container_client(self._container_name) | ||
if len(list(container_client.list_blobs(file_path))) != 1: | ||
raise ValueError() | ||
|
||
now = datetime.utcnow() | ||
url = self._client.make_blob_url(self._container_name, file_path) | ||
sas = self._client.generate_blob_shared_access_signature( | ||
self._container_name, | ||
file_path, | ||
BlobPermissions.READ, | ||
|
||
blob_client = self._client.get_blob_client( | ||
container=self._container_name, blob=file_path | ||
) | ||
url = blob_client.url | ||
account_name = self._connection_string.split(";")[1].split("=")[1] | ||
|
||
delegation_key_start_time = now | ||
delegation_key_expiry_time = delegation_key_start_time + timedelta(days=1) | ||
user_delegation_key = self._client.get_user_delegation_key( | ||
key_start_time=delegation_key_start_time, | ||
key_expiry_time=delegation_key_expiry_time, | ||
) | ||
sas = generate_blob_sas( | ||
account_name=account_name, | ||
container_name=self._container_name, | ||
blob_name=file_path, | ||
user_delegation_key=user_delegation_key, | ||
permission=BlobSasPermissions(read=True), | ||
expiry=now + self._send_file_validity, | ||
start=now - self._send_file_lookback, | ||
) | ||
|
||
return redirect(f"{url}?{sas}") | ||
|
||
def read_file(self, path): | ||
path = self._ensure_blob_path(path) | ||
|
||
blob = self._client.get_blob_to_bytes(self._container_name, path) | ||
return blob.content | ||
if path is None: | ||
raise ValueError("No path provided") | ||
blob = self._container_client.get_blob_client(path).download_blob() | ||
return blob.readall() | ||
|
||
def write_file(self, path, content): | ||
path = self._ensure_blob_path(path) | ||
|
||
self._client.create_blob_from_text(self._container_name, path, content) | ||
if path is None: | ||
raise ValueError("No path provided") | ||
self._container_client.upload_blob(path, content, overwrite=True) | ||
|
||
def save_file(self, path, file_data): | ||
path = self._ensure_blob_path(path) | ||
|
||
self._client.create_blob_from_stream( | ||
self._container_name, path, file_data.stream | ||
) | ||
if path is None: | ||
raise ValueError("No path provided") | ||
self._container_client.upload_blob(path, file_data.stream) | ||
|
||
def delete_tree(self, directory): | ||
directory = self._ensure_blob_path(directory) | ||
|
||
for blob in self._client.list_blobs(self._container_name, directory): | ||
self._client.delete_blob(self._container_name, blob.name) | ||
for blob in self._container_client.list_blobs(directory): | ||
self._container_client.delete_blob(blob.name) | ||
|
||
def delete_file(self, file_path): | ||
file_path = self._ensure_blob_path(file_path) | ||
|
||
self._client.delete_blob(self._container_name, file_path) | ||
if file_path is None: | ||
raise ValueError("No path provided") | ||
self._container_client.delete_blob(file_path) | ||
|
||
def make_dir(self, path, directory): | ||
path = self._ensure_blob_path(path) | ||
directory = self._ensure_blob_path(directory) | ||
|
||
if path is None or directory is None: | ||
raise ValueError("No path provided") | ||
blob = self.separator.join([path, directory, self._fakedir]) | ||
blob = blob.lstrip(self.separator) | ||
self._client.create_blob_from_text(self._container_name, blob, "") | ||
self._container_client.upload_blob(blob, b"") | ||
|
||
def _copy_blob(self, src, dst): | ||
src_url = self._client.make_blob_url(self._container_name, src) | ||
copy = self._client.copy_blob(self._container_name, dst, src_url) | ||
while copy.status != "success": | ||
sleep(self._copy_poll_interval_seconds) | ||
copy = self._client.get_blob_properties( | ||
self._container_name, dst | ||
).properties.copy | ||
src_client = self._container_client.get_blob_client(src) | ||
dst_blob = self._container_client.get_blob_client(dst) | ||
dst_blob.start_copy_from_url(src_client.url, requires_sync=True) | ||
|
||
def _rename_file(self, src, dst): | ||
self._copy_blob(src, dst) | ||
self.delete_file(src) | ||
|
||
def _rename_directory(self, src, dst): | ||
for blob in self._client.list_blobs(self._container_name, src): | ||
for blob in self._container_client.list_blobs(src): | ||
self._rename_file(blob.name, blob.name.replace(src, dst, 1)) | ||
|
||
def rename_path(self, src, dst): | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the future, I could add Postgres and Mongo to this dev container too, to have a single container that can run all the tests. It should be fairly easy given tests.yaml has the services setup, just copying that to docker-compose.yaml.