From 6e90488d0532d867570efb879d35242bca061520 Mon Sep 17 00:00:00 2001 From: Sean Morris Date: Mon, 8 Nov 2021 18:32:11 -0800 Subject: [PATCH 1/6] Added Documentation for creating downloader plugins Documentation related to creating plugins. --- README-PLUGINS.md | 92 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 README-PLUGINS.md diff --git a/README-PLUGINS.md b/README-PLUGINS.md new file mode 100644 index 00000000..e623cce2 --- /dev/null +++ b/README-PLUGINS.md @@ -0,0 +1,92 @@ +# nbgitpuller - downloader plugin documentation + +nbgitpuller uses [pluggy](https://pluggy.readthedocs.io/en/stable/) as a framework +to load any installed nbgitpuller-downloader plugins. There are three downloader plugins +available right now: +- [nbgitpuller-downloader-googledrive](https://github.com/jupyterhub/nbgitpuller-downloader-googledrive) +- [nbgitpuller-downloader-dropbox](https://github.com/jupyterhub/nbgitpuller-downloader-dropbox) +- [nbgitpuller-downloader-generic-web](https://github.com/jupyterhub/nbgitpuller-downloader-generic-web) + + +There are several pieces to be aware of for the plugin to work correctly: +1. The setup.cfg(or setup.py) file must have the entry_points definition. +For example: +> [options.entry_points] +nbgitpuller = +      dropbox=nbgitpuller_downloader_dropbox.dropbox_downloader + +2. The file referenced for use by nbgitpuller in the plug-in (the above example is looking for the +file, dropbox_downloader) must implement the function handle_files(query_line_args) and be decorated with `@hookimpl`. +3. As a consequence of this, the following must be imported: + - `from nbgitpuller.hookspecs import hookimpl` +4. The implementation of the handle_files function in your plugin needs to return + two pieces of information: + - the name of the folder, the archive is in after decompression + - the path to the local git repo mimicking a remote origin repo + +nbgitpuller provides a function in plugin_helper.py called handle_files_helper that handles the downloading +and returning of the correct information if given a URL, the extension of the +file to decompress(zip or tar) and the progress function(I will describe that +more later) but you are welcome to implement the functionality of handle_files_helper in your +plug-in. There may be use cases not covered by the currently available plugins like needing to authenticate against +the webserver or service where your archive is kept. Either way, it behooves you +to study the handle_files_helper function in nbgitpuller to get a sense of how this function +is implemented. + +For the rest of the steps, I refer you to the [nbgitpuller-downloader-dropbox](https://github.com/jupyterhub/nbgitpuller-downloader-dropbox) plugin. + ``` + @hookimpl + def handle_files(query_line_args): + query_line_args["repo"] = query_line_args["repo"].replace("dl=0", "dl=1") # dropbox: dl set to 1 + ext = determine_file_extension(query_line_args["repo"])` + query_line_args["extension"] = ext + loop = asyncio.get_event_loop() + tasks = handle_files_helper(query_line_args), query_line_args["progress_func"]() + result_handle, _ = loop.run_until_complete(asyncio.gather(*tasks)) + return result_handle + ``` + +The following pieces describe what happens in handle_files before, at least, in this case, we call +the handle_files_helper function: + +1) The parameter, query_line_args, is all the query line arguments you include on the nbgitpuller link. This means you + can put keyword arguments into your nbgitpuller links and have access to these arguments in the handle_files + function. + For example, you might set up a link like this: + http://[your hub]/hub/user-redirect/git-pull?repo=[link to your archive]&keyword1=value1&keyword2=value2&provider=dropbox&urlpath=tree%2F%2F + In your handle_files function, you could make this call to get your custom arguments: + + ``` + query_line_args["keyword1"] + query_line_args["keyword2"] + ``` +2) The query_line_args parameter also includes the progress function used to monitor the download_q + for messages; messages in the download_q are written to the UI so users can see the progress and + steps being taken to download their archives. You will notice the progress function is passed into + handle_files_helper and accessed like this: + ``` + query_line_args["progress_func"] + query_line_args["download_q"] + ``` +3) The first line of the handle_files function for the dropbox downloader is specific to DropBox. The URL to a file + in DropBox contains one URL query parameter(dl=0). This parameter indicates to Dropbox whether to download the + file or open it in their browser-based file system. In order to download the file, this parameter + needs to be changed to dl=1. +4) The next line determines the file extension (zip, tar.gz, etc). + This is added to the query_lines_args map and passed off to the handle_files_helper to + help the application know which utility to use to decompress the archive -- unzip or tar -xzf. +5) Since we don't want the user to have to wait while the download process finishes, we have made + downloading of the archive a non-blocking process using the package asyncio. Here are the steps: + - get the event loop + - setup two tasks: + - a call to the handle_files_helper with our arguments + - the progress_loop function + - execute the two tasks in the event loop. +6) The function returns two pieces of information to nbgitpuller: + - the directory name of the decompressed archive + - the local_origin_repo path. + +The details of what happens in handle_files_helper can be found by studying the function. Essentially, the archive is downloaded, decompressed, and set up in a file +system to act as a remote repository(e.g. the local_origin_repo path). + + From eb9a0d80878e3413d905432d650fdb154162645d Mon Sep 17 00:00:00 2001 From: sean-morris Date: Tue, 9 Nov 2021 11:34:21 -0800 Subject: [PATCH 2/6] Update README-PLUGINS.md Co-authored-by: Erik Sundell --- README-PLUGINS.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README-PLUGINS.md b/README-PLUGINS.md index e623cce2..864c596c 100644 --- a/README-PLUGINS.md +++ b/README-PLUGINS.md @@ -11,9 +11,11 @@ available right now: There are several pieces to be aware of for the plugin to work correctly: 1. The setup.cfg(or setup.py) file must have the entry_points definition. For example: -> [options.entry_points] -nbgitpuller = -      dropbox=nbgitpuller_downloader_dropbox.dropbox_downloader + + ```toml + [options.entry_points] + nbgitpuller = dropbox=nbgitpuller_downloader_dropbox.dropbox_downloader + ``` 2. The file referenced for use by nbgitpuller in the plug-in (the above example is looking for the file, dropbox_downloader) must implement the function handle_files(query_line_args) and be decorated with `@hookimpl`. From af598e74116381e8982a5784580ff498ef975caa Mon Sep 17 00:00:00 2001 From: sean-morris Date: Tue, 9 Nov 2021 11:34:50 -0800 Subject: [PATCH 3/6] Update README-PLUGINS.md Co-authored-by: Erik Sundell --- README-PLUGINS.md | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/README-PLUGINS.md b/README-PLUGINS.md index 864c596c..a090215c 100644 --- a/README-PLUGINS.md +++ b/README-PLUGINS.md @@ -36,17 +36,18 @@ to study the handle_files_helper function in nbgitpuller to get a sense of how t is implemented. For the rest of the steps, I refer you to the [nbgitpuller-downloader-dropbox](https://github.com/jupyterhub/nbgitpuller-downloader-dropbox) plugin. - ``` - @hookimpl - def handle_files(query_line_args): - query_line_args["repo"] = query_line_args["repo"].replace("dl=0", "dl=1") # dropbox: dl set to 1 - ext = determine_file_extension(query_line_args["repo"])` - query_line_args["extension"] = ext - loop = asyncio.get_event_loop() - tasks = handle_files_helper(query_line_args), query_line_args["progress_func"]() - result_handle, _ = loop.run_until_complete(asyncio.gather(*tasks)) - return result_handle - ``` + +```python +@hookimpl +def handle_files(query_line_args): + query_line_args["repo"] = query_line_args["repo"].replace("dl=0", "dl=1") # dropbox: dl set to 1 + ext = determine_file_extension(query_line_args["repo"])` + query_line_args["extension"] = ext + loop = asyncio.get_event_loop() + tasks = handle_files_helper(query_line_args), query_line_args["progress_func"]() + result_handle, _ = loop.run_until_complete(asyncio.gather(*tasks)) + return result_handle +``` The following pieces describe what happens in handle_files before, at least, in this case, we call the handle_files_helper function: From 8b912148f3822a440eb3b7eb63451515277564b0 Mon Sep 17 00:00:00 2001 From: sean-morris Date: Tue, 9 Nov 2021 11:35:03 -0800 Subject: [PATCH 4/6] Update README-PLUGINS.md Co-authored-by: Erik Sundell --- README-PLUGINS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-PLUGINS.md b/README-PLUGINS.md index a090215c..84c35397 100644 --- a/README-PLUGINS.md +++ b/README-PLUGINS.md @@ -59,7 +59,7 @@ the handle_files_helper function: http://[your hub]/hub/user-redirect/git-pull?repo=[link to your archive]&keyword1=value1&keyword2=value2&provider=dropbox&urlpath=tree%2F%2F In your handle_files function, you could make this call to get your custom arguments: - ``` + ```python query_line_args["keyword1"] query_line_args["keyword2"] ``` From 06f95d61ef5451ce710b8e05691ac627ee5fa96a Mon Sep 17 00:00:00 2001 From: sean-morris Date: Tue, 9 Nov 2021 11:35:09 -0800 Subject: [PATCH 5/6] Update README-PLUGINS.md Co-authored-by: Erik Sundell --- README-PLUGINS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README-PLUGINS.md b/README-PLUGINS.md index 84c35397..95bfcf99 100644 --- a/README-PLUGINS.md +++ b/README-PLUGINS.md @@ -67,9 +67,9 @@ the handle_files_helper function: for messages; messages in the download_q are written to the UI so users can see the progress and steps being taken to download their archives. You will notice the progress function is passed into handle_files_helper and accessed like this: - ``` - query_line_args["progress_func"] - query_line_args["download_q"] + ```python + query_line_args["progress_func"] + query_line_args["download_q"] ``` 3) The first line of the handle_files function for the dropbox downloader is specific to DropBox. The URL to a file in DropBox contains one URL query parameter(dl=0). This parameter indicates to Dropbox whether to download the From 0b9a5fc9c2854d88177f3e500e145c9dd1b6b456 Mon Sep 17 00:00:00 2001 From: Sean Morris Date: Tue, 9 Nov 2021 15:30:13 -0800 Subject: [PATCH 6/6] Moved documentation for creating downloader plugins to docs --- README-PLUGINS.md => docs/topic/nbgitpuller-downloder-plugins.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename README-PLUGINS.md => docs/topic/nbgitpuller-downloder-plugins.md (100%) diff --git a/README-PLUGINS.md b/docs/topic/nbgitpuller-downloder-plugins.md similarity index 100% rename from README-PLUGINS.md rename to docs/topic/nbgitpuller-downloder-plugins.md