[grid] Support file downloads on the node

[krmahadevan]

Fixes: #9218

On the node, set the directory to where the browser will download files to via:

a. The CLI argument --downloads-path (or)
b. via the Toml syntax of

[node]
downloads-path = "/path/to/dir/goes/here"

The GET end-points that will support file
download would be (The file will ALWAYS BE DOWNLOADED AS A ZIP FILE):

• /session/:sessionId:/se/file?filename=

Thanks for contributing to Selenium!
A PR well described will help maintainers to quickly review and merge it

Before submitting your PR, please check our contributing guidelines.
Avoid large PRs, help reviewers by making them as simple and short as possible.

Description

Motivation and Context

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• I have read the contributing document.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[diemol]

Thanks @krmahadevan! I left some comments.

[krmahadevan]

@diemol - I have fixed all the review comments, rebased off of trunk and added the changes as an additional commit for ease of review. Please help take a look

[diemol]

I checked out this PR to run a complete test but it did not work. Probably because the get method for downloadsPath configuration is not being used.

The client code I used to perform a test was:

package org.example;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.io.Zip;
import org.openqa.selenium.json.Json;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.http.HttpClient;
import org.openqa.selenium.remote.http.HttpMethod;
import org.openqa.selenium.remote.http.HttpRequest;
import org.openqa.selenium.remote.http.HttpResponse;

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Map;

import static org.openqa.selenium.remote.http.Contents.string;

public class Main {

  public static void main(String[] args) throws InterruptedException, IOException {
    FirefoxOptions options = new FirefoxOptions();
    options.addPreference("browser.download.manager.showWhenStarting", false);
    options.addPreference("browser.helperApps.neverAsk.saveToDisk", "images/jpeg, application/pdf, application/octet-stream");
    options.addPreference("pdfjs.disabled", true);

    URL gridUrl = new URL("http://localhost:4444");
    RemoteWebDriver driver =  new RemoteWebDriver(gridUrl, options);
    driver.get("http://the-internet.herokuapp.com/download");
    WebElement element = driver.findElement(By.cssSelector(".example a"));
    element.click();

    Thread.sleep(10 * 1000);

    HttpRequest request = new HttpRequest(
      HttpMethod.GET,
      String.format("/session/%s/se/file", driver.getSessionId()));
    request.addQueryParameter("filename", "my_appointments-1.pdf");
    try (HttpClient client = HttpClient.Factory.createDefault().createClient(gridUrl)) {
      HttpResponse response = client.execute(request);
      Map<String, Object> map = new Json().toType(string(response), Json.MAP_TYPE);
      String encodedContents = map.get("contents").toString();
      Zip.unzip(encodedContents, new File("/Users/diegomolina/Downloads"));
    }

    driver.quit();
  }
}

[krmahadevan]

@diemol - Thanks for catching that bug! Fixed the review comments, rebased off of trunk and force pushed changes back to this branch. Please check

[diemol]

Thank you, @krmahadevan!

[diemol]

@krmahadevan would be great if you can help us with the docs for this new feature 💪

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[krmahadevan]

@krmahadevan would be great if you can help us with the docs for this new feature 💪

@diemol - I have raised a PR for this

PR: SeleniumHQ/seleniumhq.github.io#1262

Doc links:

• https://deploy-preview-1262--jovial-austin-42fe02.netlify.app/documentation/grid/configuration/cli_options/#specifying-path-from-where-downloaded-files-can-be-retrieved
• https://deploy-preview-1262--jovial-austin-42fe02.netlify.app/documentation/grid/configuration/toml_options/#retrieving-downloaded-files-from-node

[iammac2]

Downloaded 4.8.0 and verified this feature. Working well👍🏼. Will the API also include an endpoint to clean up the downloaded files?

[titusfortner]

imo it shouldn't have its own endpoint, it should be part of the cleanup process of ending the session. Once the session is finished, the files can't be accessed again.

[iammac2]

Thank you @titusfortner and agree re session clean up. Have tried verifying this behaviour with a local Grid 4.8.0 config to verify what happens across sessions:

• Grid with one Node is running locally with the new --downloads-path CLI Node option is set i.e. java -jar selenium-server-4.8.0.jar node --downloads-path C:\tmp
• Chrome pref download.default_directory also set to same path when RemoteWebDriver initialized.
• Run a test on Node (noting session ID in Node log) that clicks to download a file from UI at the remote end and then retrieves the file via the new GET request.
• The test passes first time through with the file downloaded to Node download directory and GET response contains correct filename and contents.
• Wait for Node debug to state Stopping session
• Run the test again (i.e. new session ID in Node log) but WITHOUT the step that clicks to download the file and ONLY sends the GET request. (Would expect this scenario to fail when endpoint hit due to session cleardown).
• In this scenario the GET response still contains the same filename and contents.
• Physically checking the Node download path after Node log states Stopping session, confirms that the file is still present in the directory.

[titusfortner]

@iammac2 I just split out the work i'd like to see us do to get downloads working. The way I want to see us support things will look like this in the bindings: #11657

[iammac2]

I see, thank you for the extra info. Will keep a watch in this further development.
So existing implementation of GET /session/:sessionId:/se/files endpoint will need to change as download path will need to be generated and not set by user e.g --downloads-path and browser prefs download.default_directory? Grid Node will then hold internal reference by uuid to the download directories so it can clean them up (ref the DELETE)

[titusfortner]

That's about right. We'll see what trouble we run into (if any) in the implementation.

[vijaysp77]

@krmahadevan Can I use the preference settings for this download path inside the code (download.default_directory) instead of passing the download path while setting-up node? Because we are using docker container provisioned by aws ECS? Thanks

[krmahadevan]

@vijaysp77 This feature is going through a revamp.

The functionality will change once my PR #11702 gets merged.

So I would suggest that you follow the above PR and wait for this functionality to get streamlined.

[patilsha]

@krmahadevan I saw your PR 1702 has been merged.
Where can I find the latest Selenium Hub Image to see these changes in Selenium Grid.
Thanks for providing a solution to most awaited feature.

[diemol]

@patilsha it was merged yesterday, it will be part of the next release.

[xZero707]

I'm assuming this is not yet available in Selenium docker images (4.8.1)?

Chrome node

GET /session/:sessionId:/se/files
curl -X GET "http://172.17.53.2:4444/session/98473aa8661fc256bf7c0de98861005e/se/file?filename=100MB.bin"
= Unknown command

Firefox node

GET /session/:sessionId:/se/files
curl -X GET "http://172.17.53.2:4444/session/98473aa8661fc256bf7c0de98861005e/se/file?filename=100MB.bin"
= Unsupported method - expect POST for file upload

I also tried running this using PHP driver:

$file = $driver->executeCustomCommand('/session/:sessionId/se/file?filename=100MB.bin');

Result:

Facebook\WebDriver\Exception\UnknownCommandException : unknown command: unknown command: session/41d161b598c0fd3f85a4e09b1f5f5a1f/se/file?filename=100MB.bin`

I'm in dark. This feature is documented, yet cannot be used. Am I doing something wrong?

I defined --downloads-path /usr/downloads using env SE_OPTS and nothing. Chrome is still downloading to /home/seluser/Downloads and ignoring the path I defined.

Version: Selenium 4.8.1 (revision 8ebccac)

[krmahadevan]

@xZero707 - For this to be available in the docker selenium project, I think this should be released first. The feature has gone through some revamp. So please create an issue on the docker-selenium project that tracks this requirement and you can also cross link this issue there.
The documentation on this also needs to be changed. I haven't found time to get to this yet. I should hopefully be able to wrap the documentation before this weekend.

[xZero707]

@krmahadevan Thank you for clarification. It makes sense. I will do as you suggested.

[naveenmadd]

I checked out this PR to run a complete test but it did not work. Probably because the get method for downloadsPath configuration is not being used.

The client code I used to perform a test was:

package org.example;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.io.Zip;
import org.openqa.selenium.json.Json;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.http.HttpClient;
import org.openqa.selenium.remote.http.HttpMethod;
import org.openqa.selenium.remote.http.HttpRequest;
import org.openqa.selenium.remote.http.HttpResponse;

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Map;

import static org.openqa.selenium.remote.http.Contents.string;

public class Main {

  public static void main(String[] args) throws InterruptedException, IOException {
    FirefoxOptions options = new FirefoxOptions();
    options.addPreference("browser.download.manager.showWhenStarting", false);
    options.addPreference("browser.helperApps.neverAsk.saveToDisk", "images/jpeg, application/pdf, application/octet-stream");
    options.addPreference("pdfjs.disabled", true);

    URL gridUrl = new URL("http://localhost:4444");
    RemoteWebDriver driver =  new RemoteWebDriver(gridUrl, options);
    driver.get("http://the-internet.herokuapp.com/download");
    WebElement element = driver.findElement(By.cssSelector(".example a"));
    element.click();

    Thread.sleep(10 * 1000);

    HttpRequest request = new HttpRequest(
      HttpMethod.GET,
      String.format("/session/%s/se/file", driver.getSessionId()));
    request.addQueryParameter("filename", "my_appointments-1.pdf");
    try (HttpClient client = HttpClient.Factory.createDefault().createClient(gridUrl)) {
      HttpResponse response = client.execute(request);
      Map<String, Object> map = new Json().toType(string(response), Json.MAP_TYPE);
      String encodedContents = map.get("contents").toString();
      Zip.unzip(encodedContents, new File("/Users/diegomolina/Downloads"));
    }

    driver.quit();
  }
}

@diemol Can you pls help me here, I'm trying to test File Downloads on Jenkins Node but
Getting [ERROR] COMPILATION ERROR : at Map<String, Object> map = new Json().toType(string(response), Json.MAP_TYPE); as
no suitable method found for string(org.openqa.selenium.remote.http.HttpResponse)
[ERROR] method org.openqa.selenium.remote.http.Contents.string(java.io.File) is not applicable
[ERROR] (argument mismatch; org.openqa.selenium.remote.http.HttpResponse cannot be converted to java.io.File)
[ERROR] method org.openqa.selenium.remote.http.Contents.string(org.openqa.selenium.remote.http.HttpMessage) is not applicable [ERROR] (argument mismatch; org.openqa.selenium.remote.http.HttpResponse cannot be converted to org.openqa.selenium.remote.http.HttpMessage)

[krmahadevan]

@naveenmadd - If you wanted to build/experiment something on top of this changes, I would suggest that you do it from trunk. The feature has undergone changes after it was merged.

[naveenmadd]

@naveenmadd - If you wanted to build/experiment something on top of this changes, I would suggest that you do it from trunk. The feature has undergone changes after it was merged.

Even I verified selenium-trunk repository, I see the same implementation [ package org.openqa.selenium.grid.node > class NodeTest > @test @DisplayName("DownloadsTestCase") Map<String, Object> map = new Json().toType(string(response), Json.MAP_TYPE);]
Don't know how it's accepting in that test and not for mine? suggest if you have any insights or try from your end too once..

Thanks @krmahadevan @diemol

[diemol]

This is the test case https://github.com/SeleniumHQ/selenium/blob/trunk/java/test/org/openqa/selenium/grid/router/RemoteWebDriverDownloadTest.java#L134