How to create a custom device pool on AWS Device Farm
AWS Device Farm is the AWS cloud service we are using to provide Android and iOS devices for ExecuTorch performance benchmark CI job. Although AWS provides a large list of different devices to choose from https://aws.amazon.com/device-farm/device-list, we need to define specific sets of devices we want to run the benchmark on instead of selecting them randomly. This wiki gives a rough instruction on how to setup these custom sets called device pools on AWS.
At the moment, ExecuTorch is maintaining two device pools:
- Android S22 with ARN
arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/e59f866a-30aa-4aa1-87b7-4510e5820dfa, which includes only Samsung S22 with different Android OS version (12, 13) - Android S24 with ARN
arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/98f8788c-2e25-4a3c-8bb2-0d1e8897c0db. It includes Samsung Android S25 with Android OS version 14. - iOS 15 with ARN
arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/3b5acd2e-92e2-4778-b651-7726bafe129d. It includes iPhone 15, 15 Plus, 15 Pro, and 15 Pro Max. - Google Pixel 8
arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/d65096ab-900b-4521-be8b-a3619b69236a
The pool's ARN is then used in the benchmark job to decides which devices to use there, i.e. https://github.com/pytorch/executorch/blob/main/.github/workflows/android.yml
To do this, you will need to have access to PyTorch Dev Infra AWS account (fbossci) where the infra is setup. This permission is not granted lightly and is reviewed on a case-by-case basis. So, please consult with the team and reach out before you want to request it. On the other hand, assuming that you already have access there, here are the step to create a new device pool or to update an existing one.
-
bunnylol cloud fbosscito login to AWS console - Go to AWS Device Farm service on the console. You will see ExecuTorch as a project there, note down the project ARN
arn:aws:devicefarm:us-west-2:308535385114:project:02a2cf0f-6d9b-45ee-ba1a-a086587469e6
-
Go to the project settings and see the list of custom device pools there.
-
The two device pools above are listed there. Select and choose
Editto edit them or chooseCreate device poolto create a new one.
-
There are 2 type of device pools: a static device pool and a rule-base dynamic one.
This type of device pool is simple where you just choose the devices you want to include. As an example, Android S2x is a static device pool where some S2x devices are manually selected. The more devices you include, the more coverage we have, but also the more costly (billing by the minutes of usage). On another note, these devices are used in parallel, so a job scheduled here will take as long to finish as the slowest device (given that they are all highly available, i.e. popular devices).
Instead of selecting devices manually, a device pool can be setup using some simple rules with drop-down list and string matching (no regex). The iOS 15 pool sets the Platform to be iOS (there are only 2 platforms on AWS Device Farm, Android and iOS) and Model to be iPhone 15. As another example, if the Model is set to iPhone 1 instead, the pool will select all available devices from iPhone 11 to iPhone 15. In this case, it's important to also set the maximum number of parallel devices we want to use in the job to avoid a runaway bill. If there are more selected devices in the pool than the max value, up to the max number of devices will be selected at random.
After creating a new device pool, we need to get its ARN so that we can use it in our CI job. The command to list all ExecuTorch device pools is aws devicefarm list-device-pools --arn arn:aws:devicefarm:us-west-2:308535385114:project:02a2cf0f-6d9b-45ee-ba1a-a086587469e6 --region us-west-2 and you can find the new pool ARN from the list.