[rfc] Document using backfill API from schedules/sensors

[deepyaman]

Summary & Motivation

Background

@smackesey has interacted with several users on the following scenario:

• There is some computation (run or backfill) X that should only be launched under certain conditions
• The conditions under which X should be launched are non-trivial to compute
• It is desired to manually trigger conditional launch of X-- that is, on clicking a button, the conditions under which X should be launched are computed and X is launched if those conditions are satisfied.

Problem Statement

We do not have a good solution to the above scenario.

• Launch job Y that conditionally launches X
  • If X is a job, execute_job will work.
  • If X is a backfill, then there is no good public API to launch it. User must use the private GQL API.
• Manually trigger sensor that conditionally launches X
  • The way you manually trigger a sensor is by pressing the "Test Sensor" button, which is non-ideal and kinda buried.
  • You then have to pass each generated RunRequest through to the launchpad and manually launch.
    • This works OK if X is a job.
    • This is unworkable if X is a backfill, because you have to manually launch each RunRequest

So if X is a backfill, basically your only option is to use a job that then uses the private GQL API.

Objective

The purpose of this enhancement is to improve user experience. This will allow users to not have to resort to workarounds to perform a manual trigger a conditional launch of X.

The manifestation of this is to expose a public Python API for creating and launching asset backfills.

Specifically, they should be able to launch backfills from schedules and sensors.

[deepyaman]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [rfc] Document using backfill API from schedules/sensors #27260 👈 (View in Graphite)
• Implement Python API for launching asset backfills #26731 : 1 other dependent PR (#27060 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Deploy preview for dagster-docs-beta ready!

Preview available at https://dagster-docs-beta-rigwkogbi-elementl.vercel.app

Direct link to changed pages:

• https://dagster-docs-beta-rigwkogbi-elementl.vercel.app/guides/build/partitions-and-backfills/backfilling-data