Add CI workflow that runs the full SDG pipeline

[russellb]

This PR introduces a new e2e CI workflow that exercises the full SDG
pipeline. Some important details:

• This job runs on a dynamic github runner spawned on AWS

• The instance type has 4x NVIDIA A10G GPUs (96 GB total)

• a 4-bit quantized version of Mixtral-8x7b-instruct is used

• Training is currently skipped while we work out getting the new
  training library functional in CI

• The model is served via llama-cpp, as vllm is not yet functional in
  CI

• The job does not run automatically. It must be launched manually via
  the GitHub UI. When you launch it against a given PR, it will
  automatically add comments to that PR to make it easier to follow
  the progress and results. For more info on launching this workflow
  manually, see the instructions for similar workflows in the
  instructlab/instructlab repository:

  https://github.com/instructlab/instructlab/blob/main/docs/ci.md

Signed-off-by: Russell Bryant [email protected]

[russellb]

Note that this is a very low-risk PR since it does not run automatically anywhere, so the worst case is that when it's run manually, it fails. There may be some final tweaks needed, but it has to merge here before I can test this final version crafted for this repo. I have a working variation of this in a personal test repository, though.

[russellb]

This depends on instructlab/instructlab#1625, so the workflow switches to the PR branch. That means we're not testing against the latest CLI repo changes until that PR merges and we remove that line from this workflow

[russellb]

This depends on instructlab/instructlab#1625, so the workflow switches to the PR branch. That means we're not testing against the latest CLI repo changes until that PR merges and we remove that line from this workflow

That PR merged and the hack has been removed from this workflow. It will now always test against main of instructlab

[russellb]

@oindrillac review is definitely welcome! Let me know if you have any questions or feedback. I'm going to merge this so I can start testing it in this repo. It shouldn't affect anything else.