Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prompter interface #25

Merged
merged 10 commits into from
Jan 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .config/README.md.template
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ runbook run runbook-name.ipynb

# Background

### What is a Runbook?
## What is a Runbook?
A runbook is an executable document that combines:
- Clear markdown documentation
- Runnable code blocks
Expand All @@ -40,13 +40,13 @@ A runbook is an executable document that combines:

It's ideal for operations like encoding your Disaster Recovery Operations, spinning up a new cluster, or restoring from snapshots.

### When Should You Use This?
## When Should You Use This?
- ✅ When you need **semi-automated tools** with audit trails and safety checks
- ✅ When you want **rapid iteration** on operational procedures with built-in rollback steps
- ✅ When you need something more powerful than shell scripts but don't want to build a full application
- ✅ When you want to make complex operations both **safe and repeatable**

### Runbook Best Practices
## Runbook Best Practices
1. Structure your runbooks with:
- Clear purpose and summary
- Step-by-step descriptions
Expand All @@ -72,7 +72,7 @@ It's ideal for operations like encoding your Disaster Recovery Operations, spinn
1. Depending on auditing needs, you can either commit the "runs" folder to your repo or only keep the "binder" folder committed.
1. In case of strict auditing needs, we recommend you add auditing of commands in the local SDK as well as in your cloud provider

## Installation
# Installation

We recommend using [uv](https://docs.astral.sh/uv/) for installing runbook as a cli tool. If you already use pipx, you can use that instead.

Expand All @@ -86,7 +86,7 @@ Or pin to a version
uv tool install git+https://github.com/zph/runbook.git@$RUNBOOK_VERSION
```

## CLI
# CLI

```sh
$RUNBOOK_HELP
Expand Down Expand Up @@ -136,7 +136,7 @@ For development we use the following tools:
- [hermit](https://hermit.dev/) to manage developement tool dependencies (see .hermit/bin)
- [uv](https://docs.astral.sh/uv/) python package manager and cli runner (see pyproject.toml)

Necessary deps can be seen in [pyproject.toml](pyproject.toml) and .hermit/bin
Necessary deps can be seen in pyproject.toml and .hermit/bin

Use .hermit/bin/activate-hermit to activate the environment.

Expand Down
40 changes: 40 additions & 0 deletions .hermit/bin/publish-docs
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/bash

# Stop on errors
set -eou pipefail
set -x

# Variables
DOCS_DIR="site"
BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
TEMP_DIR=$(mktemp -d)
TEMP_GIT_DIR=$(mktemp -d)
ORIGINAL_GIT_DIR=$(git rev-parse --show-toplevel)

# Check if git working directory is clean
# if [ -n "$(git status --porcelain)" ]; then
# echo "Error: Git working directory is not clean. Please commit or stash changes first."
# exit 1
# fi

just docs
# Copy the docs to the temporary directory
cp -r "$DOCS_DIR"/* "$TEMP_DIR/"

(
cd "$TEMP_GIT_DIR"
git clone "$ORIGINAL_GIT_DIR" .
git checkout gh-pages

rm -rf ./*
cp -r "$TEMP_DIR"/* .
git add -A
git commit -m "Publish documentation [$(date)]"
# Push to local original repo
git push origin gh-pages
)

# Push to remote
git push origin gh-pages

echo "Documentation published to gh-pages branch."
5 changes: 3 additions & 2 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,10 @@ template-update:

readme:
.config/templating.sh
cp README.md docs/

docs:
uvx --with mkdocs-click --with . mkdocs serve
uvx --with sphinx-click --with myst_parser --with . --from sphinx sphinx-build -b html docs/ site

docs-release:
uvx --with mkdocs-click --with . mkdocs gh-deploy
bash .hermit/bin/publish-docs
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ runbook run runbook-name.ipynb

# Background

### What is a Runbook?
## What is a Runbook?
A runbook is an executable document that combines:
- Clear markdown documentation
- Runnable code blocks
Expand All @@ -40,13 +40,13 @@ A runbook is an executable document that combines:

It's ideal for operations like encoding your Disaster Recovery Operations, spinning up a new cluster, or restoring from snapshots.

### When Should You Use This?
## When Should You Use This?
- ✅ When you need **semi-automated tools** with audit trails and safety checks
- ✅ When you want **rapid iteration** on operational procedures with built-in rollback steps
- ✅ When you need something more powerful than shell scripts but don't want to build a full application
- ✅ When you want to make complex operations both **safe and repeatable**

### Runbook Best Practices
## Runbook Best Practices
1. Structure your runbooks with:
- Clear purpose and summary
- Step-by-step descriptions
Expand All @@ -72,7 +72,7 @@ It's ideal for operations like encoding your Disaster Recovery Operations, spinn
1. Depending on auditing needs, you can either commit the "runs" folder to your repo or only keep the "binder" folder committed.
1. In case of strict auditing needs, we recommend you add auditing of commands in the local SDK as well as in your cloud provider

## Installation
# Installation

We recommend using [uv](https://docs.astral.sh/uv/) for installing runbook as a cli tool. If you already use pipx, you can use that instead.

Expand All @@ -86,7 +86,7 @@ Or pin to a version
uv tool install git+https://github.com/zph/[email protected]
```

## CLI
# CLI

```sh
Usage: runbook [OPTIONS] COMMAND [ARGS]...
Expand All @@ -98,16 +98,16 @@ Options:
--help Show this message and exit.

Commands:
check Check language validity and formatting of a notebook.
convert Convert an existing runbook to different format
create Create a new runbook from [template]
diff Diff two notebooks
check Check the language validity and formatting of a runbook.
convert Convert a runbook between different formats
create Create a new runbook from a template
diff Compare two runbooks and show their differences
edit Edit an existing runbook
init Initialize a folder as a runbook repository
list list runbooks
list List runbooks
plan Prepares the runbook for execution by injecting parameters.
review [Unimplemented] Entrypoint for reviewing runbook
run Run a notebook
run Run a runbook
show Show runbook parameters and metadata
version Display version information about runbook
```
Expand Down Expand Up @@ -156,7 +156,7 @@ For development we use the following tools:
- [hermit](https://hermit.dev/) to manage developement tool dependencies (see .hermit/bin)
- [uv](https://docs.astral.sh/uv/) python package manager and cli runner (see pyproject.toml)

Necessary deps can be seen in [pyproject.toml](pyproject.toml) and .hermit/bin
Necessary deps can be seen in pyproject.toml and .hermit/bin

Use .hermit/bin/activate-hermit to activate the environment.

Expand Down
10 changes: 3 additions & 7 deletions NOTES.md → docs/NOTES.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
# TODO
# Roadmap and Notes

## Triage
- [x] Read `runbook plan ...` support reading params from file
- [ ] Re-export confirm(style) and gather from sh lib
- [ ] Make sure we can run directly fully through cli
- [ ] If no argument included for RUNBOOK TITLE in edit/plan/run then prompt with options
- [x] Include field 'embeds' in the metadata to help with referencing them
- [ ] --- Add helper for referencing the embeds?
- [ ] Use execute and upload output to S3 for non-interactive: https://github.com/nteract/papermill/tree/main?tab=readme-ov-file#execute-via-cli

## P0
- [ ] Add linter for runbooks (using `runbook lint`) that checks for presence of title, desc, rollback, cleanup, etc
- [ ] Add `runbook init` suggestion or automation to add `export RUNBOOK_WORKING_DIR=...` to shell initializations
- [ ] Install pre-commit.yml or git integration during `init` for secure linting and talisman
- [x] Setup versioning and bumper (using versioner from npm ecosystem @release-it and @release-it/bumper)
Expand All @@ -34,10 +33,7 @@
- [x] Should I follow the tf convention of `plan | apply`
- [x] Setup decorator to embed dry_run into shell command
- [ ] (won't do yet) Allow for executing cell by cell from commandline in a repl?
- [ ] Running cell by cell: https://github.com/odewahn/ipynb-examples/blob/master/Importing%20Notebooks.ipynb
- [ ] figure out how to store and replay individual cells
- [ ] ~~Textualize gui for tui?~~
- [ ] ~~is euphorie's tui for notebooks helpful?~~
- [ ] Running cell by cell: https://github.com/odewahn/ipynb-examples/blob/master/Importing%20Notebooks.ipynb
- [ ] Build auditability through a custom runner interface
-- Or a custom kernel wrapper?
-- https://ipython.readthedocs.io/en/stable/config/options/kernel.html
Expand Down
165 changes: 165 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Runbook

## Summary

Runbook is a powerful CLI tool that transforms your operational procedures into interactive, executable notebooks. It combines the best of documentation and automation by letting you create dynamic runbooks using Markdown, Deno, or Python.

Think of it as "infrastructure as code" meets "documentation as code" - perfect for DevOps teams who want both flexibility and reliability.

**At work, it empowered us to move 300 Mysql Clusters to TiDB with a small team [recording](https://www.youtube.com/watch?app=desktop&v=-_JoqZthrI8) over the course of 18 months.**

# Quick Start

```sh
uv tool install git+https://github.com/zph/[email protected]

# Initialize a new runbook project in a repo of your choosing
runbook init

# Create a new runbook
runbook create -l deno runbook-name.ipynb

# Edit the runbook
runbook edit runbook-name.ipynb

# Plan the runbook
runbook plan runbook-name.ipynb --embed file.json --parameters '{"arg": 1, "foo": "baz"}'

# Run the runbook
runbook run runbook-name.ipynb
```

# Background

## What is a Runbook?
A runbook is an executable document that combines:
- Clear markdown documentation
- Runnable code blocks
- Parameterized inputs for reusability
- Built-in safety checks

It's ideal for operations like encoding your Disaster Recovery Operations, spinning up a new cluster, or restoring from snapshots.

## When Should You Use This?
- ✅ When you need **semi-automated tools** with audit trails and safety checks
- ✅ When you want **rapid iteration** on operational procedures with built-in rollback steps
- ✅ When you need something more powerful than shell scripts but don't want to build a full application
- ✅ When you want to make complex operations both **safe and repeatable**

## Runbook Best Practices
1. Structure your runbooks with:
- Clear purpose and summary
- Step-by-step descriptions
- Warning signs and precautions
- Verification steps
- Execution steps in logical order
- Rollback and cleanup steps
2. Keep read-only operations flexible
3. Require explicit confirmation for destructive actions using the `confirm` flag
4. Include pre-flight checks before any system modifications
5. For critical operations, use pair execution:
- One person to run the procedure
- Another to verify and validate safety checks

## Workflow

1. Initialize a new folder project with `runbook init...`
1. Create a new runbook with `runbook create -l deno runbook-name.ipynb`
1. Edit the runbook with `runbook edit runbook-name.ipynb` (or using editor of choice) and add your title, description, steps
1. For complex runbooks, offload the coding details into an SDK that you build beside the runbooks that can be reused across multiple runbooks
1. Plan that runbook for a specific run `runbook plan runbook-name.ipynb --embed file.json --parameters '{"arg": 1, "foo": "baz"}'
1. Run the instance of a runbook with either `runbook run runbook-name.ipynb` or use VSCode to run it `code runbooks/runs/runbook-name.ipynb`
1. Depending on auditing needs, you can either commit the "runs" folder to your repo or only keep the "binder" folder committed.
1. In case of strict auditing needs, we recommend you add auditing of commands in the local SDK as well as in your cloud provider

# Installation

We recommend using [uv](https://docs.astral.sh/uv/) for installing runbook as a cli tool. If you already use pipx, you can use that instead.

```sh
uv tool install git+https://github.com/zph/runbook.git
```

Or pin to a version

```sh
uv tool install git+https://github.com/zph/[email protected]
```

# CLI

```sh
Usage: runbook [OPTIONS] COMMAND [ARGS]...

Options:
--cwd PATH Directory for operations (normally at root above runbooks, ie
../.runbook.yaml) and can be set with RUNBOOK_WORKING_DIR or
WORKING_DIR environment variables
--help Show this message and exit.

Commands:
check Check the language validity and formatting of a runbook.
convert Convert a runbook between different formats
create Create a new runbook from a template
diff Compare two runbooks and show their differences
edit Edit an existing runbook
init Initialize a folder as a runbook repository
list List runbooks
plan Prepares the runbook for execution by injecting parameters.
review [Unimplemented] Entrypoint for reviewing runbook
run Run a runbook
show Show runbook parameters and metadata
version Display version information about runbook
```

Shell completion is included via click library and enabled as follows [link](https://click.palletsprojects.com/en/8.1.x/shell-completion/#enabling-completion)

```
# Bash
# Add this to ~/.bashrc:
eval "$(_RUNBOOK_COMPLETE=bash_source runbook)"

# Zsh
# Add this to ~/.zshrc:
eval "$(_RUNBOOK_COMPLETE=zsh_source runbook)"

# Fish
# Add this to ~/.config/fish/completions/foo-bar.fish:
_RUNBOOK_COMPLETE=fish_source runbook | source
```

For advanced completion setup see [docs](https://click.palletsprojects.com/en/8.1.x/shell-completion/#enabling-completion)

# Principles

- Prefer deno for better package management and developer ergonomics with typing
- But allow for other kernels (python) as secondary option, via compatible libraries
- Make `runbook` batteries included for interfacing with shell commands and common runbook
operations

# Caveats

1. Running notebook in VScode does not set the timings necessary in notebook for being auditable and exported later
1. Recommendation: if auditable runs are needed, use jupyter via browser `runbook run TITLE`
1. Notebooks have different structured ids per cell depending on run environment
1. Recommendation: if requiring consistency, write your own pre-processor to standardize on an id format
1. Built-in shell package requires a shell environment and is only expected to run on Linux or Mac not Windows.
1. Recommendation: Windows support is out of scope for now but we'll review PRs

## Deno / Typescript
1. Parameter cells must use `let` declarations to allow for param overriding
- `var` or `let` work in Deno notebooks but only `let` works if using `runbook convert a.ipynb a.ts` and running the ts version

# Developing runbook cli

For development we use the following tools:
- [hermit](https://hermit.dev/) to manage developement tool dependencies (see .hermit/bin)
- [uv](https://docs.astral.sh/uv/) python package manager and cli runner (see pyproject.toml)

Necessary deps can be seen in pyproject.toml and .hermit/bin

Use .hermit/bin/activate-hermit to activate the environment.

# Readme Changes

README.md is generated from .config/README.md.template and should be updated there.
Empty file added docs/_static/.gitkeep
Empty file.
Empty file added docs/_templates/.gitkeep
Empty file.
15 changes: 15 additions & 0 deletions docs/cli.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
.. Runbook documentation master file, created by
sphinx-quickstart on Mon Jan 20 11:01:47 2025.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Runbook CLI
=====================

.. toctree::
:maxdepth: 2
:caption: Contents:

.. click:: runbook.cli:cli
:prog: runbook
:show-nested:
Loading
Loading