Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add workflow for updating duckdb refs #24

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

add workflow for updating duckdb refs #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/scripts/duckdb-ref-update.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash

set -euo pipefail

fetch_latest_ref() {
local REPO=$1
local REF_TYPE=$2

if [ "$REF_TYPE" = "release" ]; then
local TAG=$(gh release -R "$REPO" list --json tagName,isLatest --jq '.[] | select(.isLatest).tagName')
local LATEST_REF=$(gh api repos/"$REPO"/git/ref/tags/"$TAG" --jq '.object.sha')
else
local LATEST_REF=$(gh api repos/"$REPO"/git/refs/heads/main --jq '.object.sha')
fi

echo "$LATEST_REF"
}

update_makefile() {
local REF_VAR_NAME="$1"
local LATEST_REF="$2"

if grep -q "^.*${REF_VAR_NAME} *= *${LATEST_REF}" Makefile; then
return
fi

if [[ "$(uname)" == "Darwin" ]]; then
sed -i '' "s|^\\(.*${REF_VAR_NAME} *\\)=.*|\\1=${LATEST_REF}|" Makefile
else
sed -i "s|^\\(.*${REF_VAR_NAME} *\\)=.*|\\1=${LATEST_REF}|" Makefile
fi

echo "\\n-Updated ${REF_VAR_NAME} to ${LATEST_REF}"
}
53 changes: 53 additions & 0 deletions .github/workflows/duckdb-ref-update.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: duckdb-ref-update

on:
workflow_dispatch:
schedule:
- cron: '0 4 * * 6' # Every Saturday at 4 AM US time (America/New_York)

concurrency:
group: "${{ github.workflow }}-${{ github.run_started_at }}"
cancel-in-progress: true

jobs:
update-duckdb-refs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Fetch and Update All Refs
id: fetch-update-refs
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be good to skip the update if the duckdb ref hasn't updated. I don't think we want to update the extensions if we aren't updating dudckdb

Copy link
Contributor Author

@srikrishnak srikrishnak Feb 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that is better approach as it saves cycles but just to confirm, I think we might lose one corner case.

  1. lets sayd duckdb updated to a latest release A, and we found a stable build with all extensions and merged the commit.
  2. later, in one of the extensions, like substrait, we did a commit B on the same release A.

We might miss a chance to pull that commit B. @rymurr Thoughts?

source ./.github/scripts/duckdb-ref-update.sh
COMMIT_DETAILS=""

repos='[
{"repository": "duckdb/duckdb", "ref_var": "DUCKDB_REF", "ref_type": "release"},
{"repository": "duckdb/duckdb-aws", "ref_var": "DUCKDB_AWS_REF", "ref_type": "tip"},
{"repository": "duckdb/duckdb-httpfs", "ref_var": "DUCKDB_HTTPFS_REF", "ref_type": "tip"},
{"repository": "substrait-io/duckdb-substrait-extension", "ref_var": "DUCKDB_SUBSTRAIT_REF", "ref_type": "tip"},
{"repository": "duckdb/duckdb-iceberg", "ref_var": "DUCKDB_ICEBERG_REF", "ref_type": "tip"}
]'

echo "$repos" | jq -c '.[]' | while read -r repo; do
REPO=$(echo "$repo" | jq -r '.repository')
REF_VAR=$(echo "$repo" | jq -r '.ref_var')
REF_TYPE=$(echo "$repo" | jq -r '.ref_type')

echo "Processing $REPO with $REF_VAR and $REF_TYPE"
LATEST_REF=$(fetch_latest_ref "$REPO" "$REF_TYPE")
echo "Latest ref: $LATEST_REF"
UPDATE_OUTPUT=$(update_makefile "$REF_VAR" "$LATEST_REF")
if [[ -n "$UPDATE_OUTPUT" ]]; then
COMMIT_DETAILS+=$'\n'"$UPDATE_OUTPUT"
fi
done

echo "COMMIT_DETAILS=$COMMIT_DETAILS" >> "$GITHUB_ENV"
- name: Commit and Push Updates
uses: stefanzweifel/git-auto-commit-action@v5
with:
Copy link
Contributor Author

@srikrishnak srikrishnak Feb 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rymurr I couldn't test this step as it doesn't find the branch sd9320 on sundeck.io/duckdb-go-extensions. Wondering if it is ok to create a branch.

Or I could test this once it merges to main

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the diff as well, only part I didn't check is the final commit that gets raised.

I put a workflow_dispatch so i can trigger and see how the commit looks like once this is merged.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is goign to commit to main if run on main, I think rather than commit you want to create a PR

commit_message: |
Update DuckDB references:${{ env.COMMIT_DETAILS }}
7 changes: 6 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
DUCKDB_REPO=https://github.com/duckdb/duckdb.git
DUCKDB_REF=5f5512b827df6397afd31daedb4bbdee76520019

export DUCKDB_AWS_REF=b3050f35c6e99fa35465230493eeab14a78a0409
export DUCKDB_HTTPFS_REF=85ac4667bcb0d868199e156f8dd918b0278db7b9
export DUCKDB_SUBSTRAIT_REF=a1b341cc3df16c55535c12acce375040ffe50347
export DUCKDB_ICEBERG_REF=3060b30309d82f1059c928de7280286fcf800545

CFLAGS = -O3
CXXFLAGS = -O3
CC = ""
Expand Down Expand Up @@ -51,7 +56,7 @@ duckdb:
rm -rf duckdb
git clone --depth 1 $(DUCKDB_REPO) duckdb
cd duckdb && git fetch --depth 1 origin $(DUCKDB_REF) && git checkout $(DUCKDB_REF)
cp extension_config_local.cmake duckdb/extension/extension_config.cmake
envsubst < extension_config_local.cmake > duckdb/extension/extension_config.cmake

.PHONY: deps.header
deps.header: duckdb
Expand Down
8 changes: 4 additions & 4 deletions extension_config_local.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@ duckdb_extension_load(parquet)
duckdb_extension_load(icu)
duckdb_extension_load(aws
GIT_URL https://github.com/duckdb/duckdb-aws
GIT_TAG b3050f35c6e99fa35465230493eeab14a78a0409
GIT_TAG ${DUCKDB_AWS_REF}
)
duckdb_extension_load(httpfs
GIT_URL https://github.com/duckdb/duckdb-httpfs
GIT_TAG 85ac4667bcb0d868199e156f8dd918b0278db7b9
GIT_TAG ${DUCKDB_HTTPFS_REF}
INCLUDE_DIR extension/httpfs/include
)
duckdb_extension_load(tpch)
duckdb_extension_load(tpcds)
duckdb_extension_load(substrait
GIT_URL https://github.com/substrait-io/duckdb-substrait-extension
GIT_TAG a1b341cc3df16c55535c12acce375040ffe50347
GIT_TAG ${DUCKDB_SUBSTRAIT_REF}
)
duckdb_extension_load(iceberg
GIT_URL https://github.com/duckdb/duckdb-iceberg
GIT_TAG 3060b30309d82f1059c928de7280286fcf800545
GIT_TAG ${DUCKDB_ICEBERG_REF}
)
Loading