Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cube v6 support #3

Draft
wants to merge 45 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
64f15bd
Add celery task signature
Aug 28, 2024
ee8476a
Document proposed changes to oxidicom
Aug 28, 2024
5e63b89
WIP
Aug 28, 2024
1a3d818
Working with CUBE version 6
jennydaman Sep 7, 2024
061edde
Add Swatinem/rust-cache
jennydaman Sep 7, 2024
ffefcb9
Add CARGO_TERM_COLOR=always
jennydaman Sep 7, 2024
e078b49
Assert files against snapshot
jennydaman Sep 7, 2024
c274d4a
Add validation of instance count to get_data.sh
jennydaman Sep 7, 2024
5ec03a2
Add codecov
jennydaman Sep 7, 2024
e2ce6bb
Replace parallel with xargs
jennydaman Sep 7, 2024
9c08bde
Send LONK to NATS
jennydaman Sep 8, 2024
da25c3a
SubjectLimiter
jennydaman Sep 8, 2024
4d2f238
Rework SubjectLimiter::lock to return a RAII
jennydaman Sep 8, 2024
e1ee1b6
Add cross-compilation and remove Dockerfile
jennydaman Sep 8, 2024
b44a0bf
Switch to houseabsolute/actions-rust-cross
jennydaman Sep 8, 2024
86eea0a
Remove FIXMEs
jennydaman Sep 8, 2024
3c8b424
chmod binary
jennydaman Sep 8, 2024
33838f2
Test NATS messages
jennydaman Sep 9, 2024
961b5eb
Change order of assertions
jennydaman Sep 9, 2024
430f98c
Print last 3 payloads for debugging
jennydaman Sep 10, 2024
17b510c
Rewrite test_forget_waits_until_unlocked
jennydaman Sep 10, 2024
3c6e7c5
Add a GHA job to rerun tests
jennydaman Sep 10, 2024
9685454
Wait for a longer time
jennydaman Sep 10, 2024
42a0e5f
Sleep for a really long time in GHA
jennydaman Sep 10, 2024
e6bb83d
Sanitize pacs_name too
jennydaman Sep 10, 2024
cf00b42
Replace env! with option_env!
jennydaman Sep 10, 2024
a4716ed
Count failures
jennydaman Sep 10, 2024
4033819
Rework how the nats_subscriber_loop ends
jennydaman Sep 10, 2024
ff73cc7
Change order of assertions
jennydaman Sep 10, 2024
640cc69
Use tracing_subscriber::EnvFilter
jennydaman Sep 10, 2024
1ea90fe
cargo fmt
jennydaman Sep 10, 2024
f970fdd
Delete reset.sh
jennydaman Sep 10, 2024
bdf2e6d
Add a bunch of tracing statements
jennydaman Sep 10, 2024
05cd80d
cargo update
jennydaman Sep 11, 2024
92dab89
Compute checksums for debugging
jennydaman Sep 11, 2024
bc602b6
Remove broken "Compile test binary" step
jennydaman Sep 11, 2024
e27eaf9
Fix ci.yml
jennydaman Sep 11, 2024
51dd487
Reuse send_lonk helper function
jennydaman Sep 11, 2024
39b6aa0
Oopsies
jennydaman Sep 11, 2024
e65f5fe
Add OXIDICOM_DEV_SLEEP
jennydaman Sep 23, 2024
7f49a53
Merge branch 'cube-v6' of github.com:FNNDSC/oxidicom into cube-v6
jennydaman Sep 23, 2024
c1fb19a
Info when OXIDICOM_DEV_SLEEP is set
jennydaman Sep 25, 2024
54e9cd7
Do not panic for "No tasks were received" bug
jennydaman Sep 26, 2024
da4665b
Change default_progress_interval=500ms
jennydaman Sep 26, 2024
157c98c
Tolerate YYYY-MM-DD date string
jennydaman Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
248 changes: 158 additions & 90 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# On push: build latest images

name: CI

on:
Expand All @@ -11,104 +9,174 @@ on:
- '.github/**'
- '**.rs'
- 'Cargo.*'
- 'justfile'
- 'Dockerfile'
- 'docker-compose.yml'
- 'run.sh'
pull_request:
branches: [ master ]

env:
CARGO_TERM_COLOR: always

jobs:
test:
name: Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: FNNDSC/miniChRIS-docker@master
- uses: taiki-e/install-action@v2
with:
tool: just
- name: Start Orthanc
run: docker compose up -d orthanc
- name: Start services
run: docker compose up -d
- name: Download example data
run: docker compose up get-data
- name: Compile test binary
run: just test --no-run
- name: Integration test
run: just test
build:
name: Build
runs-on: ubuntu-22.04
run: docker compose run --rm get-data
- name: Cache rust build
uses: Swatinem/rust-cache@v2
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- name: Run tests
env:
RUST_LOG: oxidicom=debug,integration_test=debug
run: cargo llvm-cov test --locked --codecov --output-path codecov.json
- name: Print service logs
if: failure()
run: docker compose logs
- name: Upload coverage to Codecov
if: always()
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: codecov.json
fail_ci_if_error: true
build-rust:
name: Build Rust binary
runs-on: ubuntu-24.04
needs: [ test ]
strategy:
matrix:
target:
- aarch64-unknown-linux-musl
- x86_64-unknown-linux-musl
steps:
- name: Decide image tags
id: info
shell: python
run: |
import os
import itertools

def join_tag(t):
registry, repo, tag = t
return f'{registry}/{repo}:{tag}'.lower()

registries = ['docker.io', 'ghcr.io']
repos = ['${{ github.repository }}'.lower()]
if '${{ github.ref_type }}' == 'branch':
tags = ['latest']
elif '${{ github.ref_type }}' == 'tag':
tag = '${{ github.ref_name }}'
version = tag[1:] if tag.startswith('v') else tag
tags = ['latest', version]
else:
tags = []
- uses: actions/checkout@v4
- name: Cache rust build
uses: Swatinem/rust-cache@v2
- name: Build
uses: houseabsolute/actions-rust-cross@ad283b2fc65ad1f3a04fb8bf8b2b829aad4a9318
with:
target: ${{ matrix.target }}
command: build
args: --release --locked
- name: Move binary
run: |
mkdir dist
mv target/${{ matrix.target }}/release/oxidicom dist/oxidicom
- name: Calculate checksum
run: |
cd dist
sha256 oxidicom | tee oxidicom.sha256sum
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: 'build__${{ matrix.target }}'
path: dist
if-no-files-found: 'error'
build-docker:
name: Build container image
runs-on: ubuntu-24.04
needs: [ build-rust ]
steps:
- name: Download x86_64 binary
uses: actions/download-artifact@v4
- name: Print out all files
run: find -type f
- name: Print expected checksums
run: find -type f -name '*.sha256sum' -exec sh -c 'echo {} && cat {}' \;
- name: Calculate actual checksums
run: find -type f -name 'oxidicom' -exec sha256sum '{}' \;
- name: Move binaries and mark executable
run: |
mkdir -vp dist/linux/amd64 dist/linux/arm64
mv -v build__x86_64-unknown-linux-musl/oxidicom dist/linux/amd64/oxidicom
mv -v build__aarch64-unknown-linux-musl/oxidicom dist/linux/arm64/oxidicom
chmod -v 555 dist/linux/{amd64,arm64}/oxidicom
- name: Create Dockerfile
run: |
cat > Dockerfile << EOF
# syntax=docker/dockerfile:1
FROM scratch
ARG TARGETPLATFORM
COPY ./dist/\$TARGETPLATFORM/oxidicom /oxidicom
CMD ["/oxidicom"]
EOF
- uses: docker/metadata-action@v5
id: meta
with:
images: |
docker.io/fnndsc/oxidicom
ghcr.io/fnndsc/oxidicom
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
push: true
file: ./Dockerfile
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max

product = itertools.product(registries, repos, tags)
tags_csv = ','.join(map(join_tag, product))
outputs = {
'tags_csv' : tags_csv,
}
with open(os.environ['GITHUB_OUTPUT'], 'a') as out:
for k, v in outputs.items():
out.write(f'{k}={v}\n')
- uses: FNNDSC/miniChRIS-docker@master # need to run CUBE for sqlx to do compile-time validation of SQL queries
- uses: docker/setup-buildx-action@v3
with:
# builder needs to be able to see the Postgres database running in the minichris-local network,
# so that the sqlx crate can do compile-time verification of SQL commands.
driver-opts: network=minichris-local
- name: Login to DockerHub
if: github.event_name == 'push' || github.event_name == 'release'
id: dockerhub_login
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Login to GitHub Container Registry
if: github.event_name == 'push' || github.event_name == 'release'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Get database IP
id: read-network
# sqlx crate needs the environment variable DATABASE_URL to be set for compile-time validation of SQL commands.
# For unknown reasons, docker container service name DNS doesn't work inside the build, so we need to get the
# database container's IP address.
# Re-reun tests many times to detect race conditions.
# https://github.com/FNNDSC/oxidicom/issues/4
retest:
name: Rerun tests many times
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Start services
run: docker compose up -d
- name: Download example data
run: docker compose run --rm get-data
- name: Cache rust build
uses: Swatinem/rust-cache@v2
- name: Compile test binary
run: cargo test --no-run --locked
- name: Run tests
env:
RUST_LOG: oxidicom::notifier=trace,oxidicom::limiter=trace,integration_test=trace
run: |
db_container_name=$(docker ps -f 'label=com.docker.compose.service=db' --format '{{ .Names }}')
ip_address_with_subnet=$(docker network inspect minichris-local --format "{{ range .Containers }}{{ if (eq .Name \"$db_container_name\") }}{{ .IPv4Address }}{{ end }}{{ end }}")
ip_address="${ip_address_with_subnet%/*}"
database_url="postgresql://chris:chris1234@$ip_address:5432/chris"
echo "db_container_name=$db_container_name ip_address=$ip_address database_url=$database_url"
docker run --rm --network minichris-local docker.io/library/postgres:16 psql "$database_url" -c 'SELECT 1 + 1'
echo "DATABASE_URL=$database_url" >> "$GITHUB_OUTPUT"
- name: Build image
uses: docker/build-push-action@v5
id: docker_build
with:
tags: ${{ steps.info.outputs.tags_csv }}
push: ${{ steps.dockerhub_login.outcome == 'success' }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: DATABASE_URL=${{ steps.read-network.outputs.DATABASE_URL }}
set +e
failed_count=0
mkdir logs
for i in {1..20}; do
cargo test > log
rc=$?
echo "Attempt #$i --> $rc"
if [ "$rc" != '0' ]; then
((failed_count++))
mv log logs/$(date +%s).log
fi
done

if [ "$failed_count" != 0 ]; then
cat logs/*.log
echo "::error ::$failed_count out of 20 attempts failed."
exit 1
fi

Loading
Loading