Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): sync changes with nr-spar #64

Merged
merged 55 commits into from
Feb 4, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
8b81fbc
Delete sync
DerekRoberts Feb 4, 2025
902bded
feat: reorganize nr-spar-data (#1113)
DerekRoberts May 14, 2024
cfa0baa
chore(ci): build and deploy nr-spar-data's sync (#1119)
DerekRoberts May 14, 2024
65eb587
feat: chunk split and logging (#1124)
marcionemec-daitan May 14, 2024
de77d18
chore(ci): consistent .dockerignore files (#1147)
DerekRoberts May 15, 2024
40b87f2
Disable TEST_MODE by default in sync template
DerekRoberts May 15, 2024
772fe43
fix: Dockerfile for ETL Tool deployment (#1149)
marcionemec-daitan May 15, 2024
8903518
fix(ci): add run attempt to job runs, avoiding retry collisions (#1151)
DerekRoberts May 16, 2024
e281e86
fix: userid columns to handle backlash character for new Spar (#1164)
marcionemec-daitan May 20, 2024
28a94fd
fix: Userid columns transformation from Postgres data to Oracle datab…
marcionemec-daitan May 21, 2024
cd9a2f7
feat: Include data synchronization for non-domain tables from Oracle …
marcionemec-daitan May 24, 2024
92e26c1
feat: etl historical data load from oracle to postgres domain tables …
marcionemec-daitan May 30, 2024
68fc56e
chore(ci): ETL TEST_MODE=0 (#1216)
DerekRoberts Jun 11, 2024
fa4bd29
chore(deps): update dependency numpy to v2 (#1242)
renovate[bot] Jun 19, 2024
e1e08fe
feat(ci): set up sync in Docker Compose (#1248)
DerekRoberts Jun 20, 2024
7705588
feat: ETL Tool revision count update downstream (#1226)
marcionemec-daitan Jun 21, 2024
f3563e0
feat: remove codes (#1275)
ronrobb Jun 26, 2024
e905fe3
feat: ignorecolumnsonupdaate (#1291)
ronrobb Jun 27, 2024
854815d
feat(ci): compose/oc cleanup and consistency (#1306)
DerekRoberts Jul 1, 2024
4318c0f
fix: oracle dialect (#1317)
Jul 3, 2024
7e71fa1
feat: deleteinserts (#1326)
ronrobb Jul 3, 2024
515668c
feat: testingdeletes (#1354)
ronrobb Jul 8, 2024
1ae8d15
chore(ci): sync vars (#1358)
DerekRoberts Jul 8, 2024
c610509
chore(ci): split out sync envars (#1362)
DerekRoberts Jul 8, 2024
11f2074
chore(ci): more password revisions (#1364)
DerekRoberts Jul 8, 2024
760c080
feat: addressnulls (#1369)
ronrobb Jul 9, 2024
089277a
fix: return of rowcount (#1384)
ronrobb Jul 11, 2024
81b58fb
chore: any change to trigger a build for sync (#1399)
DerekRoberts Jul 16, 2024
ce54a1c
chore: small changes for visibility and simplicity (#1400)
DerekRoberts Jul 17, 2024
0b646aa
feat: seedlot_status_code rule implementation (#1407)
ronrobb Jul 17, 2024
6a175a6
chore(deps): update all non-major dependencies (#1416)
renovate[bot] Jul 18, 2024
38aa0b1
feat: ETL testing in PRs, separate deployment triggers from core apps…
DerekRoberts Jul 22, 2024
c9b9621
chore(ci): sync job conversion (#1450)
DerekRoberts Jul 25, 2024
df93df8
chore(ci): sync secrets object (#1472)
DerekRoberts Aug 2, 2024
e3041f6
feat: draftseedlots (#1473)
ronrobb Aug 2, 2024
0ba40e7
feat(ci): sync build_number envar (#1481)
DerekRoberts Aug 6, 2024
6206d5c
feat: utc to pacific and improved error handling (#1493)
ronrobb Aug 14, 2024
df476a4
chore(ci): build_number var in sync (#1502)
DerekRoberts Aug 15, 2024
3ced452
feat: fix empty string for json dates (#1508)
ronrobb Aug 15, 2024
177f3f7
chore(ci): disable etl retries, script revisions (#1515)
DerekRoberts Aug 15, 2024
4c42a4a
chore(deps): update dependency sqlalchemy to v2.0.32 (#1524)
renovate[bot] Aug 16, 2024
0f2080e
chore(ci): PROD ETL (#1533)
DerekRoberts Aug 20, 2024
1db2d90
feat: don't pull invalid json owner data (#1535)
ronrobb Aug 20, 2024
b147565
chore(deps): update all non-major dependencies (#1536)
renovate[bot] Aug 22, 2024
89005bb
feat: empty strings returned for json numerics (#1540)
ronrobb Aug 22, 2024
258b4d2
chore(ci): remove oc override (#1620)
DerekRoberts Sep 19, 2024
0cb0075
chore(deps): update dependency sqlalchemy to v2.0.34 (#1603)
renovate[bot] Sep 19, 2024
9004f3e
chore(deps): update dependency numpy to v2.1.1 (#1602)
renovate[bot] Sep 24, 2024
fed5854
chore(ci): ubuntu-24.04 runner oc changes (#1648)
DerekRoberts Sep 27, 2024
951dd0f
chore(deps): update all non-major dependencies (#1650)
renovate[bot] Oct 1, 2024
352b1b9
Feat/1693 trigger to prevent over write (#1715)
xiaopeng0202 Nov 13, 2024
4a4ea67
fix: return db excuction result (#1718)
xiaopeng0202 Nov 14, 2024
497857e
chore(deps): update all non-major dependencies (#1671)
renovate[bot] Nov 21, 2024
e552495
chore(deps): update python docker tag to v3.13 (#1744)
renovate[bot] Nov 22, 2024
6b72538
chore(deps): update all non-major dependencies (#1835)
renovate[bot] Jan 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions sync/.dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,3 @@ jest.*
.vscode
.ps1
thumbs.db

venv/**
.venv/**
69 changes: 15 additions & 54 deletions sync/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,63 +1,24 @@
# ----------------- STAGE 1 -----------------
FROM python:3.13-slim-bookworm as base
# using slim for development as the wheels are not always compat with
# alpine, and builds can be slow as a result
# FROM python:3.12-slim as base
FROM python:3.13-slim

ARG DEV=false
ENV VIRTUAL_ENV=/app/docker_venv \
PATH="/app/docker_venv/bin:$PATH"
# Receive build number as argument, retain as environment variable
ARG BUILD_NUMBER
ENV BUILD_NUMBER=${BUILD_NUMBER}

# ----------------- STAGE 2 -----------------
FROM base as builder
WORKDIR /app

# Mostly config for poetry
# - disable prompting
# - disable virtualenv creation
# - specify the path to the venv
ENV POETRY_NO_INTERACTION=1 \
POETRY_VIRTUALENVS_IN_PROJECT=0 \
POETRY_VIRTUALENVS_CREATE=0 \
POETRY_CACHE_DIR=/tmp/poetry_cache \
VIRTUAL_ENV=/app/docker_venv \
PATH="/app/docker_venv/bin:$PATH"

# install build tools, and dev libs required to compile postgres stuff
# ideally these are not going to be used as for most of the libs
# there are wheels available, however they are available in case we
# need to compile something
# Packages and nonroot user
RUN apt update && \
apt install -y --no-install-recommends gcc libpq-dev python3-dev && \
useradd -M nonroot

# Install Poetry to global python env
RUN pip install poetry==1.8.4

# Grab the app dependency declarations
COPY pyproject.toml poetry.lock ./

# create the venv in $VIRTUAL_ENV so we can control where it is being created
# unfortunately this is the only way that we can control the venv that
# poetry will use, is to create it ourselves
RUN python -m venv $VIRTUAL_ENV; . $VIRTUAL_ENV/bin/activate
# Disable cache, disable upgrade message, don't write .pyc files
ENV PIP_NO_CACHE_DIR="1"
ENV PIP_DISABLE_PIP_VERSION_CHECK="1"
ENV PYTHONDONTWRITEBYTECODE="1"

# have created and activated the venv, now install the dependencies into it
RUN poetry install --without dev --no-root -v && rm -rf $POETRY_CACHE_DIR

# ----------------- STAGE 3 -----------------
FROM base as runtime
# Copy files and install requirements
WORKDIR /app
COPY . ./
RUN pip install -r ./requirements.txt --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org


# install the postgres client, required by psycopg2
RUN apt update && apt install -y --no-install-recommends postgresql-client

# only copy over the venv from the builder stage
COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}

# RUN apk add bash
COPY src /app/src
COPY config /app/config

CMD ["python", "src/main.py"]
# Start the app
USER nonroot
CMD ["python3", "/app/src/main.py"]
56 changes: 0 additions & 56 deletions sync/README.md

This file was deleted.

803 changes: 0 additions & 803 deletions sync/poetry.lock

This file was deleted.

4 changes: 0 additions & 4 deletions sync/poetry.toml

This file was deleted.

48 changes: 0 additions & 48 deletions sync/pyproject.toml

This file was deleted.

7 changes: 7 additions & 0 deletions sync/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#cx-Oracle==8.3.0
oracledb
numpy==2.2.2
pandas==2.2.3
psycopg2==2.9.10
SQLAlchemy==2.0.34
pyyaml==6.0.2
35 changes: 15 additions & 20 deletions sync/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,25 @@ set -euo nounset
### Setup

# Several packages are not working for python 3.12
# PYVER=3.10.13

# export COMMAND=${1:-}
# if [ "${COMMAND}" == "setup" ]; then
# [ -z "${PYENV_ROOT}" ]|| export PYENV_ROOT="$HOME/.pyenv"
# command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
# eval "$(pyenv init -)"
# pyenv update
# pyenv install -s ${PYVER}
# pyenv global ${PYVER}
# python -m venv --clear venv
# source venv/bin/activate
# python -m pip install --upgrade pip
# fi
PYVER=3.10.13

export COMMAND=${1:-}
if [ "${COMMAND}" == "setup" ]; then
[ -z "${PYENV_ROOT}" ]|| export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
pyenv update
pyenv install -s ${PYVER}
pyenv global ${PYVER}
python -m venv --clear venv
source venv/bin/activate
python -m pip install --upgrade pip
fi


### Requirements

#python -m pip install -r ./requirements.txt --cache-dir ./data
poetry install

# activate env
poetry_path=$(poetry env info --path)
. $poetry_path/bin/activate
python -m pip install -r ./requirements.txt --cache-dir ./data


### Run
Expand Down
66 changes: 0 additions & 66 deletions sync/src/logging.config

This file was deleted.

22 changes: 22 additions & 0 deletions sync/src/logging.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[loggers]
keys=root

[handlers]
keys=stream_handler

[formatters]
keys=sample_formatter

[logger_root]
level=INFO
handlers=stream_handler

[handler_stream_handler]
class=StreamHandler
formatter=sample_formatter
args=(sys.stdout,)

[formatter_sample_formatter]
format=%(levelname)s: %(asctime)s: %(module)s - %(message)s
datefmt=%Y-%m-%d %H:%M:%S

Loading
Loading