Python3.8 (#341)

* initial commit
* Update Dockerfile and requirements to python3.8
* formatted using black
* format 2
* REDASH_THROTTLE_PASS_RESET_PATTERN is configured
* python3.8-slim-buster is added into the Dockerfile
* requirements file deleted instead implemented poetry files
* [Add toggle to disable public URLs](getredash/redash@205915e#diff-bb1f48a9370bdc825649c5ddeffa7ae697c7015dc58d2c3511b752b84b7df8feL178)
* [upgrade flask](getredash/redash@73f49cb)
* various implementations into settings such as `CSRF_TIME_LIMIT`
* [Remove extensions mechanism](getredash/redash#5895)
* authentication updates
* my dashboards section is implemented
* formatted some files using `black`
* dev server doesnt accept parameter of group
* big_query.py revert back into previous version
* client package-lock sha512 to sha1 convertion
* chore: Update dependencies and remove unused files
* Update Python version to 3.8 and use poetry for dev dependencies
* Update Node version to 14 and use nodenv or nvm for managing Node versions
* poetry dependency addded into unit tests
* dependency changes for unit tests
* [Removed unneeded PseudoJSON and DBPersistence classes, migrated all json contained column's type to JSONB](getredash/redash#6687)
* enhanced SETUP documentation
* postgres docker image version from 12 to latest
* [Add default limit (1000) to SQL queries](getredash/redash#5088)
* new settings are implemented
* chore: Update test_cli.py to use dynamic dbpath in test
* [Redis replace hmset -> hset](getredash/redash#6331)
* typos are fixed
* more changes for should_apply_auto_limit
* databrick query runner is enhanced
* cast_int_or_default is intruduced
* pytest lines are cleaned from test_refresh_queries

.circleci/config.yml

@@ -8,14 +8,12 @@ build-docker-image-job: &build-docker-image-job
     - checkout
     - run: sudo apt update
     - run: sudo apt install python3-pip
-    - run: sudo pip3 install -r requirements_bundles.txt
     - run: .circleci/update_version
-    - run: npm run bundle
     - run: .circleci/docker_build
 jobs:
   backend-lint:
     docker:
-      - image: circleci/python:3.7.0
+      - image: circleci/python:3.8
     steps:
       - checkout
       - run: sudo pip install flake8
@@ -73,9 +71,7 @@ jobs:
       - checkout
       - run: sudo apt update
       - run: sudo apt install python3-pip
-      - run: sudo pip3 install -r requirements_bundles.txt
       - run: npm ci
-      - run: npm run bundle
       - run:
           name: Run App Tests
           command: npm test

.github/dependabot.yml

@@ -17,12 +17,6 @@ updates:
     schedule:
       interval: 'weekly'
       day: 'saturday'
-  - directory: "/bin"
-    package-ecosystem: "pip"
-    schedule:
-      interval: "monthly"
-    ignore:
-    - dependency-name: "*"
   - directory: "/"
     target-branch: "develop"
     package-ecosystem: "pip"

.github/workflows/test-unit.yaml

@@ -13,10 +13,13 @@ jobs:
           fetch-depth: 0
       - uses: actions/setup-python@v5
         with:
-          python-version: '3.7'
+          python-version: '3.8'
+      - name: Install pip
+        run: |
+          python -m ensurepip --upgrade
       - shell: bash
         run: |-
-          python -m pip install -r  requirements_bundles.txt flake8 --user
+          pip3 install flake8
       - shell: bash
         run: |-
           export PATH=$PATH:/builder/home/.local/bin

Dockerfile

@@ -17,15 +17,10 @@ RUN if [ "x$skip_frontend_build" = "x" ] ; then \
     touch /frontend/client/dist/multi_org.html &&\
     touch /frontend/client/dist/index.html;\
   fi
-FROM python:3.7-slim-buster
 
-EXPOSE 5000
-
-# Controls whether to install extra dependencies needed for all data sources.
-ARG skip_ds_deps
-# Controls whether to install dev dependencies.
-ARG skip_dev_deps
+FROM python:3.8-slim-buster
 
+EXPOSE 5000
 
 RUN useradd --create-home redash
 
@@ -72,24 +67,27 @@ RUN apt-get update && \
 
 WORKDIR /app
 
-# Disalbe PIP Cache and Version Check
-ENV PIP_DISABLE_PIP_VERSION_CHECK=1
-ENV PIP_NO_CACHE_DIR=1
+ENV POETRY_VERSION=1.6.1
+ENV POETRY_HOME=/etc/poetry
+ENV POETRY_VIRTUALENVS_CREATE=false
+RUN curl -sSL https://install.python-poetry.org | python3 -
+
+COPY pyproject.toml poetry.lock ./
 
-# We first copy only the requirements file, to avoid rebuilding on every file
-# change.
-COPY requirements.txt requirements_bundles.txt  requirements_dev.txt  ./
-RUN if [ "x$skip_dev_deps" = "x" ] ; then pip install -r requirements.txt -r requirements_dev.txt; else echo "Skipping pip install dev dependencies" ; pip install -r requirements.txt; fi
-COPY  requirements_all_ds.txt ./
-RUN if [ "x$skip_ds_deps" = "x" ] ; then pip install -r requirements_all_ds.txt ; else echo "Skipping pip install -r requirements_all_ds.txt" ; fi
+ARG POETRY_OPTIONS="--no-root --no-interaction --no-ansi"
+# for LDAP authentication, install with `ldap3` group
+# disabled by default due to GPL license conflict
+ARG install_groups="main,all_ds,dev"
+RUN /etc/poetry/bin/poetry install --only $install_groups $POETRY_OPTIONS
 
-COPY . /app
+COPY --chown=redash . /app
 COPY --chown=redash --from=frontend-builder /frontend/client/dist /app/client/dist
 RUN chown redash:redash -R /app
 RUN find /app
 USER redash
 ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
+# The version is being set arbitrarily by the builder
 ARG version
 ENV DATAREPORTER_VERSION=$version
 ENTRYPOINT ["/app/bin/docker-entrypoint"]
-CMD ["server"]
+CMD ["server"]

Makefile

@@ -1,4 +1,4 @@
-.PHONY: compose_build up test_db create_database clean down bundle tests lint backend-unit-tests frontend-unit-tests test build watch start redis-cli bash
+.PHONY: compose_build up test_db create_database clean down tests lint backend-unit-tests frontend-unit-tests test build watch start redis-cli bash
 
 compose_build:
 	docker-compose build
@@ -22,9 +22,6 @@ clean:
 down:
 	docker-compose down
 
-bundle:
-	docker-compose run server bin/bundle-extensions
-
 tests:
 	docker-compose run server tests
 
@@ -34,20 +31,19 @@ lint:
 backend-unit-tests: up test_db
 	docker-compose run --rm --name tests server tests
 
-frontend-unit-tests: bundle
+frontend-unit-tests:
 	npm ci
-	npm run bundle
 	npm test
 
 test: lint backend-unit-tests frontend-unit-tests
 
-build: bundle
+build:
 	npm run build
 
-watch: bundle
+watch:
 	npm run watch
 
-start: bundle
+start:
 	npm run start
 
 redis-cli:

SETUP.md

@@ -3,19 +3,33 @@
 DataReporter
 
 Requirements:
-* DataReporter builds correctly with Node 12
-
+* DataReporter builds correctly with Node 14
 Consider using [nodenv](https://joshmorel.ca/post/node-virtual-environments-with-nodenv/)
-
-Requirements:
-* Data reported builds correctly with Node 12
-* Install node 12.22.12 with nodenv and ensure shims are added to PATH
+* Install node v14 with nodenv and ensure shims are added to PATH
 see for more info: https://github.com/nodenv/nodenv#how-it-works
 see https://learn.microsoft.com/en-us/windows/dev-environment/javascript/nodejs-on-wsl for windows-wsl2-nvm
 
 ```
-nodenv install 12.22.12
-nodenv local 12.22.12
+nodenv install 14
+nodenv local 14
+```
+Alternatively you can use nvm
+```
+sudo apt update
+sudo apt install curl
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
+```
+set nvm version
+```
+nvm install v14
+nvm alias default v14
+```
+You should enhance your local files in order to use v14 so type `nano ~/.bashrc` then enhance file according to below
+```
+export NVM_DIR="$HOME/.nvm"
+[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
+
+nvm use v14 > /dev/null
 ```
 
 * Build UI - Required to build ui for
@@ -32,8 +46,8 @@ nodenv local 12.22.12
 
 * Setup docker compose
     * `make up` or `docker-compose up --build`  to start required services like postgres app server
-    * `docker-compose run --rm server create_db` Will start server and run. exec /app/manage.py database create_tables.
-      This step is required **only once**.
+    * `docker-compose run --rm server create_db` Will start server and run. exec /app/manage.py database create_tables. This step is required **only once**.
+    * In case you get an error stating that Target database is not up to date, run: `docker-compose run server manage db stamp head`
     * Any change to SQL data made on python side requires to create a migration file for upgrading the required database columns: `docker-compose run server manage db migrate`
     * Later on and only if necessary, in order to upgrade local database run: `docker-compose run --rm server manage db upgrade`
 
@@ -84,17 +98,14 @@ docker-compose stop server && docker-compose run --rm --service-ports server deb
 ```
 
 ### Running tests locally
-
 First ensure that the "tests" database is created:
 ```
 docker-compose run --rm postgres psql -h postgres -U postgres -c "create database tests"
 ```
-
 Then run the tests:
 ```
 docker-compose run --rm server tests
 ```
-
 In order to test viz-lib folder you need to install dependencies and run tests because you cant have 2 react versions in the same project. To do that run below commands in the viz-lib folder:
 ```
 npm install antd@^3 react@^16.8 react-dom@^16.8 && npm run test
@@ -139,16 +150,18 @@ npm install
 npm run compile
 npm publish
 ```
-### Debugging notes
 
+### Debugging notes
+client side debugger
+```
 cd client
 npm start
-
-visit http://localhost:8080/ instead of using port 5050
-
-To run Python debugger:
+# visit http://localhost:8080/
+```
+Python debugger:
+```
 docker-compose stop server && docker-compose run --rm --service-ports server debug && docker-compose start server
-
+```
 To log messages to/from Plywood add to the Plywood env (in docker-compose) following variable: `LOG_MODE=request_and_response` or `LOG_MODE=response_only`
 
 ### Docker installation issues
@@ -159,6 +172,7 @@ rm  ~/.docker/config.json
 ```
 
 ## Docker connectivity issues for testing connection between containers
+This is usefull when testing fresh datasources
 ```bash
 >> docker network connect datareporter_default router
 >> docker inspect -f '{{range $key, $value := .NetworkSettings.Networks}}{{$key}} {{end}}' router
@@ -173,4 +187,19 @@ PING router (172.19.0.9) 56(84) bytes of data.
 --- router ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 3ms
 rtt min/avg/max/mdev = 0.057/0.781/1.506/0.725 ms
+```
+
+### How to handle package controll on Python side, [see settings](https://github.com/getredash/redash/blob/c97afeb327d8d54e7219ac439cc93d0f234763e5)
+```
+# Install Poetry locally, it has to be 1.6.1 or upper because of group usages
+pip3 install poetry==1.6.1
+
+# Uninstall Poetry locally
+curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/901bdf0491005f1b3db41947d0d938da6838ecb9/get-poetry.py | python3 - --uninstall
+
+# Install additional packages into repository for Python side
+poetry add <package-name>
+
+# Uninstall a package
+poetry remove <package-name>
 ```