Skip to content

Latest commit

 

History

History
173 lines (121 loc) · 5.85 KB

development.md

File metadata and controls

173 lines (121 loc) · 5.85 KB

Development

Quick Start

Clone

Clone this repository and init the workspace with following command:

git clone https://github.com/keboola/keboola-as-code
cd keboola-as-code
docker compose build

It is possible when you do development on local environment and use docker dev image for lint/build/test there could be problem with /vendor file.

Call make fix or go mod vendor to fix the vendor packages. Also when rebasing and there are adjustements in go.mod.

Setup ENV

Create .env file with definition of testing projects for example:

TEST_KBC_TMP_DIR=/tmp
TEST_KBC_PROJECTS_FILE=~/keboola-as-code/projects.json

The ~/your/path/to/keboola-as-code/projects.json points to gitignored file that you are supposed to create within the project. The structure of this json file is according to this schema:

[{"host":"connection.keboola.com","project":1234,"stagingStorage":"s3","backend":"snowflake/bigquery","token":"<token>", "legacyTransformation": "false/true"},...]

Staging storage can be s3, abs or gcs, according to the stack. LegacyTransformations are support only on stacks different than gcp.

Start Dev Container

Start an interactive console in a container, run:

docker compose run --rm -u "$UID:$GID" --service-ports dev bash

Start Dev Container inspecting metrics

When we would like to inspect metrics of our services, there have to be made adjustement under dev service. Use following command there, so the prometheus and dev service is connected

command: >
sh -c "git config --global --add safe.directory /code
       make run-<your-service>"

This ensures that the /code is safe directory to run. Make sure that under <your-service> you replace with desired service. E.g make run-apps-proxy. Then run this docker compose command

docker compose up -d

Inspecting prometheus

On localhost:9090 the prometheus is running the UI where your service should be scraped. You can simply check them as graph visually.

Troubleshooting

Strange issues are usually caused by incorrect permissions or file owner somewhere. Here are some tips:

In case you're using Docker Desktop on Linux remove the -u "$UID:$GID". It can cause some permission issues.

Run sudo chown -R <user>:<group> ./ to reset owner on all files.

Run docker volume ls and docker volume rm <volume> to delete the named cache volume.

When you want to use replace github.com/your/module => /newpath in go.mod, there has to be adjusted docker-compose.yml.

services:
    dev:
    ...
    volumes:
        -./:/code:z
        - cache:/tmp/cache
        - /path/in/your/system/module:/newpath
    ...

This breaks usually the editor as the replace cannot be found in /newpath. Therefore be carefull that this works only with docker image and on local you could face issues.

When in .env file there are multiple TEST_KBC_PROJECTS set up and you switch between the stacks, make sure that you rerun the

docker compose run --rm -u "$UID:$GID" --service-ports dev bash

as this command automatically fetches the .env file and sets up TEST_KBC_PROJECTS instead of you.

Run Tests

Run the test suite and download the dependencies using:

make ci

To run only the CLI E2E tests, you can use:

make tests-cli

Use the TEST_VERBOSE=true ENV to run tests with verbose output to see HTTP requests, ENVs, etc.:

TEST_VERBOSE=true go test -race -v -p 1 ./path/to/pkg... -run TestName/SubTest

Use the TEST_HTTP_CLIENT_VERBOSE=true ENV to dump all executed HTTP requests and their responses:

TEST_HTTP_CLIENT_VERBOSE=true TEST_VERBOSE=true go test -race -v -p 1 ./path/to/pkg... -run TestName/SubTest

Use the ETCD_VERBOSE=true ENV to dump all etcd operations:

ETCD_VERBOSE=true TEST_VERBOSE=true go test -race -v -p 1 ./path/to/pkg... -run TestName/SubTest

Build CLI

To compile a local CLI binary to ./target, run in the container:

make build-local

To compile the CLI binary for all architectures to ./target, run in the container:

make build

Start Documentation Server

To start the Go Doc documentation server, run the command bellow, then open http://localhost:6060/pkg/github.com/keboola/keboola-as-code/?m=all:

make godoc

m=all is important to show also private packages.

API Development

API development uses Goa code generator.

To run the API locally:

  1. Start bash in the dev container, run docker compose run --rm -u "$UID:$GID" --service-ports dev bash.
  2. Set env var with Keboola stack: export TEMPLATES_STORAGE_API_HOST=connection.keboola.com (or STREAM_STORAGE_API_HOST)
  3. In the container run make run-templates-api (or make run-stream-service)
  • The API is exposed to http://localhost:8000/ (or http://localhost:8001/
  • When the code changes, the API recompiles and restarts.

To run Stream Worker locally:

  1. Start bash in the dev container, run docker compose run --rm -u "$UID:$GID" --service-ports dev bash.
  2. Set env var with Keboola stack: export STREAM_WORKER_STORAGE_API_HOST=connection.keboola.com
  3. In the container run make run-stream-worker

See provisioning/stream/README.md or provisioning/templates-api/README.md for more details about etcd and running in Minikube.

Inspecting OpenAPI documentation

When we would like to inspect OpenAPI generated by Goa framework, simply go to localhost:port/v1/documentation. Where port is port of the service that the OpenAPI exists.

E2E Tests

See E2E_TESTS.md.

IDE setup

The scripts make mod, make fix, make ci calls go mod vendor. It syncs all dependencies (Go modules) to the vendor directory. So integration with the IDE should work automatically.