WIP: Fix bug in Knowledge Gap detection with duplicate triples

.github/workflows/docker-image-release.yml

@@ -0,0 +1,48 @@
+name: Release Docker images
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    if: github.event_name == 'workflow_dispatch' || (github.event_name == 'release' && github.event.prerelease == false)
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Build JARs
+        run: mvn -Dmaven.test.skip=true package
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+      - name: Build and push Knowledge Directory Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./knowledge-directory
+          platforms: linux/amd64, linux/arm64
+          tags: ghcr.io/tno/knowledge-engine/knowledge-directory:${{ github.event.release.tag_name }}
+          push: true
+      - name: Build and push Smart Connector Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./smart-connector-rest-dist
+          platforms: linux/amd64, linux/arm64
+          tags: ghcr.io/tno/knowledge-engine/smart-connector:${{ github.event.release.tag_name }}
+          push: true
+      - name: Build and push Admin UI Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./admin-ui
+          platforms: linux/amd64, linux/arm64
+          tags: ghcr.io/tno/knowledge-engine/admin-ui:${{ github.event.release.tag_name }}
+          push: true

.github/workflows/draft-release.yml

@@ -0,0 +1,42 @@
+name: Make draft release
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Version for new release (X.Y.Z)"
+        required: true
+        type: string
+
+env:
+  MAVEN_OPTS: -Dmaven.test.skip=true -Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true
+  MAVEN_CLI_OPTS: --batch-mode --errors --fail-at-end --show-version
+
+jobs:
+  setup:
+    runs-on: ubuntu-latest
+    container:
+      image: maven:3.9.9-eclipse-temurin-17-focal
+      options: --user 1001
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build JARs
+        run: mvn $MAVEN_CLI_OPTS package
+      - name: Create tag
+        run: |
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git config user.name "github-actions[bot]"
+          git tag -a ${{ inputs.version }} -m ${{ inputs.version }}
+          git push origin ${{ inputs.version }}
+      - name: Draft release
+        uses: softprops/action-gh-release@v2
+        with:
+          working-directory: ${{ github.workspace }}
+          files: |
+            smart-connector-rest-dist/target/smart-connector-rest-dist-${{ inputs.version }}-with-dependencies.jar
+            knowledge-directory/target/knowledge-directory-${{ inputs.version }}-with-dependencies.jar
+          tag_name: ${{ inputs.version }}
+          draft: true
+          generate_release_notes: true
+          make_latest: true

README.md

@@ -41,9 +41,11 @@ The rest of this README is structured as follows:
 - [Developer information](#developer-information)
   - [Components](#components)
   - [Release steps](#release-steps)
+  - [Code conventions](#code-conventions)
   - [(advanced) Administering a Knowledge Engine runtime](#advanced-administering-a-knowledge-engine-runtime)
     - [Starting the Knowledge Engine in local mode](#starting-the-knowledge-engine-in-local-mode)
     - [Starting the Knowledge Engine in distributed mode](#starting-the-knowledge-engine-in-distributed-mode)
+    - [Configuration](#configuration)
 
 # Demonstration videos and tutorials
 
@@ -71,7 +73,7 @@ The easiest way to start a Knowledge Engine runtime is with Docker:
 ```bash
 docker run \
 	-p 8280:8280 \
-	ghcr.io/tno/knowledge-engine/smart-connector:1.2.5
+	ghcr.io/tno/knowledge-engine/smart-connector:1.3.0
 ```
 
 The Knowledge Engine runtime is now available to use via the REST API at base URL `http://localhost:8280/rest` on your host machine.
@@ -82,18 +84,18 @@ However, running it in the above way **does not support** data exchange with rem
 To interact with other runtimes, it needs additional configuration:
 
 - An additional port mapping for the socket that listens for communication from other runtimes.
-- `KE_RUNTIME_EXPOSED_URL`: The URL via which the above socket is available for communication from the other runtime(s). You need to make sure the traffic is correctly routed to the container's port 8081 with a reverse proxy.
-- `KD_URL`: The URL on which to find the knowledge directory (to discover peers).
+- `ke.runtime.exposed.url`: The URL via which the above socket is available for communication from the other runtime(s). You need to make sure the traffic is correctly routed to the container's port 8081 with a reverse proxy.
+- `kd.url`: The URL on which to find the knowledge directory (to discover peers).
 
-The configuration can be set as follows:
+The configuration can be set as follows (note that the configuration properties below use [underscores and capital letters](#configuration))
 
 ```bash
 docker run \
   -p 8280:8280 \
   -p 8081:8081 \
   -e KD_URL=https://knowledge-directory.example.org \
   -e KE_RUNTIME_EXPOSED_URL=https://your-domain.example.org:8081 \
-  ghcr.io/tno/knowledge-engine/smart-connector:1.2.5
+  ghcr.io/tno/knowledge-engine/smart-connector:1.3.0
 ```
 
 ### Running with Java
@@ -111,7 +113,7 @@ export KE_RUNTIME_EXPOSED_URL=https://your-domain.example.org:8081
 # Start it. The argument (8280) denotes the port number at which it
 # will listen for connections to the Knowledge Engine REST API.
 java -jar -Dorg.slf4j.simpleLogger.logFile=smart-connector.log \
-  smart-connector-rest-dist-1.2.5-with-dependencies.jar 8280
+  smart-connector-rest-dist-1.3.0-with-dependencies.jar 8280
 ```
 
 The JAR can be retrieved by compiling the project:
@@ -181,15 +183,19 @@ This section gives more detailed information about the project's structure, and
 
 The Knowledge Engine project consists of the following Maven modules:
 - `smart-connector`
-  - This is the implementation of the smart connector, with the Java developer API. For instructions on how to use it, refer to [the documentation](./docs/docs/java_developer_api.md).
+	- This is the implementation of the smart connector, with the Java developer API. For instructions on how to use it, refer to [the documentation](./docs/docs/getting_started.md). The high-level design of the smart connector can be found in [the wiki](https://github.com/TNO/knowledge-engine/wiki/High%E2%80%90level-Design).
 - `smart-connector-api`
 	- This module contains interfaces for the smart connector and other classes. It is made as a separate module so that it is easy to use different implementations of the interfaces.
 - `smart-connector-rest-server`
 	- This module contains the REST API layer that is built on top of the Java Developer API.
 - `smart-connector-rest-dist`
-  - A distribution of the server that provides the REST API layer for your smart connector(s), and uses the smart connector implementation from the `smart-connector` module. For instructions on how to use it, refer to [the section below](#using-the-rest-api). For instructions on how to set it up, refer to [this section](#advanced-administering-a-knowledge-engine-runtime).
+	- A distribution of the server that provides the REST API layer for your smart connector(s), and uses the smart connector implementation from the `smart-connector` module. For instructions on how to use it, refer to [the section below](#using-the-rest-api). For instructions on how to set it up, refer to [this section](#advanced-administering-a-knowledge-engine-runtime).
 - `admin-ui`
 	- A REST API which provides meta-data about smart connectors in a knowledge network. Can be used in an administration inferface for a knowledge network. It is implemented as a knowledge base that uses metadata of other knowledge bases.
+- `reasoner`
+	- This module contains the reasoner specifically designed for the distributive nature of the knowledge engine.
+- `knowledge-directory`
+	- This module contains the Knowledge Directory which is used to find other knowledge engine runtimes.
 
 ## Release steps
 These are instructions on what to do when we release a new version of the knowledge engine.
@@ -199,40 +205,21 @@ These are instructions on what to do when we release a new version of the knowle
 	- all `pom.xml` files
 	- `openapi-sc.yaml` version
 	- Docker image tags in the Docker Compose examples.
-2. Make a commit for the release, and tag it with `git tag {x}.{y}.{z}`.
-3. `mvn deploy` (for this you need `Deploy-Token` or `Private-Token` configured in your Maven's `settings.xml`, see [GitLab's documentation on this](https://docs.gitlab.com/ee/user/packages/maven_repository/#authenticate-to-the-package-registry-with-maven))
-4. Push the commit and the tag.
-5. Build and push the new Docker images to GitLab:
-
-```bash
-docker buildx build ./smart-connector-rest-dist --platform linux/arm64,linux/amd64 --tag docker-registry.inesctec.pt/interconnect/knowledge-engine/smart-connector-rest-dist:1.2.5 --push
-docker buildx build ./knowledge-directory --platform linux/arm64,linux/amd64 --tag docker-registry.inesctec.pt/interconnect/knowledge-engine/knowledge-directory:1.2.5 --push
-docker buildx build ./admin-ui --platform linux/arm64,linux/amd64 --tag docker-registry.inesctec.pt/interconnect/knowledge-engine/admin-ui:1.2.5 --push
-```
-
-6. Build and push the new Docker images to GitHub:
-
-```bash
-docker buildx build ./smart-connector-rest-dist --platform linux/arm64,linux/amd64 --tag ghcr.io/tno/knowledge-engine/smart-connector:1.2.5 --push
-docker buildx build ./knowledge-directory --platform linux/arm64,linux/amd64 --tag ghcr.io/tno/knowledge-engine/knowledge-directory:1.2.5 --push
-docker buildx build ./admin-ui --platform linux/arm64,linux/amd64 --tag ghcr.io/tno/knowledge-engine/admin-ui:1.2.5 --push
-```
-
-7. Prepare the next SNAPSHOT version and make a commit for that too.
+2. Make a commit and push it to a new branch for the release.
+3. Make a Pull Request and merge it into master after tests have succeeded.
+4. Go to Actions on GitHub and trigger the workflow "Make draft release" with the new version as input.
+5. When the workflow is finished, go to the newly generated draft release. Check the release notes, make any changes if necessary, and publish the release.
+6. Prepare the next SNAPSHOT version and make a commit for that in a PR and merge into master.
    1. `openapi-sc.yaml`
    2. `pom.xml`
    3. Leave the non-SNAPSHOT version in this README, and in the Docker Compose examples.
-8. In GitLab, create a new release at https://gitlab.inesctec.pt/interconnect/knowledge-engine/-/releases
-   1. Use the new tag for the release
-   2. Write release notes
-   3. Find the latest version of the package at https://gitlab.inesctec.pt/interconnect/knowledge-engine/-/packages/976 and copy the link to the JAR with dependencies. Include it in "Release assets" with type "Package" and link title "Knowledge Engine REST server JAR (with dependencies)" (see previous releases)
-9. In GitHub, create a new release at https://github.com/TNO/knowledge-engine/releases/new
-   1. Use the new tag for the release
-   2. Include the same release notes
-10. Inform mailing list(s) (and [the blog](https://www.knowledge-engine.eu/blog/)) about the new release.
+7. Inform mailing list(s) (and [the blog](https://www.knowledge-engine.eu/blog/)) about the new release.
+
+## Code conventions
+The code conventions of the knowledge-engine can be found in the `/ide` folder in the Eclipse IDE format. The format can often also be imported in other Java IDEs like IntelliJ, VSCode or Netbeans.
 
 ## (advanced) Administering a Knowledge Engine runtime
-To start a new instance of the REST API knowledge engine version 1.2.5, make sure you have `git checkout 1.2.5` the tag `1.2.5`. Now make sure you run the `mvn clean install` command successfully from the root of the repository.
+To start a new instance of the REST API knowledge engine version 1.3.0, make sure you have `git checkout 1.3.0` the tag `1.3.0`. Now make sure you run the `mvn clean install` command successfully from the root of the repository.
 
 ### Starting the Knowledge Engine in local mode
 When no additional configuration parameters are provided, the Knowledge Engine will by default run in local mode. This means you can create multiple smart connectors that can communicate with each other through the REST API, but the Knowledge Engine will not connect to a knowledge directory and will not be able to connect with smart connectors running in other runtimes.
@@ -246,24 +233,27 @@ cd smart-connector-rest-dist/target
 Finally, start the server (note that you can configure a log file by including the `-Dorg.slf4j.simpleLogger.logFile=ke.log` system property to the JVM):
 
 ```bash
-java -Dorg.slf4j.simpleLogger.logFile=ke.log -cp "smart-connector-rest-dist-1.2.5.jar:dependency/*" eu.knowledge.engine.rest.Main 8280
+java -Dorg.slf4j.simpleLogger.logFile=ke.log -cp "smart-connector-rest-dist-1.3.0.jar:dependency/*" eu.knowledge.engine.rest.Main 8280
 ```
 
 If you want to run in it in the background, you can use the `nohup` linux command (which does not use the simpleLogger configuration system property, but redirects the standard err/out):
 
 ```bash
-nohup java -cp "smart-connector-rest-dist-1.2.5.jar:dependency/*" eu.knowledge.engine.rest.Main 8280 > ke.log
+nohup java -cp "smart-connector-rest-dist-1.3.0.jar:dependency/*" eu.knowledge.engine.rest.Main 8280 > ke.log
 ```
 
 ### Starting the Knowledge Engine in distributed mode
 The Knowledge Engine can also start in distributed mode, where it connects with a remote knowledge directory and where different instances of the Knowledge Engine (each instance hosting one or more smart connectors) can communicate with each other. More information about starting the Knowledge Engine in distributed mode can be found in the [documentation](docs/docs/distributed_mode.md).
 
-### Additional configuration environment variables
+### Configuration
+TNO Knowledge Engine uses the [MicroProfile Config 3.1](https://microprofile.io/specifications/config/) specification to configure its behaviour and we use [SmallRye](https://smallrye.io/smallrye-config/) as the implementation of this specification. The default configuration values can be found in the [microprofile-config.properties](./smart-connector/src/main/resources/META-INF/microprofile-config.properties) configuration file. And, as described in the specification, these configuration values can be overridden by [environment variables and system properties](https://download.eclipse.org/microprofile/microprofile-config-3.1/microprofile-config-spec-3.1.html#default_configsources). Note that environment variables can use underscores and capital letters to adhere to their naming conventions and the MicroProfile Config automatically maps those to corresponding configuration properties using [specific rules](https://download.eclipse.org/microprofile/microprofile-config-3.1/microprofile-config-spec-3.1.html#default_configsources.env.mapping).
+
+A description of all configuration properties can be found in the [`SmartConnectorConfig`](./smart-connector/src/main/java/eu/knowledge/engine/smartconnector/impl/SmartConnectorConfig.java) class. The rest of this section highlights some of these configuration properties.
 
 *Increasing the wait time for other KBs to respond*
 
-By default, a Smart Connector waits `10` seconds max for a reply from another Smart Connector when sending an ASK/POST message. This time is configurable via the `KE_KB_WAIT_TIMEOUT` environment variable and setting it to `0` means the Smart Connector will wait indefinitely (this can be useful when dealing with Human KBs).
+By default, a Smart Connector waits `10` seconds max for a reply from another Smart Connector when sending an ASK/POST message. This time is configurable via the `ke.kb.wait.timeout` property and setting it to `0` means the Smart Connector will wait indefinitely (this can be useful when dealing with Human KBs).
 
 *Increasing the HTTP timeouts*
 
-By default, a KER waits `5` seconds max for a HTTP response from another KER when sending a message via the inter-KER protocol. The time is configurable via the `KE_HTTP_TIMEOUT` environment variable.
+By default, a KER waits `5` seconds max for a HTTP connection response from another KER when sending a message via the inter-KER protocol. The time is configurable via the `ke.http.timeout` property