Support updating referenced WoT ThingModel minor version for Ditto Things

[thjaeckle]

Currently, the WoT ThingModel is only taken into account when creating new Things in Ditto. In this case, a JSON skeleton of the Thing based on the WoT TM is created.
Over time, a WoT ThingModel might however evolve, e.g. new properties, new actions or new submodels, etc. are added.

Those evolvements can be categorised in:

• non-breaking evolvements (according to Semantic Versioning changes to the MINOR version):
  • only additions to existing functionality without removing existing properties/actions/events
• breaking evolvements (according to Semantic Versioning changes to the MAJOR version)
  • also containing breaking functionality with renaming/removing existing properties/actions/events
  • out of scope for this issue - as this would require e.g. also deleting stuff from things or renaming

In my experience, non-breaking evolvements can happen very regularly, as the (exposed) functionality of IoT devices evolves quickly over time.
For such non-breaking evolvements, Ditto shall provide means to "migrate" existing things to a newer MINOR version of referenced WoT ThingModels.

Concretely, the following steps would have be done when updating the minor version of a WoT ThingModel in a Ditto thing's definition:

• definition of a Thing is modified (either via ModifyDefinition or via MergeThing for the "definition")
• Ditto checks if the "minor" version increased - and only if it did - performs a "migration":
  • resolving all "submodel" versions of the new "minor" thing model
  • update all feature definition fields with the updated "submodel" models
  • potentially adding previously non-yet-existing features in the process
  • for existing features / "submodels" which received a minor update, for each pair of <old model version>,<new model version>:
    • generate JSON skeleton for <old model version>
    • generate JSON skeleton for <new model version>
    • calculate the "diff" as Json Merge Patch from new to old JSON skeleton
    • for all "new added" (non-optional) properties:
      • create the property with the defined default value
      • but only if the Thing does not yet contain a "real" value for the property yet (which could already be sent by the real device - we don't want to overwrite "real" values with default values)

The algorithm might still not be "complete" - but in a nutshell, we have to handle:

• updating Thing attributes with default values for new TM "properties"
• updating Feature properties with default values for new TM "properties"
• updating Feature definitions
• updating the Thing's definition if all was successful
  • TODO: what to do in case of an error?
• and that - all based on a simple update request of a Thing definition

[thjaeckle]

Idea: Provide a "dry run" option to get a preview of what the "migrated" thing would look like.

[hu-ahmed]

This proposal introduces an enhanced model migration endpoint for Eclipse Ditto, aimed at supporting large-scale updates across IoT devices with varying configurations and states. The endpoint enables model migration using a high-level Thing definition URL, combined with precise RQL-based filtering and conditional patching. This approach allows users to selectively apply updates based on specific conditions within each feature, avoiding unnecessary changes and enhancing control over the migration process.

Key Features

• Unified Thing Definition URL:

The endpoint accepts a single Thing definition URL, simplifying the input and eliminating the need to specify each individual feature’s definition.
Ditto will automatically resolve all feature definitions and apply them as needed across targeted devices.

• RQL-Based Filtering for Device Selection:

Users can specify an RQL query in the filter parameter to select devices based on attributes, feature presence, or other conditions.
This allows for precise targeting of devices that match specific criteria, ensuring updates are only applied where needed.

• Conditional Feature Patching with RQL:

The migrationPayload supports feature-specific patchConditions using RQL expressions, enabling granular control over updates.
Conditions such as property existence, threshold comparisons, and complex logical combinations can be defined for each feature to ensure updates are only applied when specified conditions are met.
This approach allows for highly targeted updates, reducing redundant changes and optimizing the migration process.

• Flexible Control Over Feature Addition and Property Initialization:

The endpoint provides options to control whether missing features should be added (addMissingFeatures) and whether properties in newly added features should be initialized (initializeProperties).
This ensures that users can manage updates carefully, without creating unnecessary data for inactive or unconfigured devices.

Proposed Endpoint Specification

Endpoint: POST /things/update-definition
Request Parameters:

• thingDefinitionUrl: The URL for the high-level Thing definition.

• filter: An RQL expression specifying the target devices for the migration.

• migrationPayload: A structured set of operations for each feature, including RQL-based patchConditions.

• addMissingFeatures (optional, boolean): Default false. Controls whether missing features should be added.

• initializeProperties (optional, boolean): Default false. Specifies if default values should be set for properties in new features.

• validationMode (optional, string): Options include warn (default) or strict, controlling the response to wot-model validation for the updates.

Example Payload
Below is an example payload for applying conditional patches based on RQL expressions:

json

{
  "thingDefinitionUrl": "https://models.develop.beyonnex.org/smartheating/smart-radiator-thermostat-2.35.0.tm.jsonld",
  "filter": "and(eq(attributes/type,\"SMART_RADIATOR_THERMOSTAT\"),exists(features/operationalDataReport))",
  "migrationPayload": {
    "thermostat": {
      "configuration": {
        "set": {
          "lowBatteryBehavior": {
            "motorPositionPercent": 5,
            "thresholdVoltage": 4.5
          }
        },
        "patchConditions": "and(gt(status/motorPosition/value,0),lt(configuration/motorLimitConfiguration/limitedPathInMicroMeters,1000))"
      }
    },
    "battery": {
      "status": {
        "voltage": {
          "value": 5.9
        },
        "patchConditions": "notExists(status/voltage)"
      }
    }
  },
  "addMissingFeatures": false,
  "initializeProperties": false,
  "validationMode": "warn"
}

this granular patch-conditions will serve also the implementation of #1927