Pydantic errors are overwhelming in convenience functions

[jdeschamps]

Problem

Since #344, the Pydantic errors caused in the convenience functions are very long and are drowning the real error into a lot of useless (but consequential) errors.

Reproducing the errors

from careamics.config import create_n2v_configuration
create_n2v_configuration(
    experiment_name="N2V_example",
    data_type="arrray", # error in the data type
    axes="YX",
    patch_size=[256, 256],
    batch_size=8,
    num_epochs=30,
)

Leads to the following errors:

ValidationError: 11 validation errors for union[function-after[validate_n2v2(), function-after[validate_3D(), N2VConfiguration]],function-after[validate_3D(), N2NConfiguration],function-after[validate_3D(), CAREConfiguration]]
function-after[validate_n2v2(), function-after[validate_3D(), N2VConfiguration]].data_config.data_type
  Input should be 'array', 'tiff' or 'custom' [type=literal_error, input_value='arrray', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), N2NConfiguration].algorithm_config.algorithm
  Input should be 'n2n' [type=literal_error, input_value='n2v', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), N2NConfiguration].algorithm_config.loss
  Input should be 'mae' or 'mse' [type=literal_error, input_value='n2v', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), N2NConfiguration].data_config.data_type
  Input should be 'array', 'tiff' or 'custom' [type=literal_error, input_value='arrray', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), N2NConfiguration].data_config.transforms.2.XYFlipModel
  Input should be a valid dictionary or instance of XYFlipModel [type=model_type, input_value=N2VManipulateModel(name='...ne', struct_mask_span=5), input_type=N2VManipulateModel]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type
function-after[validate_3D(), N2NConfiguration].data_config.transforms.2.XYRandomRotate90Model
  Input should be a valid dictionary or instance of XYRandomRotate90Model [type=model_type, input_value=N2VManipulateModel(name='...ne', struct_mask_span=5), input_type=N2VManipulateModel]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type
function-after[validate_3D(), CAREConfiguration].algorithm_config.algorithm
  Input should be 'care' [type=literal_error, input_value='n2v', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), CAREConfiguration].algorithm_config.loss
  Input should be 'mae' or 'mse' [type=literal_error, input_value='n2v', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), CAREConfiguration].data_config.data_type
  Input should be 'array', 'tiff' or 'custom' [type=literal_error, input_value='arrray', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
function-after[validate_3D(), CAREConfiguration].data_config.transforms.2.XYFlipModel
  Input should be a valid dictionary or instance of XYFlipModel [type=model_type, input_value=N2VManipulateModel(name='...ne', struct_mask_span=5), input_type=N2VManipulateModel]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type
function-after[validate_3D(), CAREConfiguration].data_config.transforms.2.XYRandomRotate90Model
  Input should be a valid dictionary or instance of XYRandomRotate90Model [type=model_type, input_value=N2VManipulateModel(name='...ne', struct_mask_span=5), input_type=N2VManipulateModel]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type

The reason is that Pydantic will try to validate the overall configuration against all algorithm (N2V, CARE and N2N). The error we should be focusing on is the error coming from the N2V validation, but Pydantic also validates against CARE and N2N, which yields additional validation errors due to the incompatibility between the different algorithms, losses, transforms etc..

Solutions

1 - Avoid using configuration_factory in the convenience function

Since the convenience functions know which algorithm to create, they could simply call the specific one and not use the factory. That would require rewriting a bit the supervised convenience functions.

Advantage is that it clarifies the convenience functions output type.

2 - Filter errors

We could also keep the code base as is, the configuration factory being a one-liner, and wrap it in a try ... catch and filter the errors unrelated to the specific configuration. Less efficient, but maybe smaller foot print.

Probably too hacky.

3 - Use Pydantic Discriminator and Field(discriminator=...)

The proper way to do it with Pydantic. It might make reading a bit more complicated, but it is the most elegant and secure solution! (thanks @melisande-c)

[melisande-c]

Also using discriminated unions will decrease number of errors, with the caveat that the discriminated field has to be included when initialising the models. see toy example below:

from typing import Literal, Union
from pydantic import BaseModel, Field

class A(BaseModel):
    type: Literal["a"] = "a"
    a_param: Literal["x", "y"]

class B(BaseModel):
    type: Literal["b"] = "b"
    b_param: int

class Parent(BaseModel):
    nested: Union[A, B]

class ParentDiscriminated(BaseModel):
    nested: Union[A, B] = Field(discriminator="type")

Example errors without discriminator:

>>> Parent(nested={"type": "a", "a_param": "z"})
ValidationError: 3 validation errors for Parent
nested.A.a_param
  Input should be 'x' or 'y' [type=literal_error, input_value='z', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
nested.B.type
  Input should be 'b' [type=literal_error, input_value='a', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
nested.B.b_param
  Field required [type=missing, input_value={'type': 'a', 'a_param': 'z'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/missing

Example error with discriminator:

>>> ParentDiscriminated(nested={"type": "a", "a_param": "z"})
ValidationError: 1 validation error for ParentDiscriminated
nested.a.a_param
  Input should be 'x' or 'y' [type=literal_error, input_value='z', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error

However the trade off is pydantic will now not infer the classes from the other parameters, e.g.:

>>> ParentDiscriminated(nested={"a_param": "x"})
ValidationError: 1 validation error for ParentDiscriminated
nested
  Unable to extract tag using discriminator 'type' [type=union_tag_not_found, input_value={'a_param': 'x'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/union_tag_not_found

Whereas, without the discriminated union:

>>> Parent(nested={"a_param": "x"})
Parent(nested=A(type='a', a_param='x'))

[melisande-c]

And the TypeAdapter in configuration_factory will still work, just need to change the type to the annotated type below:

adapter: TypeAdapter = TypeAdapter( 
     Anotated[Union[N2VConfiguration, N2NConfiguration, CAREConfiguration], Field(discriminator="type")]
 ) 

We might not need to add a new discriminator field to the Configuration models and just use the algorithm config field.

[jdeschamps]

That's a very good idea!! At least for the Algorithm configuration it comes naturally as you pointed out.

Also, N2VDataConfig will disappear after @CatEek N2V PR. Simplifying a little bit the problem at hand.

Regarding the convenience functions, I still feel that they should explicitly return their configuration (`create_n2v_config(...) -> N2VConfiguration), and in that sense it would make sense to do the following:

def _create_config(...) -> tuple[GeneralAlgorithmConfiguration, DataConfiguration, TrainingConfiguration]:
    ...

def create_n2v_configuration(...) -> N2VConfiguration:
    ...
    algo, data, train = _create_config(...)
    return N2VConfiguration(
        experiment_name="",
        algorithm_config=algo,
        data_config=data,
        trainong_config=train
    )

I am currently (happy to change my mind!) not a big fan of adding yet another entry that says n2v in the configuration. In the end the configuration_factory is mostly for internal use (e.g. we load a configuration from a file or a checkpoint). Users should go through the convenience functions, which ensure proper instantiation and have "knowledge" of what configuration it should instantiate.

[melisande-c]

Having each convenience function return the correct configuration makes sense to me!

Just as a note if the discriminated field is set by default then users (and elsewhere in the code) never have to interact with it e.g.:

class N2VConfiguration(BaseModel):
    experiment_name: str
    # ...
    # other params
    # ...
    configuration_type: Literal["n2v"] = "n2v"

>>> N2VConfiguration(experiment_name="demo") # don't set configuration type
N2VConfiguration(experiment_name='demo', configuration_type='n2v')

Also what I meant in my previous comment was: if the algorithm will always be different for different configurations it might be possible to use the algorithm_config field as the discriminator field. (I haven't experimented with it though so it is yet to be seen if pydantic allows a nested model field to be a discriminator field). And then in the configuration_factory the type adapter will be:

adapter: TypeAdapter = TypeAdapter( 
    Anotated[
        Union[N2VConfiguration, N2NConfiguration, CAREConfiguration], 
        Field(discriminator="algorithm_config")
    ]
 ) 

[jdeschamps]

Also what I meant in my previous comment was: if the algorithm will always be different for different configurations it might be possible to use the algorithm_config field as the discriminator field. (I haven't experimented with it though so it is yet to be seen if pydantic allows a nested model field to be a discriminator field). And then in the configuration_factory the type adapter will be:

adapter: TypeAdapter = TypeAdapter(
Anotated[
Union[N2VConfiguration, N2NConfiguration, CAREConfiguration],
Field(discriminator="algorithm_config")
]
)

Unfortunately, and probably for reasons, discriminator fields need to be Literal.

An alternative in that direction is to use Discriminator in an Annotated type, in which we pass a callable that returns a tag used in the annotation (see here)

[melisande-c]

An alternative in that direction is to use Discriminator in an Annotated type, in which we pass a callable that returns a tag used in the annotation (see here)

Ah that's cool, might be a good solution! But I guess we wait for the N2V PR before trying to implement anything.

[jdeschamps]

So.... I couldn't wait to test it, then almost everything was already done, so I went ahead. Since I worked on the @CatEek's PR, I am fairly certain the two PRs are compatible.

Three changes to solve this issue:

• Algorithm config discrimination with algorithm keyword: this is already done in refac: Move N2V masking in lightning module #365, where we needed to change the configuration a little
• Remove N2VDataConfig, done in refac: Move N2V masking in lightning module #365 as well
• Algorithm-specific configuration with Discriminator: now proposed in feat: Improve Pydantic configuration discrimination #366

[melisande-c]

Ok but if N2VDataConfig is removed then it's no longer necessary to split at the config level?

[jdeschamps]

Hhm that's a very good point indeed. We need to think a little bit what is going to happen with the LVAE-based algorithms. Because there again the data might have requirement for different data. I will open a new issue and let's discuss it there!

#367

[jdeschamps]

Waiting for #365 to close