CrossDefModel not working as expected

[LisaWeijers]

Describe the bug
Perhaps this is just another case where I'm just missing one detail on how to use the model/method, but I'm having trouble with the whole model -> dataframe -> model process for crosssection definitions.

I've been trying to adapt the way of working from the developers training and the tutorial in the docs: https://deltares.github.io/HYDROLIB-core/0.2.0/tutorials/dataframe_handling/

Based on this, and some testing, I think this should be the code:

mdu_path= Path.cwd()/'src'/'model'/'moergestels_broek.mdu'

fm = FMModel(mdu_path)
cross_def = pd.DataFrame([cs.__dict__ for cs in fm.geometry.crossdeffile.definition])

# OR:
# m = CrossDefModel(mdu_path/'cross_section_definitions.ini')
# cross_def = pd.DataFrame([d.__dict__ for d in m.definition])

crossdef_new = CrossDefModel(definition=cross_def.to_dict("records"))

So far I tried 3 different models, and have 3 different results.
1. A small dummy model (1 branch, 2 profiles, 0 structures)
The workflow works in this situation.

2. Model from Oosterwolde area (from one of our own projects)
We know this model works in DHydro itself, but I can't even open it with hydrolib. It goes something like this:
(I also tried to get to the CrossDefModel via fm.geometry.crossdeffile.definition but that also doesn't work.)

m = CrossDefModel(Path(r'c:\dhydro\1_modellen\groot_model\werkendmodel_work_run23\fm\crsdef.ini'))

[...]
crsdef.ini -> definition -> 2182 -> KDU-72391 -> __root__
  Cross section is missing any friction specification. (type=value_error)
crsdef.ini -> definition -> 2183 -> KDU-3400 -> __root__
  Cross section is missing any friction specification. (type=value_error)
crsdef.ini -> definition -> 2184 -> KDU-3499 -> __root__
  Cross section is missing any friction specification. (type=value_error)
python-BaseException

In this case, I think it's clear it has to do with the data of the source model. Perhaps the validation is too strict, since the model itself is fine in DHydro. It is therefore annoying that this doesn't work.

3. Model from the developer training
I can read the cross section def from this model, but I cannot convert it back to a model

mdu_path= Path.cwd()/'src'/'model'/'moergestels_broek.mdu'

fm = FMModel(mdu_path)
cross_def = pd.DataFrame([cs.__dict__ for cs in fm.geometry.crossdeffile.definition])

# m = CrossDefModel(mdu_path/'cross_section_definitions.ini')
# cross_def = pd.DataFrame([d.__dict__ for d in m.definition])

crossdef_new = CrossDefModel(definition=cross_def.to_dict("records"))

Both when using fm.geometry.crossdeffile.definition and CrossDefModel() I get an error when I do CrossDefModel(definition=cross_def.to_dict("records"))

Traceback (most recent call last):
  File "C:\Users\908367\.conda\envs\hydrolib\lib\site-packages\hydrolib\core\basemodel.py", line 48, in __init__
    super().__init__(**data)
  File "pydantic\main.py", line 404, in pydantic.main.BaseModel.__init__
  File "pydantic\main.py", line 1040, in pydantic.main.validate_model
  File "pydantic\fields.py", line 734, in pydantic.fields.ModelField.validate
  File "pydantic\fields.py", line 767, in pydantic.fields.ModelField._validate_sequence_like
  File "pydantic\fields.py", line 899, in pydantic.fields.ModelField._validate_singleton
  File "pydantic\fields.py", line 723, in pydantic.fields.ModelField.validate
  File "pydantic\fields.py", line 906, in pydantic.fields.ModelField._validate_singleton
  File "pydantic\fields.py", line 913, in pydantic.fields.ModelField._apply_validators
  File "pydantic\class_validators.py", line 310, in pydantic.class_validators._generic_validator_basic.lambda12
  File "C:\Users\908367\.conda\envs\hydrolib\lib\site-packages\hydrolib\core\io\crosssection\models.py", line 100, in validate
    v = c(**v)
  File "C:\Users\908367\.conda\envs\hydrolib\lib\site-packages\hydrolib\core\basemodel.py", line 48, in __init__
    super().__init__(**data)
  File "pydantic\main.py", line 404, in pydantic.main.BaseModel.__init__
  File "pydantic\main.py", line 1040, in pydantic.main.validate_model
  File "pydantic\fields.py", line 723, in pydantic.fields.ModelField.validate
  File "pydantic\fields.py", line 906, in pydantic.fields.ModelField._validate_singleton
  File "pydantic\fields.py", line 913, in pydantic.fields.ModelField._apply_validators
  File "pydantic\class_validators.py", line 310, in pydantic.class_validators._generic_validator_basic.lambda12
  File "C:\Users\908367\.conda\envs\hydrolib\lib\site-packages\hydrolib\core\io\ini\util.py", line 45, in get_enum
    if entry.lower() == v.lower():
AttributeError: 'float' object has no attribute 'lower'
python-BaseException

Perhaps interesting, this is how the dataframe looks at that point

Expected behavior
I would expect that getting to the cross section definitions, either by reading the ini directly, or through fm.geometry, would give me crosssection definition model. I also expect to be able to convert that model to Pandas and then back again.

Version info (please complete the following information):

• OS: Windows 10
• Version: 0.2.0
• Python: 3.8

---

I hope someone can give some pointers. Am I using the methods correctly? Where do these issues come from?

Also, best wishes for the new year! :)

[arthurvd]

Ad.2:
Can you show your cross section definition for 'KDU-72391'? Or share the whole model input?

Ad.3:
I have investigated it, and this is a bug introduced because we now have "enum" datatypes, for example for the friction type. And apparently when a friction type is optional, the pandas dataframe contains a "nan" value, which causes an unnecessary error in the enum validator. I have created a separate issue for this --> #201.

[LisaWeijers]

@arthurvd
Ad.2:
Here is a snippet:

[Definition]
    Id                    = KDU-72391           
    Type                  = circle              
    Thalweg               = 0.000               
    Diameter              = 1.000               

[Definition]
    Id                    = KDU-3400            
    Type                  = circle              
    Thalweg               = 0.000               
    Diameter              = 1.000 

Full file (with a .txt thrown at the end because github doesn't want me to share .ini files ;))
crsdef.ini.txt

I will check out Ad.3 and the seperate issue after my lunch break, but thanks for your thoughts so far!

[LisaWeijers]

Ad.3
The workaround seems sufficient with the 'moergestels_broek.mdu' model!
Changing nan to None in the dataframe is enough to later read it with CrossDefModel(definition=...)

Concerning Ad.2:
I expect that the issue is related to reading the ini file and missing a required field? I'm having two thoughts regarding the matter:

• The model can be run by d-hydro in this state. Maybe it is a sloppy model, but I'd simply hope that a model that calculates fine could be opened by hydrolib methods.
• What about the user-case where the user wants to fix mistakes in their model using hydrolib? I could see myself doing such a thing in the future if I need to work with a model but I found out that all structures miss a friction value, it would be nice to open the bad model with hydrolib, fill in missing values, and export a fixed model.

I'm curious to hear more about your thoughts on the issue :)

[arthurvd]

Question 2 has now been answered+fixed under #205.
Still, your two questions are relevant enough to be answered:

• The model can be run by d-hydro in this state. Maybe it is a sloppy model, but I'd simply hope that a model that calculates fine could be opened by hydrolib methods.

Yes: if a model is valid for D-HYDRO, then HYDROLIB-core should not complain. This was a bug.

• What about the user-case where the user wants to fix mistakes in their model using hydrolib? I could see myself doing such a thing in the future if I need to work with a model but I found out that all structures miss a friction value, it would be nice to open the bad model with hydrolib, fill in missing values, and export a fixed model.

I can see more need for this indeed, I've opened a dedicated issue about bypassing validation, so we can clearly answer that there: #207.

[arthurvd]

All questions answered/fixed, or split off into separate issues. Closing this issue.