REPOSTED: error 12728: Field 'tap_nom' is not between (or at) tap_min and tap_max for 1 transformer

[nitbharambe]

Repost of a previous discussion conversation which is to be deleted

willemijnbrus
3 days ago

Hi PGM team,

We are using PGM to do a loadflow on several low voltage grids. Since today we get the following error:
batch #1, error 12728: Field 'tap_nom' is not between (or at) tap_min and tap_max for 1 transformer.
This error sometimes seems to appear and sometimes it does not. Three weeks ago we had some runs on the same input files and we did not have this error. Today we sometimes get this error and sometimes we dont. When we read our input files (grid topology) we define the tap_min and tap_max following the input files. We do not define the tap_nom. below is our code where we read the input files.
Do you have any suggestions on how we can debug this error?

Best,
Willemijn

def read_topology_pgm(
    network_id: str, modify_tap_position: int = 0
) -> tuple[GridTopology, dict[int, int]]:
    """
    Reads network topology into a PGM-ready format.
    :param network: the ID of the network to load or the path to the file
    :return: the PGM-ready format for the given `network_id`
    """
    # Use Polars for uniformity of file access (especially when loading from
    # S3), but convert to Pandas dataframe for easy NumPy conversion
    # TODO(BS): Drop dependency on Pandas by using Polars throughout
    network_file = settings.input_dir / "network" / f"network{network_id}.parquet"
    dataframe = pl.scan_parquet(network_file).collect().to_pandas()

    ean_to_load_mapping: dict[int, int] = {
        int(row["loads__EAN"]): int(row["loads__id"])
        for idx, row in dataframe[["loads__id", "loads__EAN"]].dropna().iterrows()
    }

    # loads
    len(np.array(dataframe["loads__id"]))
    loads = np.array(dataframe["loads__id"].unique())
    loads = loads[~np.isnan(loads)]

    loads_node = np.array(dataframe["loads__node"])
    loads_node = loads_node[~np.isnan(loads_node)]

    loads_status = np.array(dataframe["loads__status"])
    loads_status = loads_status[~np.isnan(loads_status)]

    loads_type = np.array(dataframe["loads__type"])
    loads_type = loads_type[~np.isnan(loads_type)]

    loads_p = np.array(dataframe["loads__p_specified"])
    loads_p = loads_p[~np.isnan(loads_p)]

    loads_q = np.array(dataframe["loads__q_specified"])
    loads_q = loads_q[~np.isnan(loads_q)]

    # node
    node = initialize_array(
        DatasetType.input, ComponentType.node, len(dataframe["nodes__id"].dropna())
    )
    node["id"] = np.array(dataframe["nodes__id"].dropna())
    node["u_rated"] = np.array(dataframe["nodes__u_rated"].dropna())

    # line
    line_dt = dataframe["lines__id"].dropna()
    line = initialize_array(DatasetType.input, ComponentType.line, len(line_dt))
    line["id"] = np.array(dataframe["lines__id"].dropna())
    line["from_node"] = np.array(dataframe["lines__from_node"].dropna())
    line["to_node"] = np.array(dataframe["lines__to_node"].dropna())
    line["from_status"] = np.array(dataframe["lines__from_status"].dropna())
    line["to_status"] = np.array(dataframe["lines__to_status"].dropna())
    line["r1"] = np.array(dataframe["lines__r1"].dropna())
    line["x1"] = np.array(dataframe["lines__x1"].dropna())
    line["c1"] = np.array(dataframe["lines__c1"].dropna())
    line["tan1"] = np.array(dataframe["lines__tan1"].dropna())
    line["i_n"] = np.array(dataframe["lines__i_n"].dropna())

    # load
    sym_load = initialize_array(DatasetType.input, ComponentType.sym_load, len(loads))
    sym_load["id"] = loads
    sym_load["node"] = loads_node
    sym_load["status"] = loads_status
    sym_load["type"] = loads_type
    sym_load["p_specified"] = loads_p
    sym_load["q_specified"] = loads_q

    # source
    source = initialize_array(DatasetType.input, ComponentType.source, 1)
    source["id"] = np.array(dataframe["sources__id"].unique())[
        ~np.isnan(np.array(dataframe["sources__id"].unique()))
    ]
    source["node"] = np.array(dataframe["sources__node"].unique())[
        ~np.isnan(np.array(dataframe["sources__node"].unique()))
    ]
    source["status"] = np.array(dataframe["sources__status"].unique())[
        ~np.isnan(np.array(dataframe["sources__status"].unique()))
    ]
    source["u_ref"] = np.array(dataframe["sources__u_ref"].unique())[
        ~np.isnan(np.array(dataframe["sources__u_ref"].unique()))
    ]

    # transformer
    transformer_id = np.array(dataframe["transformers__id"])[
        ~np.isnan(dataframe["transformers__id"])
    ]
    transformer = initialize_array(
        DatasetType.input, ComponentType.transformer, len(transformer_id)
    )
    transformer["id"] = transformer_id
    transformer["u1"] = np.array(dataframe["transformers__u1"])[
        ~np.isnan(dataframe["transformers__u1"])
    ]
    transformer["u2"] = np.array(dataframe["transformers__u2"])[
        ~np.isnan(dataframe["transformers__u2"])
    ]
    transformer["sn"] = np.array(dataframe["transformers__sn"])[
        ~np.isnan(dataframe["transformers__sn"])
    ]
    transformer["pk"] = np.array(dataframe["transformers__pk"])[
        ~np.isnan(dataframe["transformers__pk"])
    ]
    transformer["i0"] = np.array(dataframe["transformers__i0"])[
        ~np.isnan(dataframe["transformers__i0"])
    ]
    transformer["p0"] = np.array(dataframe["transformers__p0"])[
        ~np.isnan(dataframe["transformers__p0"])
    ]
    transformer["from_node"] = np.array(dataframe["transformers__from_node"])[
        ~np.isnan(dataframe["transformers__from_node"])
    ]
    transformer["to_node"] = np.array(dataframe["transformers__to_node"])[
        ~np.isnan(dataframe["transformers__to_node"])
    ]
    transformer["from_status"] = np.array(dataframe["transformers__from_status"])[
        ~np.isnan(dataframe["transformers__from_status"])
    ]
    transformer["to_status"] = np.array(dataframe["transformers__to_status"])[
        ~np.isnan(dataframe["transformers__to_status"])
    ]
    transformer["uk"] = np.array(dataframe["transformers__uk"])[
        ~np.isnan(dataframe["transformers__uk"])
    ]
    transformer["winding_from"] = np.array(dataframe["transformers__winding_from"])[
        ~np.isnan(dataframe["transformers__winding_from"])
    ]
    transformer["winding_to"] = np.array(dataframe["transformers__winding_to"])[
        ~np.isnan(dataframe["transformers__winding_to"])
    ]
    transformer["clock"] = np.array(dataframe["transformers__clock"])[
        ~np.isnan(dataframe["transformers__clock"])
    ]
    transformer["tap_side"] = np.array(dataframe["transformers__tap_side"])[
        ~np.isnan(dataframe["transformers__tap_side"])
    ]
    transformer["tap_pos"] = np.array(
        dataframe["transformers__tap_pos"] + modify_tap_position
    )[~np.isnan(dataframe["transformers__tap_pos"])]
    transformer["tap_min"] = np.array(dataframe["transformers__tap_min"])[
        ~np.isnan(dataframe["transformers__tap_min"])
    ]
    transformer["tap_max"] = np.array(dataframe["transformers__tap_max"])[
        ~np.isnan(dataframe["transformers__tap_max"])
    ]
    transformer["tap_size"] = np.array(dataframe["transformers__tap_size"])[
        ~np.isnan(dataframe["transformers__tap_size"])
    ]

    # transformer tap regulator
    transformer_tap_regulator = initialize_array(
        DatasetType.input, ComponentType.transformer_tap_regulator, 1
    )
    transformer_tap_regulator["id"] = [888888888]
    transformer_tap_regulator["regulated_object"] = transformer_id
    transformer_tap_regulator["status"] = [1]
    transformer_tap_regulator["control_side"] = [BranchSide.to_side]
    transformer_tap_regulator["u_set"] = [400.0]
    transformer_tap_regulator["u_band"] = [20.0]
    transformer_tap_regulator["line_drop_compensation_r"] = [0.0]
    transformer_tap_regulator["line_drop_compensation_x"] = [0.0]

    # TODO: Ensure that all `sym_load`s have CONSTANT POWER type
    # # Filter all arrays that have constant power
    # selected_loads = np.array([load for load in pgm_network['sym_load'] if (load[3] == LoadGenType.const_power).any()])
    # col_names = selected_loads['id']
    # # Assign the ID's of the objects that should be changed / updated within the model to loads in the loadfile.
    # loads = e_load

    pgm_topology = {
        ComponentType.node: node,
        ComponentType.line: line,
        ComponentType.transformer: transformer,
        ComponentType.sym_load: sym_load,
        ComponentType.source: source,
        ComponentType.transformer_tap_regulator: transformer_tap_regulator,
    }

mgovers
3 days ago
Maintainer
Hi @willemijnbrus , thank you for your question.

While I cannot know for sure without seeing the actual data, I think you are running into the following edge case:

If you look at the input data for Transformers ( https://power-grid-model.readthedocs.io/en/stable/user_manual/components.html#transformer ), then you can see that:

tap_nom is optional and defaults to 0 if you do not provide it.
tap_nom has a valid value if and only if (tap_min <= tap_nom <= tap_max) or (tap_min >= tap_nom >= tap_max)
if not, you will receive the error message you ran into.
It seems you are running into the edge case that 0 is not in the range [tap_min, tap_max], i.e.:

tap_min > 0 and tap_max > 0 or tap_min < 0 and tap_max < 0
In that case, the default value for tap_nom is not a good value for the transformer and you will need to set it yourself.

As a solution, I propose the following. If tap_nom is not relevant for your use case, then you may e.g. set it to any of the following values, whichever makes sense:

tap_pos (if you set it in the input data)
tap_min
tap_max
(tap_min + tap_max) // 2
tap_min or tap_max, whichever is closer to 0.
EDIT: cfr. https://github.com/orgs/PowerGridModel/discussions/43#discussioncomment-11817887 (credits to @TonyXiang8787 )

It is not a problem that tap_min is higher than tap_max. It happens actually regularly. PGM requires that tap_pos and tap_nom have to be between tap_min and tap_max. The following combination of values is valid:

tap_min = 5
tap_max = 1
tap_nom = 3
tap_pos = 4
However, tap_nom=0 is not valid value.

1 reply
@willemijnbrus
willemijnbrus
3 days ago
Author
Hi Martijn, we will have a check if 0 is always in the range of tapmin_tap_max. and also thank you for your suggestion of setting it another value if tap_nom is not important. Very helpful tips. We will try those!

Answer selected by mgovers

BartSchuurmans
5 hours ago
Additional information: in some cases, tap_min was higher than tap_max in our input data, which was why no valid value for tap_pos / tap_nom was possible.

2 replies 1 new
@TonyXiang8787
TonyXiang8787
5 hours ago
Maintainer
Hi @BartSchuurmans,

It is not a problem that tap_min is higher than tap_max. It happens actually regularly. PGM requires that tap_pos and tap_nom have to be between tap_min and tap_max. The following value is valid:

tap_min = 5
tap_max = 1
tap_nom = 3
tap_pos = 4
However, tap_nom=0 is not valid value.

@BartSchuurmans
BartSchuurmans
5 hours ago
Hi @TonyXiang8787, thanks for your quick response. This is good to know, I assumed there was an error in our input data.

[mgovers]

Answer Copy-pasted from OP:

Hi @willemijnbrus , thank you for your question.

While I cannot know for sure without seeing the actual data, I think you are running into the following edge case:

If you look at the input data for Transformers ( https://power-grid-model.readthedocs.io/en/stable/user_manual/components.html#transformer ), then you can see that:

• tap_nom is optional and defaults to 0 if you do not provide it.
• tap_nom has a valid value if and only if (tap_min <= tap_nom <= tap_max) or (tap_min >= tap_nom >= tap_max)
• If not, you will receive the error message you ran into.
• It seems you are running into the edge case that 0 is not in the range [tap_min, tap_max], i.e.:

tap_min > 0 and tap_max > 0 or tap_min < 0 and tap_max < 0
In that case, the default value for tap_nom is not a good value for the transformer and you will need to set it yourself.

As a solution, I propose the following. If tap_nom is not relevant for your use case, then you may e.g. set it to any of the following values, whichever makes sense:

• tap_pos (if you set it in the input data)
• tap_min
• tap_max
• (tap_min + tap_max) // 2
• tap_min or tap_max, whichever is closer to 0.

EDIT: (credits to @TonyXiang8787 )

It is not a problem that tap_min is higher than tap_max. It happens actually regularly. PGM requires that tap_pos and tap_nom have to be between tap_min and tap_max. The following combination of values is valid:

tap_min = 5
tap_max = 1
tap_nom = 3
tap_pos = 4

However, tap_nom=0 is not valid value.

[willemijnbrus]

Thank both of you for your answer. If I understand correctly:
tap nom default = 0 if not provided
tap nom should be in between tap min and tap max
tap min of 5 and tap max of 1 is itself not a problem but if tap nom is not given and default is set to 0, then it will result in tap nom not in between tap min and tap max. Is this correct?

[mgovers]

Thank both of you for your answer. If I understand correctly:
tap nom default = 0 if not provided
tap nom should be in between tap min and tap max
tap min of 5 and tap max of 1 is itself not a problem but if tap nom is not given and default is set to 0, then it will result in tap nom not in between tap min and tap max. Is this correct?

That is exactly correct, yes 😄

[willemijnbrus]

thank you for your fast reply!