[Core] Categorical Features

[NudnikShpilkis]

What happened + What you expected to happen

According to the Exogenous Variables Tutorial:

When including exogenous variables always use a scaler by setting the scaler_type hyperparameter. The scaler will scale all the temporal features: the target variable y, historic and future variables.

That means that all non-static variables are scaled and treated as real numbers. Neural-Forecast should give the option of specifying categorical features that should be embedded and non-categorical features that should be scaled. Using the example from the tutorial gen_forecast should be scaled, while week_day should be treated as a categorical.

Versions / Dependencies

1.6.4

Reproduction script

https://nixtlaverse.nixtla.io/neuralforecast/examples/exogenous_variables.html

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rd1886]

Do we have a workaround for this in the meantime?

Thanks!

[candalfigomoro]

I think the handling of categorical exogenous variables is one of the few areas where neuralforecast is lacking. Other libraries such as Darts https://github.com/unit8co/darts allow you to specify for different models (e.g. TFT) what the static categorical variables etc. are so that they can be handled appropriately, however neuralforecast does not provide any ad hoc handling for categorical variables.

[candalfigomoro]

@elephaint What do you think about introducing support for categorical variables (with categorical embedding) in neuralforecast for models that should support them, such as TFT and perhaps TiDE (others?). I ask because I see that you are unifying the API of the models in #1023

[elephaint]

@candalfigomoro Makes sense, maybe a possibility for an EXOGENOUS_CAT attribute in each model. I'd like to unify the APIs of the different classes, then fix some of the losses and make a major doc improvement. I need to think/discuss priorities on that.

[dbalabka]

Also, I would like to add that the original Google implementation of TiDE divides all features into categorical and numerical, which is used to pass features via embeddings:
https://github.com/google-research/google-research/blob/8f818e8fc802c31571a89084d4c61922bf26483e/tide/models.py#L164-L169

Additionally, separating features into two groups can help implement numerical embeddings that could improve the forecasts' quality (https://arxiv.org/abs/2203.05556). AFAIK GCP implementation of TiDE utilizes numerical embeddings, which is different from the paper's original implementation.