Should DAG Conf be made immutable?

[nsAstro]

There have been a few instances where users write DAG's which edit the value of the 'dag_conf' object during the course of a DAG run. This can lead to various undefined behaviors such as race conditions and deserialization issues when custom objects are passed in. Currently this model is accessible through the DAG kwargs and can be freely edited by any task of a given DAG run. Personally, I feel that this should be a little more structured and protected by at least requiring the use of explicit get and set methods similar to XCOMs. Additionally, it might help to have some guidance on proper use of DAG config and the dangers of using it as a mutable model.

From my original understanding, I thought the purpose of DAG_Conf was to preserve any relevant state/metadata of a given DAG run so that it could be rerun later to give the exact same behavior. For this reason, I would generally expect that dag_conf be immutable or at most only accept new keys without allowing mutation of old ones. I'm curious as to others' thoughts on this.

[MatrixManAtYrService]

If there are cases where we want users to mutate context["dag_run"].conf, it probably would be best to make the process more explicit.

For instance, I like how dataclasses.replace and pyrsistent evolvers allow the user to get a mutated copy of the immutable object. We could then have them pass it into an update_conf function or something like that. That way there is a place for guarding against problematic updates and explicitly supporting other ones.

[potiuk]

Feel free to make PRs for those, if you think it is worth. Only "show me the code" makes sense in this case - general discussion about immutability is simply not convincing enough and not clear enough what benefits it brings. Showing the codea and examples where it would be helpful as part of it might be more convincing.

Basically if thins are onlhy implemented in OSS if they .... get implemented.