Site torch dataset update

[Sukh-P]

Pull Request

Description

PR to update the Site Torch Dataset to return samples as xarray Datasets for easier conversion into netcdf files which is the preferred current format of saving samples.

This PR includes:

• Adding a new functions to process the sample dict (dict with xr DataArrays) into one Dataset
• Reordering of when .compute() is called since now we combine multiple DataArrays into a Dataset we can call compute after this is done
• Removed unused site specific parts from original process and combine function
• Updating unit tests now that the data type of the sample has changes in the Torch Dataset
• Updating some time interval syntax to stop a deprecation warning (unrelated to the above changes)

TODO for Site Dataset pipeline overall (won't be in this PR):

• Check this works by creating some samples and adding logic into PVNet to create samples ✅ PR here: Add sample saving for Site Dataset PVNet#290
• Removed saving solar coordinates data in samples for now, current idea is to use the numpy batch functions in here in PVNet to create this data when converting to a numpy batch (if this seems messy may add some logic in here to add to the solar position coordinates to the Dataset)
• Add new functions to go from a Dataset to NumpyBatch/TensorBatch (for passing to the PVNet model)
• Add logic to PVNet/ocf-data-sampler to read the netcdfs and convert to NumpyBatch/TensorBatches and then train a model

[peterdudfield]

Thanks @Sukh-P great to push this forward.

A few quick thoughts, and sorry if these seem obvious and have already been answered

1. is the ideal still to convert the site batch dataset to a dict of tensors ready for the model (PVnet)? if so, will this code sit in here, or in PVNet
2. For different torch dataloaders, do we have an idea of how to do this for the three different process? 1. make batches, 2. load batches and train model, 3. running inference. It would be a shame to have separate torch datasets for each loader, but perhaps there is a simple way to do this. This is very much in your TODO section, so perhaps you have thought about this already / going to next.
3. This might be related to 2., but do you know where the combining samples to batch process fits in?

[Sukh-P]

@peterdudfield thanks, I have tried to answer these:

1. is the ideal still to convert the site batch dataset to a dict of tensors ready for the model (PVnet)? if so, will this code sit in here, or in PVNet

Yes that's still the plan, perhaps still making a NumpyBatch if we want to have a more generic intermediary format, and yes the code will probably be added to here but then called in PVNet, can make that clearer in the TODO list above

2. For different torch dataloaders, do we have an idea of how to do this for the three different process? 1. make batches, 2. load batches and train model, 3. running inference. It would be a shame to have separate torch datasets for each loader, but perhaps there is a simple way to do this. This is very much in your TODO section, so perhaps you have thought about this already / going to next.

I think the longer term plan is to move towards one batch format (netcdf) and have a common interface with batches through a batch object, in this way things will be more generalised and we will have less of having a different way to do things each time, I imagine this will need a bit more thought and can be improved after adding in a working pipeline for sites, can create an issue/discussion around this after we have

3. This might be related to 2., but do you know where the combining samples to batch process fits in?

So I think this is managed by having a function which does some stacking of samples like here into a batch and the Torch DataLoader where you specify how many samples would be in a batch

[Sukh-P]

To stop a syntax deprecation warning from popping up

[peterdudfield]

Could you add a TODO here for normalization satellite data

[peterdudfield]

#87

[peterdudfield]

One small comment could be added, but otherwise looks great

[AUdaltsova]

Out of curiosity bc I think I've missed a conversation somewhere: am I right in thinking that eventually gsp version will also output netcdfs and we will merge it into this function and move convert_to_numpy_batch into the training pipeline?

[Sukh-P]

Yep exactly I think that is the current idea, the convert_to_numpy_batch logic can still be in this repo and then we can call that in PVNet in the training pipeline

[AUdaltsova]

nice!

[AUdaltsova]

Looks super good, thanks a lot for doing all this and the tests!