2025 ocf-data-sampler - let's make it happen

[peterdudfield]

Copying over from ML-notes

Thank you @Sukh-P @dfulu @AUdaltsova @felix-e-h-p for all your work on this so far

Here are the tasks to do, in order to fully move over from ocf-datapipes to ocf-data-sampler

Phase 1

• Update example configs - PR
• Satellite scaling Satellite normalization #87
• Site torch dataset update #82
• Can save Site samples using ocf-data-sampler/PVnet script PR
• Logic going from netcdfs to tensor batches (Sukh start, maybe review Peter) - sit in ocf-data-sampler - Add training torchdataset #99
• Make separate torch_dataset to load premade netcdf samples - Add training torchdataset #99
• Test training for site level dataset
• Update backtest scripts to use ocf-data-sampler, for uk and site. @zakwatts might be to help on this. TODO create issue in PVNet
• Add time features for sites #77
• Add collate function #100
• Change numpy_batch to numpy_sample #103
• Refactor PVNet/forecast apps after the name change from "batch" to "samples", TODO create issues to do this renaming

Phase 2:

• Train a site level model, and compare with models. This could be an India model as its ready to go.
• Re train UK GSP PVnet model, with satellite scaling. @dfulu to do this later this month by testing end to end
• Fully Remove ocf_datapipes from PVnet, probably move some functions/class from ocf_datapipes to ocf-data-sampler
• Add torch dataset for training summational model (James/Felix), some work done already by James but using an older version of ocf-data-sampler
• Train the summation models (both intraday/DA) with ocf-data-sampler (James/Felix)
• Simplify sample/batch interface (with a Sample/batch class) so there is a common interface used between all samples - Create batch class #71
• Pvnet Da: Use data-sampler in new projects going forward - Train PVnet DA with ocf-data-sampler PVNet#289
• GDM: Use data-sampler in new projecy
• Fully move uk-pvnet-app over to ocf-data-sampler - PVnet Intraday: Upgrade to ocf-data-sampler uk-pvnet-app#116
• Fully move india-forecast-app over to ocf-data-sampler india-forecast-app#164

Not do/ do later: Training like to like batches with ocf-data-sampler and ocf-datapipes

Future:

• Add more features that were in ocf_datapipes, support for different NWPs etc
• Save in zipped folder with separate netcdfs per input
• aswell as lots of other issues
• visualise batches in internal dashboard