From my experience in this field, there is a distinct lack of reproducibility and benchmarking capability of new method submissions for forecasting.
Typically, a new submission has standard data requirements of things like temperature, cloud cover, and other variables. They tend to be predictably from renalyses (MERRA2, ERA5) or NWP.
Many of these methods lack reproducibility or trust, and therefore impact. There is simply not a robust enough method of testing. The standard way of benchmarking is to calculated the skill score and compare it to smart persistence (persistence of the clear-sky index to avoid diurnal irradiance patterns). Whilst smart persistence is straightforward, the testing phase is usually at a singular location. There is no saying how well it would do elsewhere, nor if compared to all other methods.
Therefore, I think there is an interesting topic or potential that mainly requires computer skill rather than any science knowledge.
The desire is that, when a researcher believes they have developed the ideal solution, they can submit the code to the OCF benchmarking repository. Two things must be guaranteed at this point (1) security, that the code will not be made public or shared before publication, and (2) an entirely unbiased benchmarking evaluation
Once uploaded, the OCF benchmarking tests the code against a multitude of locations and interesting situations.
Then, it returns metrics of scores, and perhaps even standardised graphics of the preferred style.
Once published, the code is then freely available to all for download.
The obvious challenges are:
- having all the data required to perform the diverse mix of approaches.
- providing a standard code framework template and debugging so that the researcher can produce a workable method that will actually complete when submitted.
2.1 I expect this would be some python script that loaded in the required packages, and then showed how you access some/all of the variables, and then shows what return should be provided preceded by copious assertions to ensure the actual output.
- Ensuring that our benchmark is up to scratch!
The hope would be that OCF would then be the go-to location for forecasting.
I have collaborations with probably the best in this space who could easily provide info/guide on 1 and 3, and perhaps even contribute directly.
There is a possibility that I can dedicate funds next year to making this happen from my new post. But nothing is confirmed yet.