presentations/intro-to-open-radar-science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# An Introduction to Open Radar Science"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is Open Science?\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://www.earthdata.nasa.gov/s3fs-public/2021-11/Circle_Diagram_UPDATE_2.jpg?VersionId=pFRniRpjtgc_MEXUJKi9_sXLoMsSX.pB\" width=\"500\"/>\n",
    "</div>\n",
    "\n",
    "Image based on [Ramachandran et al. 2021](https://doi.org/10.1029/2020EA001562)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What Does this look like in the Weather Radar Community?\n",
    "- Since the late 2000s (and even before) there has been a number of major open source projects released (see e.g. https://openradarscience.org).\n",
    "- Some of them are in a mature stage and are widely used in an academic (mostly) but also operational environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Most make use of modern tools (e.g. github, conda, docker) and practices (e.g. Continuous Integration, automatic tests) that make them easy to evolve and deploy\n",
    "- Most are backed by major weather services or academic institutions\n",
    "Projects are not competing among them but collaborating : Best practices and inter-operability are discussed regularly and joint open source courses have been organized for years at major radar conferences (AMS, ERAD)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Conceptual Idea of the Open Radar Stack"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Common weather radar processing workflows\n",
    "\n",
    "![radar workflows](images/open-radar-workflow-tasks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The Open Radar Tools\n",
    "\n",
    "![open radar tools](images/open-radar-tools.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Additional Tools that Can be used with/for Weather Radar Data\n",
    "\n",
    "- [MetPy](https://unidata.github.io/MetPy/latest/index.html)\n",
    "    - a collection of tools in Python for reading, visualizing, and performing calculations with weather data\n",
    "- [tobac](https://tobac.readthedocs.io/en/latest/)\n",
    "    - a Python package for rapidly identify, track and analyze clouds in different types of gridded datasets, such as 3D model output from cloud-resolving model simulations or 2D data from satellite retrievals."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Introductions of the Core Open Radar Stack"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Wradlib *(Keep the magic to the minimum (let the user decide))*\n",
    "- https://wradlib.org\n",
    "- One of the oldest packages (2011)\n",
    "- Open platform for collaborative development of algorithms\n",
    "- Python-based\n",
    "- Linux/Windows/Mac\n",
    "- Flat data model that allows maximum flexibility to interact with the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- xarray readers available (now in xradar as of 2.0)\n",
    "- Comprehensively addresses the full radar processing chain\n",
    "- Mainly geared to interactive use in research but used in operations too\n",
    "- Easy to install (PyPI, conda, Docker Hub)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Functionality\n",
    "<div>\n",
    "<img src=\"images/wradlib-functionality.svg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sample Image\n",
    "<div>\n",
    "<img src=\"images/wradlib-images.png\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Py-ART *(It's all about the data model)*\n",
    "- https://arm-doe.github.io/pyart/\n",
    "- Created in the context of the ARM program (2013)\n",
    "- Open platform for collaborative development of algorithms\n",
    "- Mostly Python-based (some modules in C, Cython and FORTRAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Linux/Windows/Mac\n",
    "- Core : Radar object that structures the radar data and metadata mirroring the C/F Radial standard\n",
    "- Limited scope. Base block to built upon\n",
    "- Rich ecosystem of packages: ART-VIEW, PyTDA, PyDDA, TINT, Pyrad...\n",
    "- Easy to install (PyPI, conda)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Functionality\n",
    "<div>\n",
    "<img src=\"images/pyart-functionality.svg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sample Image\n",
    "<div>\n",
    "<img src=\"images/pyart-images.png\" width=\"800\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pyrad *(Flexible and replicable data processing chains with no programming)*\n",
    "- https://github.com/MeteoSwiss/pyrad\n",
    "- Initially developed at MeteoSwiss. Now shared development between MeteoSwiss and Météo-France\n",
    "- Python-based weather radar data processing framework capable of operating in real time or off-line\n",
    "- Core based on ARM-DOE Py-ART (Pyrad major contributor)\n",
    "- Easy to install (PyPI, conda)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Functionality\n",
    "<div>\n",
    "<img src=\"images/pyrad-functionality.svg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sample Image\n",
    "<div>\n",
    "<img src=\"images/pyrad-images.jpg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LROSE *(High quality building blocks forcomplex workflows)*\n",
    "- http://lrose.net/ \n",
    "- Based on legacy of NCAR and CSU tools\n",
    "- Fast native cross-platform applications\n",
    "- Mostly C++\n",
    "- Linux/Mac/partially Windows\n",
    "- Many stand-alone tools\n",
    "- Stores data in CF/Radial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Functionality\n",
    "<div>\n",
    "<img src=\"images/lrose-functionality.svg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sample Image\n",
    "<div>\n",
    "<img src=\"images/lrose-images.png\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### BALTRAD *(Advanced Weather Radar Network)*\n",
    "- Heritage from the Nordic Network NORDRAD. Partly funded by the EU. BALTRAD and BALTRAD+ projects (2009-2014). 13 partners in 10 countries\n",
    "- Real-time data exchange and data processing\n",
    "- Sub-packages written in different languages\n",
    "    - Data exchange: JAVA\n",
    "    - Data processing: C and Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Linux/Mac\n",
    "- Distributed networking, partners exchange polar data and process them using a common toolbox\n",
    "- Uses ODIM-H5\n",
    "- Documentation: https://baltrad.github.io/  \n",
    "- Code: https://github.com/baltrad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Functionality\n",
    "<div>\n",
    "<img src=\"images/baltrad-functionality.svg\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sample Image\n",
    "<div>\n",
    "<img src=\"images/baltrad-images.png\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Conclusions on the Open Radar Stack\n",
    "- Acknowledge software when you use it – treat it like a paper\n",
    "    - Most projects have Digital Object Identifiers (DOIs)\n",
    "- Contribute back to projects\n",
    "    - Report bugs, suggest enhancements, share feedback!\n",
    "- Do not be afraid to contribute\n",
    "    - We all started somewhere – your code will be reviewed + tests ensure things do not break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Open Source does not end at making things open\n",
    "    - Support, documentation, consistency, etc. are required\n",
    "- Take a look at existing projects to see if you can collaborate/coordinate!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Open Radar Community"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### More on Open Science: Beyond the Tools\n",
    "<div>\n",
    "<img src=\"images/secret-sauce.png\"/>\n",
    "</div>\n",
    "\n",
    "- Open Science = Open Data + Open Algorithms + Open Software + Open Peer Review + Open Access Publication"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Thereby, science process and outputs can be:\n",
    "    - ✅ Transparent\n",
    "    - ✅ Reproducible\n",
    "    - ✅ Transferrable\n",
    "    - ✅ Collaborative\n",
    "    - ✅ Credible!\n",
    "    - ✅ Durable, sustainable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div class=\"admonition alert alert-info\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Important</p>\n",
    "    No conflict between Intellectual Property Rights (IPR) and licensing.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Recommendations for Open Scientists\n",
    "- Use Git for collaborative change/version code management\n",
    "    - GitHub: github.com \n",
    "    - GitLab: gitlab.com\n",
    "- Linux virtual machine on your local computer\n",
    "    - Cost-effective near-replication of official environments\n",
    "    - Good for development, even while traveling\n",
    "    - VM is the starting point, a vehicle, for creating transferrable computational environments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Today’s cloud instance is an “elaborated” VM (Jupyterhub/Binderhub)\n",
    "- If you have a choice, use an open programming language\n",
    "- Avoid proprietary algorithms like Numerical Recipes\n",
    "- Publish code\n",
    "- Publish data\n",
    "- Publish code & data associated with a study/paper\n",
    "- Publish in Open Access journals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Open Radar Forum - Establishing a Community of Practice\n",
    "https://openradar.discourse.group/\n",
    "\n",
    "<div>\n",
    "<img src=\"images/open-radar-forum.png\"/>\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusions\n",
    "https://openradarscience.org\n",
    "\n",
    "<div>\n",
    "<img src=\"images/openradar-landing-page.png\"/>\n",
    "</div>\n",
    "\n",
    "- There is no “silver bullet” for open science\n",
    "- Try to use open languages where possible\n",
    "- Share your work!\n",
    "- Join the conversation on the forum\n",
    "- You can find more posts relevant to the Open Radar Community on our website (at the top) and on the forum\n",
    "- We have monthly meetings! Join!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}