notebooks/xradar-pyart/xradar-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../images/logos/xradar_logo.svg\" width=250 alt=\"xradar\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Xradar Basics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "1. Xradar general overview\n",
    "1. Radar data IO\n",
    "2. Radar data georeferencing\n",
    "3. Data visualization\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "| Concepts | Importance | Notes |\n",
    "| --- | --- | --- |\n",
    "| [Intro to Xarray](https://foundations.projectpythia.org/core/xarray.html) | Necessary |  Basic features |\n",
    "| [Radar Cookbook](https://projectpythia.org/radar-cookbook/README.html) | Necessary |  Radar basics |\n",
    "| [Matplotlib](https://foundations.projectpythia.org/core/matplotlib.html) | Necessary |  Plotting basic features |\n",
    "- **Time to learn**: 30 minutes\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xradar as xd\n",
    "import pyart\n",
    "import wradlib as wrl\n",
    "\n",
    "import fsspec\n",
    "import numpy as np\n",
    "from xarray.backends.api import open_datatree\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import cartopy.crs as ccrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xradar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Xradar` is a `Python` package developed to streamline the **reading** and **writing** of radar data across various formats, with exports aligned to standards like ODIM_H5 and CfRadial. Born from the **Open Radar Science Community's** collaboration at **ERAD2022**, `Xradar` uses an `xarray-based` data model, compatible with the upcoming **CfRadial2.1/FM301** standard. This ensures seamless integration with other xarray-based software and existing open-source radar processing tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Xradar data model\n",
    "Xradar leverages the upcoming FM301 standard, a subset of CfRadial2.0, using Xarray to efficiently manage radar data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DataTree Structure (CfRadial2.1/FM301 standard)\n",
    "Xradar employs `xarray.DataTree` objects to organize radar sweeps within a single structure, where each sweep is an `xarray.Dataset` containing relevant metadata and variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../images/CfRadial2.1.svg\" width=500 alt=\"xradar\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xradar importers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Xradar supports several importers to handle different radar data formats. These importers typically include:\n",
    "\n",
    "- **ODIM_H5**: For reading radar data in the ODIM_H5 format, which is widely used in Europe and supported by the EUMETNET OPERA program.\n",
    "\n",
    "- **CfRadial**: For importing data in the CfRadial format, which is commonly used in the atmospheric sciences community.\n",
    "\n",
    "- **SIGMET/IRIS**: For ingesting radar data from SIGMET/IRIS formats, which are used by various weather radar systems globally.\n",
    "\n",
    "- **GAMIC**: For reading data from GAMIC radars, which use their proprietary format.\n",
    "\n",
    "For more importers check [here](https://docs.openradarscience.org/projects/xradar/en/stable/importers.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find some Italian radar data located at the Erad 2024 Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the URL and path for the cloud\n",
    "URL = 'https://js2.jetstream-cloud.org:8001/'\n",
    "path = f'pythia/radar/erad2024'\n",
    "\n",
    "fs = fsspec.filesystem(\"s3\", anon=True, client_kwargs=dict(endpoint_url=URL))\n",
    "\n",
    "fs.glob(f\"{path}/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = fs.glob(\"pythia/radar/erad2024/20240522_MeteoSwiss_ARPA_Lombardia/Data/Cband/*.nc\")\n",
    "files[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "radar_files = [f\"s3://{i}\" for i in files]\n",
    "radar_files[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_files = [\n",
    "    fsspec.open_local(\n",
    "        f\"simplecache::{URL}{i}\", s3={\"anon\": True}, filecache={\"cache_storage\": \".\"}\n",
    "    )\n",
    "    for i in files[:5]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can open one of this `nc` files using `xradar.io.open_cfradial1_datree` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dt = xd.io.open_cfradial1_datatree(local_files[0])\n",
    "display(dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this `Xarray.Datatree` object, the first two nodes contains radar metadata and the others contains `Xarray.Datasets` for each sweeps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sweep in dt.children:\n",
    "    try:\n",
    "        print(f\"{sweep} - elevation {np.round(dt[sweep]['sweep_fixed_angle'].values[...], 1)}\")\n",
    "    except KeyError:\n",
    "        print (sweep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore the `1.0` degrees elevation ('sweep_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_sw2 = dt[\"sweep_2\"].ds\n",
    "display(ds_sw2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xradar visualization\n",
    "\n",
    "We can make a plot using [`xarray.plot`](https://docs.xarray.dev/en/latest/user-guide/plotting.html) functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_sw2.reflectivity.plot(\n",
    "    cmap=\"ChaseSpectral\",\n",
    "    vmax=60,\n",
    "    vmin=-10, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The radar data in the `Xarray.Dataset` object includes range and azimuth as coordinates. To create a radial plot, apply the [`xradar.georeference`](https://docs.openradarscience.org/projects/xradar/en/stable/georeference.html) method. This method will generate x, y, and z coordinates for the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds_sw2 = ds_sw2.xradar.georeference()\n",
    "display(ds_sw2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a radial plot passing `x` and `y` coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds_sw2.reflectivity.plot(\n",
    "    x=\"x\", \n",
    "    y= \"y\",\n",
    "    cmap=\"ChaseSpectral\",\n",
    "    vmax=60,\n",
    "    vmin=-10, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data slicing\n",
    "\n",
    "We can use the power of `Xarray` to acces data within the first 50 kilometers by [slicing](https://foundations.projectpythia.org/core/xarray/xarray-intro.html#slicing-along-coordinates) along coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds_sw2.sel(range=slice(0, 5e4)).reflectivity.plot(\n",
    "    x=\"x\", \n",
    "    y= \"y\",\n",
    "    cmap=\"ChaseSpectral\",\n",
    "    vmax=60,\n",
    "    vmin=-10, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's suposse we want to subset between 90 and 180 degrees angle in azumith"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_sw2.sel(azimuth=slice(90, 180), range=slice(0, 5e4)).reflectivity.plot(\n",
    "    x=\"x\", \n",
    "    y= \"y\",\n",
    "    cmap=\"ChaseSpectral\",\n",
    "    vmax=60,\n",
    "    vmin=-10, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps, we just what to see radar reflectivity along the 100 degrees angle in azimuth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds_sw2.sel(azimuth=100, method=\"nearest\").reflectivity.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xradar integration\n",
    "\n",
    "### Py-Art\n",
    "\n",
    "`Xradar` datatree objects can be ported to `Py-ART` radar objects using the pyart.xradar.Xradar method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "radar = pyart.xradar.Xradar(dt)\n",
    "fig = plt.figure(figsize=[10, 8])\n",
    "display = pyart.graph.RadarMapDisplay(radar)\n",
    "display.plot_ppi_map('reflectivity', sweep=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wradlib\n",
    "\n",
    "`Wradlib` functionality can also be applied to `Xarray.Datatree` objects. For example, the [`wradlib.georef`](https://docs.wradlib.org/en/latest/georef.html) module can be used to enrich data by adding geographical coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in list(dt.children):\n",
    "    if \"sweep\" in key:\n",
    "        dt[key].ds = dt[key].ds.wrl.georef.georeference(\n",
    "            crs=wrl.georef.get_default_projection()\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt[\"sweep_0\"].ds.reflectivity.sel(range=slice(0, 5e4)).plot(\n",
    "    x=\"x\", \n",
    "    y= \"y\",\n",
    "    cmap=\"ChaseSpectral\",\n",
    "    vmax=60,\n",
    "    vmin=-10, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xradar exporters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In xradar, exporters convert radar data into various formats for analysis or integration. Exporting is supported for recognized standards, including:\n",
    "\n",
    "- CfRadial1\n",
    "- CfRadial2\n",
    "- ODIM\n",
    "- Zarr\n",
    "\n",
    "Although [`Zarr`](https://zarr.readthedocs.io/en/stable/getting_started.html) is not a traditional standard format, it provides an analysis-ready, cloud-optimized format that enhances data accessibility and performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt.to_zarr(\"tree.zarr\", consolidated=True, mode=\"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_back = open_datatree(\"tree.zarr\", engine=\"zarr\",chunks={})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(dt_back)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "`Xradar` is a Python library designed for working with radar data. It extends xarray to include radar-specific functionality, such as purpose-based accessors and georeferencing methods. It supports exporting data in various formats, including CfRadial1, CfRadial2, ODIM, and Zarr. xradar facilitates the analysis, visualization, and integration of radar data with other tools and systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources and references\n",
    " - [Xradar](https://docs.openradarscience.org/projects/xradar/en/stable/index.html)\n",
    " - [Radar cookbook](https://github.com/ProjectPythia/radar-cookbook)\n",
    " - [Py-Art landing page](https://arm-doe.github.io/pyart/)\n",
    " - [Wradlib landing page](https://docs.wradlib.org/en/latest/index.html)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "Python 3"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "display_name",
       "op": "patch"
      }
     ],
     "key": "kernelspec",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "Python3"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "display_name",
       "op": "patch"
      }
     ],
     "key": "kernelspec",
     "op": "patch"
    }
   ]
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}