docs/examples/04-Run-Synsat-for-ICON-Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Synsat for ICON Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will \n",
    "- load ICON test data (limited area simulation, stored in separate files, from IFCES2 project)\n",
    "- run Synsat on ICON and\n",
    "- finally store the Synsats into a netcdf file\n",
    "\n",
    "The workflow is similar to [02-Run-Synsat-for-ERA5-Data.ipynb](02-Run-Synsat-for-ERA5-Data.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Env and Load Libraries "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os, sys\n",
    "os.environ['RTTOV_PYTHON_WRAPPER'] = '/work/bb1262/tools/rttov/rttov-v13.2/wrapper'\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no git hash can be obtained\n"
     ]
    }
   ],
   "source": [
    "import synsatipy\n",
    "\n",
    "from synsatipy.synsat import SynSat\n",
    "\n",
    "import synsatipy.synsat_example_data as synsat_example_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps towards Synsat "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step I: Initialize Synsat class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] set cloud / aerosol file to  /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/cldaer_visir/sccldcoef_msg_3_seviri.dat\n",
      "... [synsat] load coefficient file /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/rttov13pred54L/rtcoef_msg_3_seviri_o3.dat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:16  Load coefficients:\n",
      "Load successful >>>>> inst_id : 1, nchannels : 6.\n",
      " 2024/05/05  13:05:16  /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/rttov13pred54L/rtcoef_msg_3_seviri_o3.dat\n",
      " 2024/05/05  13:05:16  /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/cldaer_visir/sccldcoef_msg_3_seviri.dat\n"
     ]
    }
   ],
   "source": [
    "s = SynSat( )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step II: Load Example Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] read data from file  /work/bb1376/data/icon/atlantic-cases/paulette/ifces2-atlanXL-20200907-exp021/POSTPROC//3d_full_base_DOM02_ML_20200912T000000Z_regrid7km.nc\n"
     ]
    }
   ],
   "source": [
    "iconname = synsat_example_data.get_example_data( 'icon02' )\n",
    "s.load( iconname )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ICON data have been loaded into a data handler. You can inspect the data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "icon = s.synsat.data_handler.input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.Dataset> Size: 2GB\n",
      "Dimensions:  (lon: 1094, lat: 767, lev: 70, time: 1)\n",
      "Coordinates:\n",
      "  * lon      (lon) float32 4kB -85.0 -84.94 -84.87 ... -15.18 -15.11 -15.05\n",
      "  * lat      (lat) float32 3kB 0.0 0.06 0.12 0.18 ... 45.78 45.84 45.9 45.96\n",
      "  * height   (lev) float64 560B 1.0 2.0 3.0 4.0 5.0 ... 66.0 67.0 68.0 69.0 70.0\n",
      "  * time     (time) datetime64[ns] 8B 2020-09-12\n",
      "Dimensions without coordinates: lev\n",
      "Data variables:\n",
      "    p        (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    t        (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    q        (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    clwc     (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    ciwc     (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    cswc     (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n",
      "    SKT      (time, lat, lon) float32 3MB dask.array<chunksize=(1, 767, 1094), meta=np.ndarray>\n",
      "    T2M      (time, lat, lon) float32 3MB dask.array<chunksize=(1, 767, 1094), meta=np.ndarray>\n",
      "    SP       (time, lat, lon) float32 3MB dask.array<chunksize=(1, 767, 1094), meta=np.ndarray>\n",
      "    cc       (time, lev, lat, lon) float32 235MB dask.array<chunksize=(1, 39, 767, 1094), meta=np.ndarray>\n"
     ]
    }
   ],
   "source": [
    "print(icon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual ICON data are not really small. At this time, however, the data are not loaded into memory. \n",
    "\n",
    "Maybe this is really too large for our small example here. **Let's redo the input and subset the ICON data!**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step IIb: Subset and Thean Load Again Example Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] read data from file  /work/bb1376/data/icon/atlantic-cases/paulette/ifces2-atlanXL-20200907-exp021/POSTPROC//3d_full_base_DOM01_ML_20200912T000000Z_regrid7km.nc\n"
     ]
    }
   ],
   "source": [
    "iconname = synsat_example_data.get_example_data( 'icon01' )\n",
    "s.load( iconname, isel = {'lon': slice(0,None,4), 'lat': slice(0,None, 4)} )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ICON data have been loaded into a data handler. You can inspect the data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "icon = s.synsat.data_handler.input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.Dataset> Size: 104MB\n",
      "Dimensions:  (lon: 274, lat: 192, lev: 70, time: 1)\n",
      "Coordinates:\n",
      "  * lon      (lon) float32 1kB -85.0 -84.74 -84.49 ... -15.62 -15.37 -15.11\n",
      "  * lat      (lat) float32 768B 0.0 0.24 0.48 0.72 ... 45.12 45.36 45.6 45.84\n",
      "  * height   (lev) float64 560B 1.0 2.0 3.0 4.0 5.0 ... 66.0 67.0 68.0 69.0 70.0\n",
      "  * time     (time) datetime64[ns] 8B 2020-09-12\n",
      "Dimensions without coordinates: lev\n",
      "Data variables:\n",
      "    p        (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    t        (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    q        (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    clwc     (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    ciwc     (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    cswc     (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n",
      "    SKT      (time, lat, lon) float32 210kB dask.array<chunksize=(1, 192, 274), meta=np.ndarray>\n",
      "    T2M      (time, lat, lon) float32 210kB dask.array<chunksize=(1, 192, 274), meta=np.ndarray>\n",
      "    SP       (time, lat, lon) float32 210kB dask.array<chunksize=(1, 192, 274), meta=np.ndarray>\n",
      "    cc       (time, lev, lat, lon) float32 15MB dask.array<chunksize=(1, 39, 192, 274), meta=np.ndarray>\n"
     ]
    }
   ],
   "source": [
    "print(icon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, much better and smaller now!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step III: Setup Run Options and Start Execution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s._options.Nthreads=4\n",
    "s._options.NprofsPerCall = 4000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(0, 4000, None)}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IR emissivity atlas loaded successfully\n",
      "Atlas deallocated.\n",
      " 2024/05/05  13:05:24  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(4000, 8000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:28  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(8000, 12000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:32  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(12000, 16000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:36  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(16000, 20000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:40  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(20000, 24000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:43  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n",
      " 2024/05/05  13:05:44  rttov_check_reg_limits.F90\n",
      "     Input water vapour profile exceeds upper coef limit (profile number =      989)\n",
      " 2024/05/05  13:05:44  Limit   =    20.3040\n",
      " 2024/05/05  13:05:44  p (hPa) =    97.1505\n",
      " 2024/05/05  13:05:44  Value   =    20.7966\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(24000, 28000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:47  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(28000, 32000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:51  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(32000, 36000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:55  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(36000, 40000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:05:59  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(40000, 44000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:06:02  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(44000, 48000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:06:06  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(48000, 52000, None)}\n",
      "... [synsat] atlasses are already loaded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 2024/05/05  13:06:10  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... [synsat] running chunk {'profile': slice(52000, None, None)}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IR emissivity atlas loaded successfully\n",
      "Atlas deallocated.\n",
      " 2024/05/05  13:06:14  Running RTTOV using nthreads =    4 and nprofs_per_call =     4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 8s, sys: 42.1 s, total: 1min 51s\n",
      "Wall time: 57.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "s.run( chunked = True )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step IV: Extract the Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "icon_synsat = s.extract_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.Dataset> Size: 3MB\n",
      "Dimensions:  (time: 1, lon: 274, lat: 192)\n",
      "Coordinates:\n",
      "  * time     (time) datetime64[ns] 8B 2020-09-12\n",
      "  * lon      (lon) float32 1kB -85.0 -84.74 -84.49 ... -15.62 -15.37 -15.11\n",
      "  * lat      (lat) float32 768B 0.0 0.24 0.48 0.72 ... 45.12 45.36 45.6 45.84\n",
      "Data variables:\n",
      "    bt062    (time, lon, lat) float64 421kB 226.2 226.2 226.1 ... 233.6 232.2\n",
      "    bt073    (time, lon, lat) float64 421kB 244.4 244.3 244.2 ... 257.2 256.5\n",
      "    bt087    (time, lon, lat) float64 421kB 278.9 278.9 278.8 ... 285.9 286.0\n",
      "    bt108    (time, lon, lat) float64 421kB 282.8 282.7 282.6 ... 287.9 288.1\n",
      "    bt120    (time, lon, lat) float64 421kB 280.9 280.8 280.8 ... 287.1 287.4\n",
      "    bt134    (time, lon, lat) float64 421kB 252.2 252.1 252.1 ... 264.1 264.3\n",
      "Attributes:\n",
      "    author:                Fabian Senf\n",
      "    contact:               senf@tropos.de\n",
      "    institution:           Leibniz Institute for Tropospheric Research\n",
      "    creation_time:         2024-05-05 13:06:15.126531\n",
      "    synsat_version:        0.1\n",
      "    synsat_githash:        Undefined\n",
      "    license:               CC-BY SA 3.0\n",
      "    _local_software_path:  /home/b/b380352/proj/2022-01_RTTOV-Synsat/synsatip...\n",
      "    input_filename:        /work/bb1376/data/icon/atlantic-cases/paulette/ifc...\n"
     ]
    }
   ],
   "source": [
    "print( icon_synsat )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step V: Store Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... store synsat at ./Data/synsat_icon_example_data.nc\n"
     ]
    }
   ],
   "source": [
    "outdir = './Data'\n",
    "\n",
    "if not os.path.isdir( outdir ):\n",
    "    os.makedirs( outdir )\n",
    "outfile = f'{outdir}/synsat_icon_example_data.nc'\n",
    "\n",
    "print(f'... store synsat at {outfile}')\n",
    "icon_synsat.to_netcdf(outfile)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SynSatiPy (Python 3.10)",
   "language": "python",
   "name": "python3-synsatipy"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}