Skip to content

Commit

Permalink
move utility cells up to avoid declaration issues
Browse files Browse the repository at this point in the history
  • Loading branch information
@srinijammula
srinijammula committed Dec 12, 2024
1 parent f8942d8 commit d317e4e
Show file tree
Hide file tree
Showing 3 changed files with 158 additions and 158 deletions.
176 changes: 88 additions & 88 deletions book/chapters/amsr.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,93 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "aab13cdf",
"metadata": {},
"source": [
"## Utility Functions for Data Processing and Analysis Pipeline\n",
"\n",
"### 1. Library Imports and Setup for Snow Data\n",
"\n",
"- `KDTree`: From `scipy.spatial`, used for performing efficient nearest-neighbor searches in spatial datasets.\n",
"- `plot_all_variables_in_one_csv`: A custom function from `convert_results_to_images`, used for visualizing processed data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e6d98378",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import h5py\n",
"import subprocess\n",
"import pandas as pd\n",
"import numpy as np\n",
"from datetime import datetime\n",
"from scipy.spatial import KDTree\n",
"import time\n",
"from datetime import datetime, timedelta, date\n",
"import warnings\n",
"import sys\n",
"# from convert_results_to_images import plot_all_variables_in_one_c\n",
"\n",
"homedir = os.path.expanduser('~')\n",
"work_dir = \"../data/gridmet_test_run\"\n",
"test_start_date = \"2024-07-18\"\n",
"western_us_coords = \"../data/dem_file.tif.csv\""
]
},
{
"cell_type": "markdown",
"id": "2550b08d",
"metadata": {},
"source": [
"### 2. Identifying Binary Files\n",
"\n",
"Here we determine whether a given file is a binary file or a text file.\n",
"\n",
"- We attempt to open the file in binary mode (`'rb'`) and read a chunk of bytes (1024 bytes).\n",
"- And the we check for null bytes (`b'\\x00'`), which are common in binary files. If a null byte is found, then it is binary file.\n",
"- Next, we check for a high percentage of non-printable ASCII characters by converting the byte chunk to characters and filtering out non-printable ones. If the chunk has no printable characters, the file is considered binary.\n",
"- If neither of the above conditions are met, the function assumes the file is a text file."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "7004281e",
"metadata": {},
"outputs": [],
"source": [
"def is_binary(file_path):\n",
" try:\n",
" with open(file_path, 'rb') as file:\n",
" # Read a chunk of bytes from the file\n",
" chunk = file.read(1024)\n",
"\n",
" # Check for null bytes, a common indicator of binary data\n",
" if b'\\x00' in chunk:\n",
" return True\n",
"\n",
" # Check for a high percentage of non-printable ASCII characters\n",
" text_characters = \"\".join(chr(byte) for byte in chunk if 32 <= byte <= 126)\n",
" if not text_characters:\n",
" return True\n",
"\n",
" # If none of the binary indicators are found, assume it's a text file\n",
" return False\n",
"\n",
" except FileNotFoundError:\n",
" print(f\"File '{file_path}' not found.\")\n",
" return False\n",
" except Exception as e:\n",
" print(f\"An error occurred: {e}\")\n",
" return False\n",
" "
]
},
{
"cell_type": "markdown",
"id": "05b77e80",
Expand Down Expand Up @@ -1507,94 +1595,6 @@
" # Copy the file to the destination directory\n",
" shutil.copy(source_file_path, destination_dir)"
]
},
{
"cell_type": "markdown",
"id": "aab13cdf",
"metadata": {},
"source": [
"## Utility Functions for Data Processing and Analysis Pipeline\n",
"\n",
"### 1. Library Imports and Setup for Snow Data\n",
"\n",
"- `KDTree`: From `scipy.spatial`, used for performing efficient nearest-neighbor searches in spatial datasets.\n",
"- `plot_all_variables_in_one_csv`: A custom function from `convert_results_to_images`, used for visualizing processed data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e6d98378",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import h5py\n",
"import subprocess\n",
"import pandas as pd\n",
"import numpy as np\n",
"from datetime import datetime\n",
"from scipy.spatial import KDTree\n",
"import time\n",
"from datetime import datetime, timedelta, date\n",
"import warnings\n",
"import sys\n",
"# from convert_results_to_images import plot_all_variables_in_one_c\n",
"\n",
"homedir = os.path.expanduser('~')\n",
"work_dir = \"../data/gridmet_test_run\"\n",
"test_start_date = \"2024-07-18\"\n",
"western_us_coords = \"../data/dem_file.tif.csv\""
]
},
{
"cell_type": "markdown",
"id": "2550b08d",
"metadata": {},
"source": [
"### 2. Identifying Binary Files\n",
"\n",
"Here we determine whether a given file is a binary file or a text file.\n",
"\n",
"- We attempt to open the file in binary mode (`'rb'`) and read a chunk of bytes (1024 bytes).\n",
"- And the we check for null bytes (`b'\\x00'`), which are common in binary files. If a null byte is found, then it is binary file.\n",
"- Next, we check for a high percentage of non-printable ASCII characters by converting the byte chunk to characters and filtering out non-printable ones. If the chunk has no printable characters, the file is considered binary.\n",
"- If neither of the above conditions are met, the function assumes the file is a text file."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "7004281e",
"metadata": {},
"outputs": [],
"source": [
"def is_binary(file_path):\n",
" try:\n",
" with open(file_path, 'rb') as file:\n",
" # Read a chunk of bytes from the file\n",
" chunk = file.read(1024)\n",
"\n",
" # Check for null bytes, a common indicator of binary data\n",
" if b'\\x00' in chunk:\n",
" return True\n",
"\n",
" # Check for a high percentage of non-printable ASCII characters\n",
" text_characters = \"\".join(chr(byte) for byte in chunk if 32 <= byte <= 126)\n",
" if not text_characters:\n",
" return True\n",
"\n",
" # If none of the binary indicators are found, assume it's a text file\n",
" return False\n",
"\n",
" except FileNotFoundError:\n",
" print(f\"File '{file_path}' not found.\")\n",
" return False\n",
" except Exception as e:\n",
" print(f\"An error occurred: {e}\")\n",
" return False\n",
" "
]
}
],
"metadata": {
Expand Down
44 changes: 22 additions & 22 deletions book/chapters/fsCA.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,28 @@
"create an account in urs.earthdata.nasa.gov for earth access"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "377eb1f1",
"metadata": {},
"outputs": [],
"source": [
"work_dir = '../data/fsca'\n",
"def date_to_julian(date_str):\n",
" \"\"\"\n",
" Convert a date to Julian date.\n",
" \"\"\"\n",
" date_object = datetime.strptime(date_str, \"%Y-%m-%d\")\n",
" tt = date_object.timetuple()\n",
" \n",
"\n",
" # Format the result as 'YYYYDDD'\n",
" julian_format = str('%d%03d' % (tt.tm_year, tt.tm_yday))\n",
"\n",
" return julian_format"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -336,28 +358,6 @@
"## 3.5.6 Utility Functions"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "377eb1f1",
"metadata": {},
"outputs": [],
"source": [
"work_dir = '../data/fsca'\n",
"def date_to_julian(date_str):\n",
" \"\"\"\n",
" Convert a date to Julian date.\n",
" \"\"\"\n",
" date_object = datetime.strptime(date_str, \"%Y-%m-%d\")\n",
" tt = date_object.timetuple()\n",
" \n",
"\n",
" # Format the result as 'YYYYDDD'\n",
" julian_format = str('%d%03d' % (tt.tm_year, tt.tm_yday))\n",
"\n",
" return julian_format"
]
},
{
"cell_type": "code",
"execution_count": 12,
Expand Down
96 changes: 48 additions & 48 deletions book/chapters/gridmet.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,54 @@
"GridMET is a high-resolution dataset that provides daily surface meteorological data for the contiguous United States"
]
},
{
"cell_type": "markdown",
"id": "524ca247",
"metadata": {},
"source": [
"## Calculate specific Operation Day\n",
"\n",
"Here we calculate the date exactly three days before the current date and returns it as a formatted string.\n",
"\n",
"- `current_date`: A `datetime` object representing the current date and time.\n",
"- `three_days_ago`: A `datetime` object representing the date three days before the current date.\n",
"- `three_days_ago_string`: A string representing the date three days ago, formatted as \"YYYY-MM-DD\".\n",
"- `test_start_date`: A string that stores the returned value from `get_operation_day()`, representing the operation day used in other parts of the code."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "29df8904",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-08-16\n"
]
}
],
"source": [
"from datetime import datetime, timedelta\n",
"def get_operation_day():\n",
" # Get the current date and time\n",
" current_date = datetime.now()\n",
"\n",
" # Calculate three days ago\n",
" three_days_ago = current_date - timedelta(days=3)\n",
"\n",
" # Format the date as a string\n",
" three_days_ago_string = three_days_ago.strftime(\"%Y-%m-%d\")\n",
"\n",
" print(three_days_ago_string)\n",
"\n",
" return three_days_ago_string\n",
"\n",
"test_start_date = get_operation_day()"
]
},
{
"cell_type": "markdown",
"id": "4aac33f5",
Expand Down Expand Up @@ -1340,54 +1388,6 @@
" return (abs(array - value)).argmin()"
]
},
{
"cell_type": "markdown",
"id": "524ca247",
"metadata": {},
"source": [
"## 11. Calculate specific Operation Day\n",
"\n",
"Here we calculate the date exactly three days before the current date and returns it as a formatted string.\n",
"\n",
"- `current_date`: A `datetime` object representing the current date and time.\n",
"- `three_days_ago`: A `datetime` object representing the date three days before the current date.\n",
"- `three_days_ago_string`: A string representing the date three days ago, formatted as \"YYYY-MM-DD\".\n",
"- `test_start_date`: A string that stores the returned value from `get_operation_day()`, representing the operation day used in other parts of the code."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "29df8904",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-08-16\n"
]
}
],
"source": [
"from datetime import datetime, timedelta\n",
"def get_operation_day():\n",
" # Get the current date and time\n",
" current_date = datetime.now()\n",
"\n",
" # Calculate three days ago\n",
" three_days_ago = current_date - timedelta(days=3)\n",
"\n",
" # Format the date as a string\n",
" three_days_ago_string = three_days_ago.strftime(\"%Y-%m-%d\")\n",
"\n",
" print(three_days_ago_string)\n",
"\n",
" return three_days_ago_string\n",
"\n",
"test_start_date = get_operation_day()"
]
},
{
"cell_type": "markdown",
"id": "ea2c5e26",
Expand Down

0 comments on commit d317e4e

Please sign in to comment.