Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Help]: Downloading Landmask_error #199

Open
Sci123-lab opened this issue Jan 7, 2025 · 4 comments
Open

[Help]: Downloading Landmask_error #199

Sci123-lab opened this issue Jan 7, 2025 · 4 comments

Comments

@Sci123-lab
Copy link

  • PyGMTSAR version: 2024.8.30.post3
    Python 3.11

I am having a trouble with step downloading landmask
Any suggestion to how to fix this ? what could be wrong

Tiles().download_landmask(AOI, LANDMASK, product='1s').fillna(0).plot.imshow(cmap='binary_r')

_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\hasan\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
r = call_item()
^^^^^^^^^^^
File "C:\Users\hasan\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in call
return self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\hasan\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py", line 598, in call
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\hasan\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py", line 598, in
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\hasan\anaconda3\envs\PyGMTSAR\Lib\site-packages\pygmtsar\Tiles.py", line 160, in _download_tile
os.remove(tile_filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\hasan\AppData\Local\Temp\S43E170.nc'
"""

The above exception was the direct cause of the following exception:

PermissionError Traceback (most recent call last)
Cell In[18], line 1
----> 1 Tiles().download_landmask(AOI, LANDMASK, product='1s').fillna(0).plot.imshow(cmap='binary_r')

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\pygmtsar\Tiles.py:234, in Tiles.download_landmask(self, geometry, filename, product, skip_exist, n_jobs, debug)
224 def download_landmask(self, geometry, filename=None, product='1s', skip_exist=True, n_jobs=8, debug=False):
225 """
226 Download and merge gzipped NetCDF tiles for land mask.
227
(...)
232 Tiles().download_landmask(S1.scan_slc(DATADIR), 'landmask.nc')
233 """
--> 234 return self.download(
235 base_url = 'https://gmtlandmask.pechnikov.workers.dev/{product}',
236 #base_url = 'https://alexeypechnikov.github.io/gmtlandmask/{product}',
237 path_id = '{SN2}',
238 tile_id = '{SN2}{WE3}.nc.gz',
239 archive = 'gz',
240 filetype = 'netcdf',
241 geometry = geometry,
242 filename = filename,
243 product = product,
244 n_jobs = n_jobs,
245 joblib_backend = 'loky',
246 skip_exist = skip_exist,
247 debug = debug)

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\pygmtsar\Tiles.py:201, in Tiles.download(self, base_url, path_id, tile_id, archive, filetype, geometry, file_id, filename, product, n_jobs, joblib_backend, skip_exist, debug)
198 joblib_backend = 'sequential'
200 with self.tqdm_joblib(tqdm(desc=f'Tiles Parallel Downloading', total=(right-left+1)*(top-bottom+1))) as progress_bar:
--> 201 tile_xarrays = joblib.Parallel(n_jobs=n_jobs, backend=joblib_backend)(joblib.delayed(self._download_tile)
202 (base_url, path_id, tile_id, file_id, archive, filetype, product, x, y, debug)
203 for x in range(left, right + 1) for y in range(bottom, top + 1))
205 tile_xarrays = [tile for tile in tile_xarrays if tile is not None]
206 if len(tile_xarrays) == 0:

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:2007, in Parallel.call(self, iterable)
2001 # The first item from the output is blank, but it makes the interpreter
2002 # progress until it enters the Try/Except block of the generator and
2003 # reaches the first yield statement. This starts the asynchronous
2004 # dispatch of the tasks to the workers.
2005 next(output)
-> 2007 return output if self.return_generator else list(output)

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:1650, in Parallel._get_outputs(self, iterator, pre_dispatch)
1647 yield
1649 with self._backend.retrieval_context():
-> 1650 yield from self._retrieve()
1652 except GeneratorExit:
1653 # The generator has been garbage collected before being fully
1654 # consumed. This aborts the remaining tasks if possible and warn
1655 # the user if necessary.
1656 self._exception = True

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:1754, in Parallel._retrieve(self)
1747 while self._wait_retrieval():
1748
1749 # If the callback thread of a worker has signaled that its task
1750 # triggered an exception, or if the retrieval loop has raised an
1751 # exception (e.g. GeneratorExit), exit the loop and surface the
1752 # worker traceback.
1753 if self._aborting:
-> 1754 self._raise_error_fast()
1755 break
1757 # If the next job is not ready for retrieval yet, we just wait for
1758 # async callbacks to progress.

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:1789, in Parallel._raise_error_fast(self)
1785 # If this error job exists, immediately raise the error by
1786 # calling get_result. This job might not exists if abort has been
1787 # called directly or if the generator is gc'ed.
1788 if error_job is not None:
-> 1789 error_job.get_result(self.timeout)

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:745, in BatchCompletionCallBack.get_result(self, timeout)
739 backend = self.parallel._backend
741 if backend.supports_retrieve_callback:
742 # We assume that the result has already been retrieved by the
743 # callback thread, and is stored internally. It's just waiting to
744 # be returned.
--> 745 return self._return_or_raise()
747 # For other backends, the main thread needs to run the retrieval step.
748 try:

File ~\anaconda3\envs\PyGMTSAR\Lib\site-packages\joblib\parallel.py:763, in BatchCompletionCallBack._return_or_raise(self)
761 try:
762 if self.status == TASK_ERROR:
--> 763 raise self._result
764 return self._result
765 finally:

PermissionError: [Errno 13] The process cannot access the file because it is being used by another process: 'C:\Users\hasan\AppData\Local\Temp\S43E170.nc'

@AlexeyPechnikov
Copy link
Owner

The potential fix is included in commit d670e23. You can install the updated version directly from GitHub to try it out:

pip3 install -Uq git+https://github.com/AlexeyPechnikov/gmtsar.git@pygmtsar2#subdirectory=pygmtsar

@Sci123-lab
Copy link
Author

Thanks, the download issues are solved, however there is another issue I am facing..

So, when I run the download dem code for example (now commented out), it works and download dem, but then when I run the code to display the DEM dem.plot.imshow (cmap='cvidis'), it works and <matplotlib.image.AxesImage at 0x7fde55b16690> appears on the window which means the plot should show up but then the kernel restarts by itself. My laptop is good one, with quite good specs (corei7 13th gen MSI) and enough RAM (16) and SSD (512GB) so I dont think it should be a memory concern, is there a way to know whats wrong exactly, some issue with matplotlib or anything else ????

Screenshot 2025-01-08 174956

@AlexeyPechnikov
Copy link
Owner

It runs on Linux and macOS, but you encounter problems on Windows. I generally don’t recommend using Windows for computational tasks. Instead, the PyGMTSAR project offers a Docker image on DockerHub (https://hub.docker.com/r/pechnikov/pygmtsar) so you can run everything in a Linux environment via Docker on Windows.

If you still prefer to debug your issue on Windows, it may be related to how your Python libraries were installed. Try using PyPI versions instead of Anaconda/Conda or other third-party package archives.

@Sci123-lab
Copy link
Author

OKAY GOT IT. THANKS. I WILL TRY ON DOCKER IMAGE AND SHARE WHAT I GET.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants