speed up download for "infinite" basemap

[mikilterribile]

Dear contextily Team,

Good evening. Since last year, I have been developing a GIS and hydraulic software, which is freely available at www.hydro-gis.com. I have been utilizing your contextily package in Python to download online basemaps from various sources. I find contextily to be simple, effective, and user-friendly, and I would like to express my gratitude for your excellent work.

I am now looking to advance a step further by creating an "infinite" basemap using contextily. The idea is to have a background where, as the user pans or zooms in/out, the software automatically downloads the correct tiles and displays them, creating the impression of a seamless, infinite basemap.

However, I am encountering a challenge: the download speed of contextily tiles is too slow for this purpose. I have explored several methods to expedite this process, such as using bounds2img instead of bounds2raster, but I haven't observed any significant time difference. I also tried implementing multithreading to enable multiple simultaneous requests to the server, which showed some improvement but was still insufficient.

Could you provide any advice or suggestions on how I might speed up the tile download process using contextily? Any guidance or insights would be greatly appreciated.

Thank you for your attention to this matter.

Best regards,

Michele Zucchelli

[martinfleis]

I am not sure contextily is built for this kind of a use case. It would probably be interesting to see some profiling of the download to understand where the bottleneck is but it may be better to switch something more suitable (and probably written in JavaScript if that is an option).

[mikilterribile]

I know nothing about JavaScript, and even if I did, I wouldn't know how to use JavaScript libraries within a Python program. So, I don't think JavaScript is a good option for me. I could try to access the Python code of the 'Bound2Draster' function in Contextily and try to understand how to speed it up and optimize it. However, honestly, I don't think I am capable of doing it.

Thank you for your answer.

Best regards,

Michele Zucchelli.

[darribas]

Depending on your application, something like geopandas.GeoDataFrame.explore() might be a Python solution that provides what you want? Under the hood, it is using Leafmap, which is a javascript framework, but you do not need to touch it directly. My sense is that this would be the best way to provide an "infinite map" than contextily, which is designed for static (finite) maps.

[mikilterribile]

Yes, but the application I developed uses matplotlib embedded in a wxpython application. So all the maps, rasters, vectors are displayed within a matplotlib canvas. I want to show the infinite basemaps as a background for the layers, so I had to embed the folium map in my matplotlib canvas, witch is impossible, I think. Thank you for your answer, best regards, Michele Zucchelli. Il dom 21 gen 2024, 15:37 Dani Arribas-Bel ***@***.***> ha scritto:

…

Depending on your application, something like geopandas.GeoDataFrame.explore() <https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.explore.html> might be a Python solution that provides what you want? Under the hood, it is using Leafmap, which is a javascript framework, but you do not need to touch it directly. My sense is that this would be the best way to provide an "infinite map" than contextily, which is designed for static (*finite*) maps. — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A55OONGJBMYZYJPFDQI4RNTYPUR3ZAVCNFSM6AAAAABBYUTLKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSGY2TONJUGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[darribas]

Ahhh I see. I'm not sure I have an answer for this, I'm sorry. This is not a use case we had in mind when developing contextily so it is possible it's not ideally suited. Best luck!

[mikilterribile]

Hi, trying to develop this functionality for my gis application, I had to improve some contextily functionalities, speeding up the download of the tiles and the reproject (warp) function. I'll be very glad to share this improvement with you, the contextily team, but I don't know what is the best way to do it or wheter if you are interested in. Let me know about it. Best regards, Michele Zucchelli.

[martinfleis]

We are surely interested! Ideally, open a pull request within this repository. Alternatively, you can point us to the code uploaded somewhere if you're not comfortable with PRs.

[mikilterribile]

Ok, tomorrow I ' ll do it Il gio 25 gen 2024, 17:16 Martin Fleischmann ***@***.***> ha scritto:

…

We are surely interested! Ideally, open a pull request within this repository. Alternatively, you can point us to the code uploaded somewhere if you're not comfortable with PRs. — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A55OONDZ2XO5ECAHRRPFZ6TYQKAO7AVCNFSM6AAAAABBYUTLKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGUZTIMRZGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mikilterribile]

Hi, I've tryed open a pull request but I cannot, 'cause I'm not a collaborator and I don't know how to become a collaborator. So I'm going to share my edits by email. Hi everyone, I have modified contextily on my own in order to speeding up download and reprojection. I would like to share this improvements with you: 1) Use always threads instead of processes to handle multiple connections. Simply change the following line in the function: "bounds2img": "preferred_backend = "threads" The threads are much faster to be initialized and for operations I/O like the tiles dowload python allows the GIL. 2)Create a session with the request package. The session keeps active the connection with the server, speeding up the tiles downlod time. session = requests.Session() # Configura il numero massimo di retry per le connessioni retries = Retry(total=5, backoff_factor=0.1, status_forcelist=[500, 502, 503, 504]) session.mount('http://', HTTPAdapter(max_retries=retries)) session.mount('https://', HTTPAdapter(max_retries=retries)) timeout = 10 #10 secondi def _retryer(tile_url, _, __): """ Attempt to get a tile from a URL. Arguments --------- tile_url : str Properly-formatted URL for a tile provider. Returns ------- request object containing the web response. """ try: response = session.get(tile_url, headers={"user-agent": USER_AGENT}, timeout = timeout) response.raise_for_status() # Solleva un'eccezione per codici di risposta 4xx o 5xx except requests.HTTPError as http_err: if response.status_code == 404: raise requests.HTTPError(f"Tile URL resulted in a 404 error. Check your tile URL:\n{tile_url}") from http_err else: raise # Rilancia l'eccezione per altri errori HTTP return response 3)I have implemented a reproject function that is faster, because doesn't create a temporary file but it is "on the fly": def warp_tiles(img, extent, t_crs, resampling=Resampling.bilinear): """ Riproietta un array NumPy da un CRS a un altro. Args: - img: Array NumPy dell'immagine. - t_crs: Sistema di coordinate destinazione. Returns: - img_reprojected: Array NumPy dell'immagine riproiettata. """ src_crs = 3857 # Ottieni le dimensioni dell'immagine e la trasformazione di partenza e di arrivo height, width, bands = img.shape # --- https://rasterio.readthedocs.io/en/latest/quickstart.html#opening-a-dataset-in-writing-mode minX, maxX, minY, maxY = extent x = np.linspace(minX, maxX, width) y = np.linspace(minY, maxY, height) resX = (x[-1] - x[0]) / width resY = (y[-1] - y[0]) / height transform = from_origin(x[0] - resX / 2, y[-1] + resY / 2, resX, resY)#trasformazione di partenza # riproietta le coordinate w, e, s, n = extent w, s = GIS.Convert_coordinates(w, s, src_crs, t_crs) e, n = GIS.Convert_coordinates(e, n, src_crs, t_crs) extent_reproj = (w, e, s, n) minX, maxX, minY, maxY = extent_reproj x = np.linspace(minX, maxX, width) y = np.linspace(minY, maxY, height) resX = (x[-1] - x[0]) / width resY = (y[-1] - y[0]) / height transform_t = from_origin(x[0] - resX / 2, y[-1] + resY / 2, resX, resY) #trasformazione di arrivo # Prepara l'array di output img = img.transpose(2, 0, 1) # Riproietta l'immagine img_reprojected = np.empty(shape=(bands, height, width), dtype=img.dtype) for i in range(bands): # Itera sui canali dell'immagine (per esempio, RGB) reproject( source=img[i, :, :], destination=img_reprojected[i, :, :], src_transform=transform, src_crs=src_crs, dst_transform=transform_t, dst_crs=t_crs, resampling=resampling) #restituisci l'immagine e l'estensione riproiettati return img_reprojected.transpose(1, 2, 0), extent_reproj I hope you enjoy this functions, let me know if you have any doubt or wheter I committed any error. Best regards, Michele Zucchelli. Il giorno gio 25 gen 2024 alle ore 17:54 Michele Zucchelli < ***@***.***> ha scritto:

…

Ok, tomorrow I ' ll do it Il gio 25 gen 2024, 17:16 Martin Fleischmann ***@***.***> ha scritto: > We are surely interested! Ideally, open a pull request within this > repository. Alternatively, you can point us to the code uploaded somewhere > if you're not comfortable with PRs. > > — > Reply to this email directly, view it on GitHub > <#231 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A55OONDZ2XO5ECAHRRPFZ6TYQKAO7AVCNFSM6AAAAABBYUTLKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGUZTIMRZGQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >