OpenStreetMap 🗺 3D renderer with three-d

[DerKarlos]

Most of you will know www.openstreetmap.org already. I do a lot of rendering the OSM data in 3D.

First I used ThreeJS for a plain viewer:

Next I used BabylonJS for a more realistic web front-end:

But as I like WebAssembly (more theoretically) I intend to use three-d now. Most functions I need seem to be there. Otherwise I will ask. The three-d API looks quite familiar:
🔘 User defined meshes
🔘 PBRMaterial = pong material
🔘 Water and Woods I already could see in the example "Spider game" (I can't find the sources)
🔘 Camera controls are customising, I assume.

There will be questions! The tools are always a challenge to me.
I wouldn't mind if one of you would join my project, for test or "consulting" me as a newbie.

If you are interested, I will describe the project structure and components, the intended functions and solutions in all details.
And you will get me as a tester for three-d

More about me: 👴 🇩🇪 Professional test engineer, now retired.
I use 🍎 "macOS-Linux" and Atom as IDE.
About Rust I only read some documents and made some tutorials. That will be an adventure to me.
www.twitter.org/osm__go
https://wiki.openstreetmap.org/wiki/User:-karlos- (a bit outdated)

++++++

Project overview:
There is a server doing the first steps: Get the OSM data Planet file (or Countries),
split it in 3x3 km tiles, create 3D objects (buildings, areas, ...) into a PBF file.
The frontend, using three-d, will request this tiles, decode them and create meshes.
It is like a infinite game level. The user may move or fly around on the virtual Earth,
new areas/tiles with real OSM data get visible inside a sky dome or out of the fog.

Project details:
The PBF file contains a material list with references to PBRMaterial textures.
There is a texture- and a materials-cash. Each PBF file brings its own list of materials.
The 3D objects are either triangle geometries or shape extrudes or instances of models.
There are also car models (obj) and rotating parts (windmills); may be trains to.
The 3x3 km PBF tile is split in 400x400 m tiles and different LoD layers.
Each layer as one mesh with Material groups is loaded into the GPU.
The tiles in the view focus will be loaded first, threes after the buildings.
The loading of additional tiles must not disturb the render cycle!
do be refined ...

[asny]

Hi @DerKarlos. Sounds like a really interesting project 👍 I will be happy to help if you have questions or experience problems with three-d and I will appreciate any feedback. The source for the spider game is not publicly available at the moment, but you can take a look at the forest example. Custom meshes and custom camera controls are supported while PBR materials is a recent addition so there might still be changes to that part.

[DerKarlos]

Hi @asny, thank you for your motivating replay.
Above an overview of the project is added.
I asked for the Spider to see how the trees are made. OSM2World needs a lot of trees with different LoD. We may brainstorm about this later.
So PBR is not the same as Pong but you are on the way. There is no hurry.

Meanwhile I got back into Rust a bit, made Hello Rust and Hello Cargo.
The Rust WASM book first example did not work. But this did:
https://developer.mozilla.org/en-US/docs/WebAssembly/Rust_to_wasm
Doing your triangle example this way did not work:
error[E0425]: cannot find function set_panic_hook in this scope
But I cloned your three-d and could build and run it native on my Mac Air and in Firefox :-)
Could you provide an example how to use three-d in other projects?
(Sorry, I am bad with all this tools and frameworks)
You may have looks in the project, I started:
https://github.com/DerKarlos/O2W-three-d-Rust-WASM

There is some new feature "Mesh Material groups" I will need, and write a request for.

[asny]

No problem 👍

I asked for the Spider to see how the trees are made.

See the "forest" example.

So PBR is not the same as Pong but you are on the way. There is no hurry.

You mean Phong? Phong shading is supported, has been for a while.

error[E0425]: cannot find function set_panic_hook in this scope

Maybe you need to add console_error_panic_hook crate? Please copy the dependencies in the Cargo.toml file also.

Could you provide an example how to use three-d in other projects?

The examples are stand-alone applications using three-d

[DerKarlos]

Just saw my "first light" of the Rust viewer! Prommising.

[DerKarlos]

Could you give me an hint how to activate transparency / alpha?
I am using your Texture2D, render_with_texture and Tree1Leave.png
There is a format: Format, but must I set it? How?

And SVG needs to be converted to bitmap first.
Do you have a recommendation for a cradle?

[asny]

You need to set the "blend" parameter in RenderStates (an input to render_with_texture) to Some(BlendParameters::TRANSPARENCY). I have no idea about SVG conversion.

[DerKarlos]

This works for cargo build (binary) but not for wasm-pack build --target web
Shall I write an Issue? Do you need my code?
(I will invite you to my repository anyway)

[DerKarlos]

Most things are running well now. But for SVG I have a question:
I use resvg to create a pixmap from my SVG.
I can write it to a PNG and it's ok.
But how to use as / put it into a CPUTexture?
This results in a black only texture:

    _cpu_texture = CPUTexture{
        data:   pixmap.data().to_vec(),
        width:  pixmap.width(),
        height: pixmap.height(),
        // format: Format::RGBA,
        ..Default::default()
    };

I found your image_from_bytes but could not se, how to use it.

[asny]

This is all you need to do. Maybe check your pixmap data and check that there are no webgl errors by looking at the console in a browser.

[DerKarlos]

I don't test in wasm yet, only binary. And there are no messages or errors.
I don't know how to check the data.
But if you are to bussy, I know other Rust user to ask.

[DerKarlos]

Hey @ASYN, by asking and requesting and issuing, I certainly disturb you working on other parts of three-d.
There is no hurry. If you just give me some "later", I will continue by doing other parts of my project.
If I could help, please tell me. May be I could do some three-d documentation from the point of a user.

You know, OpenStreetMap is a big thing but only voluntary, not by money. And me to: just a hobby.
But still: do you have some way for donations?

If my requests are just to much for you, do you know similar projects / cradles, also enabeling Rust and 3D by WebGL?

[asny]

Don't worry, I will respond, but it might take some time. I generally don't have a lot of time, but right now I am especially busy, hopefully it will be better soon 💪 Documentation is certainly always appreciated, and also testing and giving feedback as you are already doing. I don't have any way of donations right now, but I would also rather just do it as a hobby, at least for now. And I don't think your requests are too much, at least it seems like you have all the features you need right now.

[DerKarlos]

It's getting interesting: First colors

[DerKarlos]

Yes, texture_2d works, thank you!
I had to learn about Reference Counted and HashMap, uh!
std::rc::Rc::new(texture_2d) is before the map, and afterwards:
rc = map.get(url).unwrap().clone();
The map speeds up the load about twice.

I first use Loader::load for all textures and afterwards Texture2D::new and both takes time:
Could we interlace both? While the CPU is loading more textures from the web by DMA, the first textures could be send to the GPU.

I used cpu_material to set the texture AND the color. ("multiplication")
The texture_2d must not have a base-color, a mesh should not have it.
ColorSource or material could?

[asny]

Yeah, that would be nice but might be quite difficult to do. I think it is better to simplify how loading works and then start one loading for each texture. That means there is one callback for each texture and you transfer that texture to the GPU just when it is loaded. Probably requires the RefCell trick though.

[DerKarlos]

By building with --release, magically the call Texture2D::new runs quick like hell.
So only the many textures Loader::load in the CPU needs time.
And as I see in the WASM browser version, all textures are loaded NOT in parallel.
An option for ::load would be nice. Else I will use many ::load. In this case, I may try to call Texture2D::new in each callback.
Loading textures has always always the main app start delay. I consider to show the materials with color only
and update the Material later when the texture is loaded. That may need your help some day.

[DerKarlos]

How / Is it possible to load a texture (or other files) while the render cycle is running?
In my App, a user may move the camera of the rendered area. New "tiles", extending the area should be loaded.

[asny]

I will try to make loading more simple, it was already on my todo list. It is difficult though because how web (async) and rust (borrow checker) works but should be doable with RefCell.

[DerKarlos]

It almost works? Could you give me a hint please

main:
let rc_refcell_option_loaded: Rc<RefCell<Option<Loaded>>> = Rc::new(RefCell::new(None));
let clone1 = rc_refcell_option_loaded.clone();
let clone2 = rc_refcell_option_loaded.clone();
loaded:
let mut refmut = clone1.borrow_mut();
refmut.replace(loaded);
render_loop:
let refmut = clone2.borrow_mut();
let option = &*refmut; // What???  &*
match option { 
    Some(loaded) => {
        let bytes = loaded.get_bytes("./src/test.txt");
                    ^^^^^^ `loaded` is a `&` reference, so the data it refers to cannot be borrowed as mutable

[asny]

You probably just need &mut* instead of &* but it can also be much simpler. You don't need clone2 and you could use if let Some(ref mut loaded) = rc_refcell_option_loaded.borrow_mut() {...} In the render loop. Might need a * somewhere, i haven't tested it.

[DerKarlos]

Yes! &mut * did it. Rust isn't a programming language, it's a syntax disaster ;-) I need to talk about this kind of madness in person.
If I do understand Rc, I can't remove clone2. And the compiler agrees:
window.render_loop(move |frame_input: FrameInput| // Begin a new frame with an updated frame input
^^^^^^^ value used here after move
Anyway, this enables me to sort the functions as I am used to in TypeScript. Thanks again.

Yes again: The whole Rc madness can be "hidden" in my class. In the new, both clones are let. One is used by the loader closure, one stored in the class to be used by the check, called by render cycle.

Well, now I may load more areas/data if needed. But as Loader::Load is syncrounous, the render cycle freezes. That may be an Issue for later.

[asny]

I'm currently trying to do such a struct so you don't need Rc and RefCell and so on. It is working but I will have to try to make a nicer interface.
Async loading is added to the list, but it is quite difficult to do if you want the same code for desktop and web.

[DerKarlos]

I will try it. RC is cheating the ownership of Rust ;-)

Meanwhile I added (many) instances in my project. There is no Z-Order inside of the Wood-Mesh :-(

https://osmgo.org/wasm/zorder.png

And most times, you can run my test there: http://osmgo.org/o2t/index.html

[asny]

I think it is because the transparent pixels of the quad is still writing to the depth buffer. It can be fixed somewhat by not writing to the depth buffer if that pixel is transparent, but need to implement alpha to coverage to fix it properly. Are you using the Imposters? Because that should apply the first fix and it should look much better (but not perfect).

[DerKarlos]

Imposters?..!: No and yes. My trees are already textures (with alpha), once made manually by "screenshots" of a 3D trees.
So it's simply 2 crossed "rectangles" (2 treeangles) with backside visible.
But all of them are merged into 4* type-CPUMesh! With ThreeJS & Babylon, I had to add settings for z-order inside a mesh.
I consider to try your InstancedMesh, but assume, the many (>22000*4) instances will need more render time
because I have to call render_with_lighting for each of them!

[DerKarlos]

If I add depth_test: DepthTestType::Always or NotEqual, the "crossing" transparency is ok, but not the z-order of the tree positions. In examples/forest, LessOrEqual is used, an fine because there is no "crossing"

[DerKarlos]

To get the right z order and good looking trees in Babylon.JS version, I have to set
tree.transparencyMode = BABYLON.Material.MATERIAL_ALPHATEST;

[DerKarlos]

I just made a speed test with --release and the CPU =texture=> CPU war really fast :-)
Just some more motivation how useful three-d will be:

[DerKarlos]

Before, by adding my ExtrutionGeometries, a lot more details got visible, like railway beams or bus stops and wast baskeds.
You also see the red roof is working now :-)

Now some cars are visible. A OBJ/MTL model is loaded, multiplied 5 times and the carrosserie color is changed. Many instances of each color may get visible, if the OSM data contain parking grounds.
I started with three-d texture example and a car was visible soon. But ...

The Order of the Meshes and the Materials were not sorted according to the names. I edited my MTL as workaround.

The model includes 8 Meshes. There is no way in three-d to handle they as one 3D object.
A merge is not possible without "Color Groups"
There is no parent-child concept: the transformation of child and parent are multiplied. May be I will write some code later. Now I just do it with loops.

I (Rust beginner) could not manage to clone the CPU_Material to set different albedos, so I instanced Material mutiple times.

From the example I used the pipeline without really knowing what that means. And I was happy as it also works with InstancedMeshes.
So now I have only one InstancedMesh per Material: Mesh 1 to 7 and Mesh 0 5 times for the colors.
The cars should not get rendered if they are far from the camera. I could do that in the CPU code. Is there a way to let the GPU do it?

It was brain twisting to get my code in a "class". I would like to talk about it with you some day.
I could not manage to create the constant list of instance tuples before but inside the render cycle. Rust or Three-D?

All this was done in my car example. But as I merged it into the main code, there were differences in the three-d API!
You will have updated GitHub. The example got the new version but my main did not update! How can I initiate this? I had to create a new project, got get it running.

[asny]

I'm back from vacation, so will try to answer all of your questions, but please ask again if I miss some of them.

When loading meshes and materials, the meshes should have a material_name field that corresponds to the name field in one of the materials, so this is how you find the correct material for the mesh.

Loops are fine for rendering several meshes that belong together, you can group it in a struct yourself if you want.

The cars should not get rendered if they are far from the camera. I could do that in the CPU code. Is there a way to let the GPU do it?

You probably need to adjust the far plane of the camera.

I could not manage to create the constant list of instance tuples before but inside the render cycle. Rust or Three-D?

Probably rust.

[DerKarlos]

Ok, names, no OBJ Load "sorting as a service" :)

The far plane would hide all meshes. I do a "Level of Detail" concept, so different things need to hide or get replaced by simpler meshes in different distances. As there seems no CPU/GL function, I have to do it in the CPU cyclically for many objects.

Still reading the Rust doc ...

[asny]

No, several meshes can use the same material so I don't know how that sorting should be and it is easy to find the correct material by looping through the material list and check for the correct name.

Arh, now I think I understand, no you have to do that on the CPU but the CPU is also more suited for logical things like that.

[DerKarlos]

Your GitHub uploads are ok; I like to get new functions like the albedo color-texture mix, even as I have to adapt my code quiet much.

This time, the camera pan control was gone. Now I use the FlyControl and will make my own later.

My code, to Tc textures did not work any more. I had a view in your code: Material creates Texture2D itselves and there seems to be a texture cache in three-d.
So I removed my cache. But the texture loading into the GPU is not cashed now. Do you have a hint, how to use your or my own cashe?
Surprisingly, If i build --release, all that CPU load time shrinks to almost nothing, dubious but great!.

One more thing: My OSM file is large and I often get timeout. Maybe reqwest::Url Options could be accesable to extend the time?

(Is it ok this way or should I write more Issues?)

[asny]

Building with --release makes a huge difference, so always test with --release if you want to test performance.

I don't understand about the texture cache, when you create a texture it is on the GPU and stays there as long as you keep that instance alive so if you create a material, just keep that material alive. If you want to reuse textures, then create it yourself and pass a RC to several materials, that will keep the texture alive.

Please make an issue about exposing reqwest options.

[DerKarlos]

Yes!
Before your last Git upload, my cashing code was newby-odd. I used Texture2D and Material.
After the upload it did not compile and I did not see the way. So I removed the cash and used CPUTexture and CPUMaterial.
Now i tried it again, the code is simpler and works. Thank you!

[asny]

Great and no problem 🙂

[DerKarlos]

Some of my URI start like "data:image/png;base64,iVBORw0KGg...
We may call it an inline-texture.
I can create an [u8] like the one I would get by loading an image file:
bytes = base64::decode(&uri[22..]).unwrap();
It should be use in an cpu_texture:
let cpu_texture = create_image(self.bytes);
cpu_material.albedo_texture = Some( cpu_texture );
I found some code in Three-d and I can compile this "create_image",
but I don't know how to call it.

#[cfg(feature = "image-io")]
fn create_image(bytes: &[u8]) -> CPUTexture {
use crate::core::*;
use image::DynamicImage;
let img = image::load_from_memory(bytes);

cpu_texture = CPUTexture {
    data: img.to_bytes(),
    width: img.width(),
    height: img.height(),
    format,
    ..Default::default()
}

}

Or is there a way to add the bytes to the "loaded"?

[DerKarlos]

Oh! I dad seen insert_bytes, but couldn't find it, thank you!

[DerKarlos]

Yes it works! the crade base64 converts the URI part to a byte array and by loaded.insert_bytes(data_url,bytes); it is added to the loaded files. Next I extract it by &loaded.image(data_url). This sequence of in and out seems a bid odd. May be, I just could functions of three-d to convert the bytes to a Texture2D?
Later I may add this data feature into three-d: If the url/uri starts with "data:" three-d should not load a file but directly use the base64.

[DerKarlos]

If the uri ends with a ".svg", my code now does not call &loaded.image(texture_url) but &loaded.remove_bytes and converts the Scripted Vector Graphic by the crades usvg/resvg/tiny_skia to a PNG in a byte buffer. Again I use insert_bytes/image to convert it. See the screenshot with the trafic signs.
As with "data:" this could be a feature of three-d one day.

Now the odd part: the svg convertion uses crates using OS memory functions, not avaiable in WASM :-( That may be improved; not your business. On the other side, all web browsers are able to visualize SVG without previous convertion. At last inside a HTML page. I am not sure, if WASM or the Web.JS part of wasm-pack could use this. If yes, three-d could do it too, once up on a time ...

I will not write any requesting issues now. Firstly I want to understand an try how to branch and improve three-d.

[asny]

First, the reason why you should add the bytes to loaded (loaded.insert_bytes) before converting (loaded.image) is that a file (gltf, obj etc) might consist of several files. A 3d file usually consist of a main file, textures and binary data. To convert that into a model, we need for all of the resources to be loaded. However, a shorthand for converting an image could be added.

Also, the other requests for loading are definitely valid, so please make issues for them if you want them, but it might not be high priority.

[DerKarlos]

For OBJ I have to load the MDL file myself. GLTF is a zip and three-d will extract all files into the pool to show the model. If I have "data: URIs, and want to use the .obj or .gltf of three-d, of course I have to insert them.
A image shorthand will be great, thanks.
There are Issues #144 / #145 now. This is not prior. (#141 is still my top sorrow)
And I am happily looking forward to your Ambient-Occlusion merge. (#131)