Home

Neural Network for Digital Twin (NNDT)

Tutorials

1. Space model and generators
2. Trainable tasks

Developer documentation

Early vision

NNDT is a framework for physically informed neural networks and implicit neural representation of 3D objects. The main scope of this code is computational anatomy and computational physiology. We try to implement as many as possible methods from these fields in a neural-powered way. The framework includes basic building blocks, such as data loaders, layers, neural network models, as well as, end-user inference tools. We expect undergraduate students and junior researchers as early adopters of this project. For this reason, it should provide an easy-to-start experience. Also, we expect Ph.D. students, postdocs, and industry researchers as the main target audience for this software. For this reason, a solution must include the most prominent and advanced approach in the target scope.

Maintainers and contributors

The basic code is based on Vladislav Dordiuk's research notebooks, which were reevaluated and rewritten as a single project by Konstantin Ushenin. Small improvements, quality of code, and CI/CD are being supported by Maksim Dzhigil.

Software architecture

Figure 1. The software architecture. Stars mark files that is not implemented in v0.0.1.

External dependencies:

• anytree helps to implement tree-based data structures
• vtk is a most popular library for scientific and medical visualization of 3D and 2D objects
• haiku and jax are libraries for deep learning from Google DeepMind
• matplotlib is a data visualization library

Building blocks:

• space_model include classes and functions for the 3D scene representation. These keep data consistent and provide functions to access data.
• generators include out-of-the-box data generators for NN training
• layers include classes that are inherited from haiku.Modules
• trainable_task includes wrappers over Haiku transformations. It joins together NN hyperparameters, data tensors, and functions
• visualization and controllers are lightweight NN training frameworks with data visualization capabilities
• baselines is baseline solvers for a certain type of data

End-user code:

• demos_notebooks is a collection of Jupiter/Collab tutorials
• demos_preliminary and demos are long training tasks
• bash_tools are command line tools for NNDT

Glossary

• NN is a neural network.
• SDF is a signed distance function.
• SDT is a signed distance tensor. Wikipedia and some articles name this as the signed distance function field (SDFF). We name this SDT to avoid double FF on the end and to clearly diverge function F(x,y,z,t)=f from the discrete 3D array f[i_x, i_y, i_z, i_t] = f.
• PS(ps) is a physical space. Physical space presents all values in real physical units without scaling or normalization. The correctness of objects in PS is not automatically protected in the space model. Only, the input file correctness keeps this.
• NS is a normalized space. All objects here are normalized to boost NN performance. Representers of the space model keep NS consistency when loading data.

Preliminary demos

Figure 2. The shape interpolation demo (sdf_multiple_files.py). first and forth shape are original, the second and third are generated by neural network.

Figure 3. The mesh segmentation demo (mesh_segmentation.py). The left image is expected segmentation, and the right is predicted by NN.