Consider xtensor

[danielballan]

@n-log-n I'm writing this from SciPy, where it occurred to me that xtensor might be a great tool for connecting this codebase to numpy arrays. I'm not a C++ developer, so I'm not a position to personally evaluate it, but the developers are well known in the SciPy community and this project has a solid reputation. I'd be interested to know what you think.

[iamfaisalkhan]

@danielballan I looked at xtensor. Its going to be little complicated to adopt a new data format internally. Also, we already use something similar called Eigen that has python binding.

Eigen is probably a step lower than xtensor given that internally xtensor also use it. With Eigen we ran into issue where handling sparse matrices for multi-tau became an issue. The multi-tau averages out values at each level. At every averaging step, the Eigen copies the values to a new matrix given that most sparse implementations are read only. This ended up adding a huge overhead for us. Instead, I started using a custom format optimized for lookup.

It is probably better to initially start using the code directly through the C++ interfaces. Since the output is written to HDF5 directly, that can then be read into by python code.

[danielballan]

I am interested in getting the g2 result out of C++ without going through HDF5 serialization. It is often the case that additional steps, outside the scope of xpcs-eigen, such as nonlinear curve fitting, are necessary to complete the full calculation.

[sureshnaps]

Dan,
xpcs-eigen calculates g2 and g2 stderr and that is it. It also has the --g2out flag as an arg to make it save G2,IP and IF for each pixel. This can be used to make dynamic maps as the user wants and renormalize to calculate a new g2 and g2err.

By design, the fitting of the g2s is not part of this. This is handled in python in a separate code which I am happy to share, it is not on git but we should put it there anyway. This code can take some work as it was done by a student some time ago, it works but is dated.

I would say, first step, just run a file from CHX and get the results. I can help with matching the results or something else you need. If things are need with regard to this, feel free to write to me.

[danielballan]

By design, the fitting of the g2s is not part of this.

Right, I think I didn't make my point clearly. I agree it is good to keep xpcs-eigen at its current scope and let any additional analysis be done with separate libraries. I am hoping it can be made easier to get data in and out of xpcs-eigen without going through HDF5 serialization.

I would say, first step, just run a file from CHX and get the results.

See comment.