pico openxr sdk: Unable to load xr-extension function: xrGetEyeTrackingDataPICO, error-code: -7

[Jia-WeiFang]

Hi,
I'm using the pico neo 3 pro eye HMD, which supports eye tracking,
and I want to use the xrGetEyeTrackingDataPICO function declared in openxr_pico.h, which was the pico openxr sdk that was originally included in the project.

I have followed the same steps in openxr_program.cpp to initialize the function as below:

PFN_xrGetEyeTrackingDataPICO m_pfnXrGetEyeTrackingDataPICO = nullptr;

GetPicoInstanceProcAddr("xrGetEyeTrackingDataPICO", m_pfnXrGetEyeTrackingDataPICO);
(in the InitializeExtensions() function)

and use the function as:

m_pfnXrGetEyeTrackingDataPICO(m_session, eyeTrackingData);

However, when I run the program, I got this error message:

RustStdoutStderr: [14:06:04.479][Warning] Unable to load xr-extension function: xrGetEyeTrackingDataPICO, error-code: -7

"error-code: -7" means "XR_ERROR_FUNCTION_UNSUPPORTED"
It sounds like this function is not supported(?

Am I implementing the function wrong? Or Is there any way to use this function to get eye-tracking data?

Thanks very much!

[korejan]

@Jia-WeiFang Typically you would need to add the extension name to the list of extensions to enable. Looking at pico's header, most of their extensions are grouped with an extension name define but I do not see such defines for the eye tracking.

Pico currently have little to no documention on their OpenXR runtime and Pico's OpenXR sdk has not been updated since October 2021 (ignore the timestamp on the site, it's still the same version) but they have made updates to their runtime so it is possible that these function signatures are no longer valid, it maybe that the runtime now implements XR_EXT_eye_gaze_interaction multi-vendor extension or both even.

I would suggest that you run the debug version of alxr (preferrably with debuggable flags on in alxr-android-client/pico/Cargo.toml), ALXR will log all the extensions a particular runtime supports on start-up.

Another thing to note is that with android based OpenXR runtimes it is typical that extensions require uses_feature/permissions to be added to AndroidManifest.xml (which you can add in alxr-android-client/pico/Cargo.toml), sometimes even require user permission requests (check alxr-android-client\src\permissions.rs) before they become available/usuable. Again Pico do not documenet this so you may need to look at their Unity Plugin code to find out which AndroidManifest entries you need to add, they'll be a cs file for AndroidManifest.xml generation.

[korejan]

I have seen what you said that there is no such extension name defined for eye tracking, but I wonder which part of the scripts or maybe the HMD itself will determine what extensions can be used? It looks like you just add the extension name to the list by yourself, and use IsExtEnabled to check whether the extensions we want to use are in the list we just create?

Yes in OpenXR / Vulkan you can check if a particular extension is supported before trying to create device / instance with the list of extensions you want enabled. This is what m_availableSupportedExtMap map in alxr is for.

As you said, xrGetEyeTrackingDataPICO maybe not be available anymore, they now implement XR_EXT_eye_gaze_interaction, right? (Did I misunderstand you?)

Yes possibly, this is why I recommend you run the debug version and check the logs on start-up, alxr will print out the list of supported extensions by a particular runtime.

Is it necessary to add the permission in Cargo.toml? I can not find the permissions part for every extension that have already been used in the scripts, so I guess it is optional? (like if I didn't implement it yet, I can still use the function correctly?)

It depends on the extension, typically for stuff like eye / face tracking that usually requires permissions because of privacy concerns. As Pico do not docuement this properly all I can suggest is looking at the code of Pico's Unity plugin to find that out. If it does require it you need to do it otherwise it wont be available to use by alxr.

I think the error is because I did not add the permission part (?

What is most likely happening is that you're not using the structs of XR_EXT_eye_gaze_interaction but Pico's own structs and they are different. Also with this extension, on top of enabling the extension, possibly adding use_permission/feature for AndroidManifest.xml you need to check the result of XrSystemEyeGazeInteractionPropertiesEXT::supportsEyeGazeInteraction (look at the begining of InitializeHandTrackers to see simillar code).

Please note that Pico released a new version of their OpenXR SDK a few days, in the past their OpenXR runtime was not standard conforming, had bugs and unpublished extensions. They just recently passed conformance tests but the runtime update will only be availble for PUI 5.2.x, an update for Pico Neo 3 is aimed for release on 2022/12/15.

I have PN3 link, once I get this update I will be upgrading the Pico OpenXR SDK to this version, I would suggest for now wait until they this runtime update is released (it maybe a PUI update or seperate OpenXR runtime update), wait for me to upgrade the SDK. Then run the debug version and get the log on start-up which prints all the supported extensions.

[Jia-WeiFang]

Hi,
@korejan Thanks again for your reply!
Last week I have contacted the pico developer support team member, and they said OpenXR SDK 2.0.5 with any PUI version may work.

May I ask which Pico OpenXR SDK we used now in ALXR?

Thanks a lot,
Jia-Wei

[korejan]

@Jia-WeiFang There is no version 2.0.5 of either the Native or OpenXR SDK, only 2.0.1 and the newest one v2.1.0, here, A few months ago the link for 2.0.1 was updated (it was originally released on October 2021), they must have made a made mistake and re-uploaded 2.0.1...

ALXR is using 2.0.1 currently, I want to update to 2.1.0 but I can't unless I support 2 Pico builds because the PN3-Link model is still on PUI 4.8.0.0 and v2.1.0 sdk requires at least v5.2.x

[Jia-WeiFang]

Dear @korejan
Hi,
You are correct that there is no version 2.0.5 of OpenXR SDK. I think they mistake it as the unity one.

In the past few weeks, I have communicated with the pico engineer and they eventually gave me an eye tracking demo project and the runtime that support eye tracking in pico.

With the new runtime, I can use the eye-tracking function without updating the PUI version or the SDK version. (Since they said the new SDK and PUI will be released maybe in March, but I can not wait until that time so they gave the runtime for an alternative plan.)

But when I was implementing the eye-tracking code, I can not get the gaze position correctly.
There is no error message, but the position will always be the same.
I followed the flow that you track the hands and use the functions the demo project from pico has implemented in the client.

The code looks undoubtedly correct, so I wonder in the ALXR client, should it need to add some other codes that allow the client to get the gaze position? (because their demo project can work fine, so I wonder whether the problem is from ALXR?)
I implemented the getting-eye-position functions in RenderLayer function in openxr_program.cpp and the initialization parts in interaction_manager.h.

If you don't mind, maybe I can share my current implementation with you to check whether it is correct?

Thanks a lot for your kind help,
Jia-Wei

[korejan]

@Jia-WeiFang sorry I took so long to reply, If you can find me on discord you can contact me and if didn't know already I have since added support for using the latest version of Pico's OpenXR SDK.

[DUNSON288]

hello i want to stream remotely, but when the alxr client running, i can 't see the host name, so i can 't input the host name in the server,how to solve the problem?
Thanks a lot

[Jia-WeiFang]

Did your PC and hmd connect with the same wifi? (alvr and alxr only support streaming in LAN)
If they were, you may not need to input any host name in the server,
the server should find the client automatically; then you can start streaming.

[DUNSON288]

thank you， my PC and hmd are connect with different wifi