Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save user function output to the host output bytes. Does not use… #43

Closed
wants to merge 1 commit into from
Closed

save user function output to the host output bytes. Does not use… #43

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion template/src/index.ts.ejs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ export function <%- ex.name %>(): number {
<% } -%>
<% } else if (ex.output.type === 'string') { -%>
Host.outputString(output)
<% } else if (ex.output.type === 'number') { -%>
Host.outputBytes(Float32Array.of(output).buffer)
Comment on lines +53 to +54
Copy link
Contributor

@bhelx bhelx Dec 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! I think there is a misunderstanding of what this is doing though and this isn't something we generally support. At the boundary level (inputs and outputs), every piece of data that goes across needs an encoding. we can't really pass bytes across purely based on types. it might seem like a contradiction given the case above checks for type === 'string', but that's a backward compat thing and will eventually be changed to something like isUtf8Encoded(ex.output).

So to pass a float, what you need to do here is either choose an existing encoding or we create a new encoding. Two possible encoding you could choose would be either json or raw binary.

example w/ contentType application/json:

// note: precision is not really limited by the JSON format but rather by the language
> JSON.stringify(3.1415926535)
'3.1415926535'
> JSON.parse('3.1415926535')
3.1415926535

example w/ contentType = application/x-binary

// note: your precision can be 32 or 64 bit depending on your need
> new Uint8Array((new Float32Array([3.145926])).buffer)
Uint8Array(4) [ 218, 86, 73, 64 ]
> new Float32Array(z.buffer)[0];
3.1459264755249023

Copy link
Author

@dp-mason dp-mason Dec 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed explanation! So, just to be clear, the way my schema is currently set up, the plugin constructed from the default bundle throws an error. If I am using application/x-binary with number an float as my type and format, then I will be expected to implement that interface myself. If I am using application/json or another potential standard encoding, the default bundle should be expected to produce a functioning plugin out of the box if the encoding is supported and there is valid stub code. Is my understanding accurate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am using application/x-binary with number an float as my type and format

Sorry if it's not clear yet, those shouldn't be valid combinations. number and json are valid combinations because json specifies how to encode and decode numbers. With application/x-binary, there isn't really a valid type other than buffer. we don't know how to map a float to a binary in a cross language way without a specified encoding. x-binary is effectively "no encoding" and it just gives you a raw buffer that you have to map to and from yourself.

So your options are:

  1. we can add support for another encoding. one that can maybe map these simple types with minimal effort across languages.
  2. you can use contentType: application/x-binary & type: buffer and map yourself
  3. you can fork this template and make up your own rules

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, gotcha, this make sense. My use case is kinda weird. I think I will make my own rules for now. If it helps, the motivation behind this combination is a desire to generate code that clearly indicates to the user that they are supposed to make a function that returns a float, but then behind the scenes I would like to extend my PDK in order to pass it back to the host in a buffer. I am assuming that the combination of contentType: application/x-binary & type: buffer would generate code that asks the user to return a buffer in their user-defined function.

It is still nice to also have a gateway to directly access the user-implemented function that returns the return val in an encoding like JSON for debug purposes, which I am not sure would automatically be generated for me if I used type: buffer

I think I've figured out how to do what I need, open to trade some more ideas if it seems like there is more to discuss here.

<% } else { -%>
Host.outputBytes(output)
<% } -%>
Expand All @@ -59,4 +61,3 @@ export function <%- ex.name %>(): number {
}

<% }) %>

Loading