regarding sprixels being planes and things like that

[joseluis]

following up from #1446 & related to #1388...

Before start talking about design, since I can't say I've really followed the latest API developments, could you tell me what would be the current simple way of outputting an array of pixels as sixels? So maybe I can start playing with it a bit too without much confusion at first.

[dankamongmen]

you could either (1) create an image file with such an array, load it with ncvisual_from_file(), and render it with ncvisual_render() using NCBLIT_PIXEL, or (2) create an ncvisual from memory using ncvisual_from_rgba(), and do the same. either way, you'll first need to call notcurses_check_pixel_support() and verify a return greater than 0. this last step is admittedly annoying; i can explain why i went this way in a bit (have a meeting now).

[joseluis]

Ok I was finally able to show some sixels. I had trouble compiling again and had to do it without media support. I wasn't sure if NcVisual was going to work, but it does.

I'm gonna try to finish at least in part the bindings for NcVisual in rust. NcVisualOptions is the most complicated options structure yet, because of the intermixed optional plane argument and the optional horizontal aligment flag, which affects how to interpret the other parameters...

Question: I don't get why we have to pass rowstride in ncvisual_from_rgba. Isn't it going to be always cols * 4 ?

[joseluis]

Another question: in ncvisual_render it says: "In this case, 'style' must be NCSTYLE_NONE". But I believe that's a typo and it should say "In this case, 'scaling' must be NSCALE_NONE".

[dankamongmen]

the restriction regarding NCSTYLE_NONE is just flat-out wrong or at least obsolete. removing, thanks!

[dankamongmen]

regarding rowstride, no, it is common in graphics libraries to pad rows; we need this to interoperate with at least ffmpeg without some manner of row consolidation.

i also needed this, for instance, to pipe nesticle's renderer into ncvisual_from_bgra().

as for why this is done, i imagine things like making it easier for parallel threads to operate, or cache benefits, https://en.wikipedia.org/wiki/Stride_of_an_array

[dankamongmen]

alright so here are my approximate thoughts on the subject:

• backwards compatibility suggests keeping them as ncplanes and using ncvisual_render() with NCBLIT_PIXEL. since the pre-1.0 days, i've been touting NCBLIT_PIXEL (well, i think it was NCBLIT_SIXEL once upon a time). you could have been using it since late 2019 and just decaying back naturally, waiting for it to finally appear. the ncvisual concept has proven pretty successful IMHO, and it would be a shame to depart from it. with that said...
• sprixels are pretty different, both in artificial semantics we desire (can't print directly on them, etc.) and reality (they must be printed as a whole, they have weird replacement semantics, they're wildly different sizes than any other blitter when NCSCALE_NONE is used, etc.). with that said...
• maybe letting people print onto sprixel planes is a reasonable thing to do? we could certainly assign it meaningful semantics, it would just be a big pain in the ass. probably a rich source of bugs, and very likely hard to do fast. and after all...
• there are many actions which are valid on sprixel planes (lateral move, depth move). of course...
• many actions are kinda confusing -- would ncplane_resize() stretch the sprixel to go along with the plane? we don't do that for normal rendered visuals -- once blitted, they're a matrix of cells, not treated as an image, and this is by design. of course...
• we can very well have a plane bigger than the sprixel it contains, just more pain in the ass. though we cannot have a plane smaller than the sprixel it contains, and i don't much care for that asymmetry.
• if we don't make them different types, and we don't define meaningful in-plane semantics, we have to have guards in every function that writes to a plane, to check for a sprixel. even worse...
• this makes for a very confusing situation regarding the case where we provide a preexisting plane to ncvisual_render(NCBLIT_PIXEL). do we check to ensure there are no non-default cells (yuck)? do we resize the plane to match the sprixel? we have a precedent with the other blitters -- depending on scaling, you either fit to the plane, or only print that part of the visual which fits.
  • if we don't allow bring-your-own-plane, we'd need a way to pass the size of the new plane, right? except we don't do this for regular blitting, so maybe not -- there, the new plane case is always based on the size of the render, or of the parent plane, and that hasn't caused us problems. but there we allow bring-your-own-plane, so that's kinda moot. i think we have to support scale+render, since we do for the other blitters, which means either specifying a new plane's geometry, or being able to provide a plane. we're never going to get rid of the latter for other blitters, so i think it's best we support it for sprixels.
  • maybe we just say "if you pass a plane for ncvisual_blit(NCBLIT_PIXEL), it's gonna get wiped"?
  • or maybe we do allow the plane to have other stuff on it?

i think this comes down to the question: do we want to support other crap on a sprixel plane? and now that i've written most of the rendering/rasterizing code around this, i think that's a much easier proposition than i'd originally anticipated. basically, everywhere along that path that we check to see if an intersecting plane has a sprixel at all, we'd instead want to look at the single intersecting cell and see if cell_sprixel_p() is true there. even blocking writes over a sprixel isn't hard, since all the cells are clearly identified as such. it's no harder than a plane-wide guard, at least.

[dankamongmen]

Ok now I undestand. I have to have it already prepared with the chosen colors before creating the ncvisual.

or you can just let notcurses quantize it for you, which i have put significant effort into, and think i can improve further still. do you have an example image that is unacceptably quantized?

Question: What do I check in order to know the number of color registers? notcurses_palette_size? And how to detect if a terminal uses raw RGBA like kitty?

i do not currently export these, and would rather not.

[joseluis]

or you can just let notcurses quantize it for you, which i have put significant effort into, and think i can improve further still. do you have an example image that is unacceptably quantized?

It's not that. The thing is I want to render pixel-art with a limited palette that I can be in control of.

Question: What do I check in order to know the number of color registers? notcurses_palette_size? And how to detect if a terminal uses raw RGBA like kitty?

i do not currently export these, and would rather not.

so how could I know the maximum number of colors that I can use in a pixel image?

[dankamongmen]

so how could I know the maximum number of colors that I can use in a pixel image?

i guess i'd need to export it, bleh. and by that logic i'd probably want to export maxy/maxx as well.

the problem is that these are all implementation details that i'm trying hard to abstract away. for instance, it's possible that i could cut up a sixel to get multiple palettes, and thus provide more colors than are available for a single sixel. i see what you're saying, though. let me ponder it.

[dankamongmen]

btw, details of how sixel works mean that you're not necessarily going to get this level of control anyway without more work. are you assuming that if, say, there are 256 color registers, you can perfectly reproduce 256 different 8bpc RGB values? no sir; sixel works on a [0..100] range for RGB. so if you had FFFFFF and FEFEFE, guess what, those are going to both be mapped to 100,100,100 (and thus be, and count as, a single color).

[joseluis]

I believe this sixel color topic merits it's own discussion. Let's continue in #1512

[dankamongmen]

you commented somewhere that the comment on ncvisual_from_rgba() was hard to understand; i have (hopefully) improved it.

[dankamongmen]

i'm thinking the thing to do is this:

• only blit sprixels to (a) planes created within the ncvisual_render() call (i.e. n = NULL in the ncvisual_options) and (b) planes which already have sprixels. note that this implies we can never ensprixelate the standard plane.
• probably go through and disable most output operations on a sprixel plane. maybe we allow them sometime in the future. i think they can only be confusing, and we won't.

but they won't be different types, because it's too useful to have them both as ncvisuals.

sure do wish C had a real type system, le sigh.

[joseluis]

With those imposed limitations, is still going to be possible to have piles of sprixel planes with their relative z-index ? ... I still have to make more pixel tests, but I wouldn't like for them to become too limited.

[dankamongmen]

you ought definitely be able to stack bitmaps, though at the moment their relative z-order isn't honored (this is a bug, and will be fixed: #1501). in Sixel, you will only be able to see the topmost non-transparent pixel. in Kitty, you'll be able to get blending with a non-opaque alpha value. it is possible that in the future i will implement synthesized blending (i.e. if you have a single ncvisual atop another, with the one on top yellow with 50% alpha, and the one below blue with 100% alpha, i will synthesize a new bitmap that is yellow or whatever), but it's not yet on the roadmap (and might never be, since this could get very messy).

i'm unsure what led you to think multiple bitmaps would not be supported -- perhaps something i said wasn't clear? i should probably add an example of this to notcurses-demo.

[joseluis]

you ought definitely be able to stack bitmaps, though at the moment their relative z-order isn't honored (...) in Sixel, you will only be able to see the topmost non-transparent pixel. in Kitty, you'll be able to get blending with a non-opaque alpha value

ah, good to know

i'm unsure what led you to think multiple bitmaps would not be supported

Oh, I thought you were talking about disabling most plane operations... I didn't realize that by "output" you were just probably referring to the text related ones.

only blit sprixels to (a) planes created within the ncvisual_render() call (i.e. n = NULL in the ncvisual_options) and (b) planes which already have sprixels.

Right now the only way I know to have scaling is by creating a bigger plane and referring to it in the ncvisual_options (if no ncplane is provided, one will be created using the exact size necessary to render the source with perfect fidelity). Is that API going to change? Also, I don't really undestand what b) means, could expand a little bit more?

[dankamongmen]

Oh, I thought you were talking about disabling most plane operations... I didn't realize that by "output" you were just probably referring to the text related ones.

yep, i only mean those which draw onto the plane.

Right now the only way I know to have scaling is by creating a bigger plane and referring to it in the ncvisual_options (if no ncplane is provided, one will be created using the exact size necessary to render the source with perfect fidelity).

you ought also be able to call ncvisual_resize() directly.

Is that API going to change? Also, I don't really undestand what b) means, could expand a little bit more?

i just mean that you can blit bitmaps to planes to which you have already blitted a bitmap, killing the old one. but it looks like this is going to be loosened, and you'll be able to blit a bitmap to any plane (that's how things are right now, anyway) except the standard plane (i think you can do this now, but i don't want you to be able to).

any restrictions are going to be viewed first through the lens of "can this restriction be efficiently enforced", so anything that requires checking every cell of a plane is likely a non-starter. with that said, my current model is "if you blit a sprixel to a plane, any existing cellwork, and any cellwork you might apply, will be nonfunctional" which is admittedly a terrible, unintuitive decision. i will likely reject cellwork on a sprixel plane because that's trivially easy to check (the issue there is just checking it at all entry points), but rejecting a sprixel on a plane which already has some cellwork is unlikely to happen.

[joseluis]

Another question: Is it possible to show the same pixel buffer in multiple planes? On the tests I've done with a shared buffer, once I render the new ncvisual the old one dissappears...

[dankamongmen]

that sounds like a bug (very possible, as i don't believe i've tested this case). you're saying that if you prepare a single ncvisual,and then call ncvisual_render() with NCBLIT_PIXEL twice, creating two diffrerent planes, only one shows the bitmap? that's definitely a bug if so.

[joseluis]

take a look if you can at the pixel-cell example, I updated it with this behaviour.

[dankamongmen]

will do, thanks. if you're pretty sure this is what's going on, if you could just file a bug on it now with a pointer to this, that would be appreciated -- i've got a lot going on right now and want to make sure i don't forget about this.