Support creating highlighting for Typst output document

[cderv]

This is different than the other issue which is about the highlighting syntax support for typst block

• Highlighting for typst #184

Here I am thinking it would be great to have Syntax Highlighting Pandoc's feature be supported for Typst output so that we could get the same highlighting and theme than in other format (e.g. setting --highlight-style pygments)

Opening the issue here to discuss as this is relevant to https://pandoc.org/MANUAL.html#syntax-highlighting provided by skylighting

It seems Typst uses its own logic though with

• Syntax based on sublime-syntax (https://typst.app/docs/reference/text/raw/#parameters-syntaxes)
• Theme based on thTheme (https://typst.app/docs/reference/text/raw/#parameters-theme)

So maybe it is matter of just provided same themes for typst or syntax closer to XML KDE (which would require a conversion to sublime-syntax file I guess).

Anyhow, opening for discussion after a first talk with @tarleb

[jgm]

What this is requesting, if I understand correctly, is a new package skylighting-format-typst that formats a skylighting token stream as typst. Is that right? That should be feasible.

[cderv]

What this is requesting, if I understand correctly, is a new package skylighting-format-typst that formats a skylighting token stream as typst. Is that right?

Yes exactly.

[jgm]

Now that I've implemented this, I am perplexed about how to integrate it into pandoc. The natural thing to do would be (as with other writers) having it trigger when optHighlightStyle is a Just value. When optHighlightStyle is Nothing, it would behave as it does now and just use the native typst format for highlighting. The problem I see is this: pandoc's default is to set optHighlightStyle to Just pygments. So pandoc would default to generating incredibly ugly and unidiomatic typst code blocks. Of course this could be defeated by setting --no-highlight on the command line, but people won't know to do this and it would nice for the default behavior to be idiomatic.

[cderv]

With LaTeX output, Pandoc will by default highlight code block using pygments and add custom command for \textcolor{} right ?

So we don't want the same behavior for typst ?

Or is it because Typst has it own highlighting system by default and it would be better for it to be the default ?

Just sharing some more thoughts here - I think I see this as :

Typst has its own highlighting system by default (https://typst.app/docs/reference/text/raw/) with a default theme (which can be customized at typst level using thTheme format). It would be best for Typst document produced with Pandoc to use this own system by default.

So maybe this means that different default could be used by format ? Is that something conceivable ?

I draw a parallel with the citation support, for which specific Typst consideration to use Pandoc feature. If --citeproc needs to be used, it requires -t typst-citations to avoid typst #cite() to be used.

So maybe this would mean a new flag or option / extensions needs to be added for selecting Highlighting System ?

• Default would be Skylighting Highlighting engine for most format
• Typst would default to Built-in Highligting engine, and Skylighting could be opt-in (and default to pygments)

This probably is quite similar to defaulting to --no-highlight for -t typst. Maybe just more explicit, and open new possibilities. I know format like Revealjs have also configurable highlighting engine (https://revealjs.com/code/ powered Highlight.js as a revealjs plugin), and having a way to opt-out skylighting and opt-in this built-in engine would offer flexibilies in output type produced by Pandoc.

[jgm]

With LaTeX output, Pandoc will by default highlight code block using pygments and add custom command for \textcolor{} right ?

So we don't want the same behavior for typst ?

Or is it because Typst has it own highlighting system by default and it would be better for it to be the default ?

That's the relevant difference. There's already a built-in, idiomatic way to highlight code in typst, which results in clean and editable source. I suspect most users who are generating typst that they'll edit would prefer this to pandoc-highlighted code. Pandoc-highlighted typst output would be useful mainly for people who are using pandoc to produce a pdf via typst.

Typst has its own highlighting system by default (https://typst.app/docs/reference/text/raw/) with a default theme (which can be customized at typst level using thTheme format). It would be best for Typst document produced with Pandoc to use this own system by default.

One option we could explore is providing a way to export a skylighting style in the thTheme format so it could be used with typst. I'm not sure if that's what you're suggesting here. But it looks like this has to be a separate file so it wouldn't give you a one-step conversion, and it wouldn't exactly match pandoc's tokenization since it wouldn't use skylighting for that.

Other options:

• We could set the default for everything no --no-highlighting, so that you have to explicitly use --highlight-style to get highlighting. This is what I'd do if I were starting afresh, but it would break existing behavior so it's not ideal.
• We could just accept messy default typst output with highlighted code.
• We could make the default highlighting setting depend on the output format (which is what you're proposing).

Maybe there are others? @tarleb @silby do you have thoughts about this?

[tarleb]

Thinking out loud: the problem is also at the heart of the --listings option for LaTeX output. Maybe we can somehow consolidate this by adding a highlighting option with possible values default|idiomatic|styled or similar. Then default could be format-dependent, and --listings becomes equivalent to --highlighting=idiomatic.

[jgm]

Or, --highlighting=default|idiomatic|styled|none, with --no-highlighting becoming a deprecated synonym for --highlighting=none and default being the default. I think I like this idea best. However, it would require API changes in WriterOptions.

[jgm]

Or instead of styled we could just put the style...and then a separate --highlight-style wolud not be needed.

[cderv]

by adding a highlighting option with possible values default|idiomatic|styled or similar.

I think this is a good idea too. This is similar to what I meant by adding a flag to select Highlighting System.

Or instead of styled we could just put the style...and then a separate --highlight-style wolud not be needed.

If I understand correctly proposal, --highlighting=default for HTML or PDF in current situation would be to use Skylighting. So --highlighting=default or --highlight=pygment would be equivalent right ?

Regarding Typst idiomatic support for .thTheme file, I think it could be interesting to support -t typst --highlighting=idiomatic --highlight-style=<user-provided-file-with-thTheme-extensions>. It would be passed to a template variable to set in the right place in Typst raw code.
Or not something interesting to support ?

[jgm]

The proposal is that default would by output-format dependent. So, default for HTML would by skylighting with pygments style, but for Typst it would be native typst format.

To summarize, the proposal would be:

In T.P.Options, replace writerHighlightStyle :: Maybe Style with something like writerHighlighting :: HighlightOption, where

data HighlightOption =
    Skylighting Style
  | DefaultHighlighting
  | NoHighlighting

In command line options, add --highlighting=none|default|<stylename>|<themepath>, and deprecate --highlight-style and --no-highlighting (but still support them for a while as synonyms).

About .thTheme, I wasn't proposing to have pandoc read .thTheme files. Rather, the idea was that pandoc or skylighting could be taught to produce a .thTheme file corresponding to a pandoc highlighting style. This would remove one motivation for using skylighting with Typst.

[jgm]

I've created an open pandoc issue to track this
jgm/pandoc#10525
Further comments should go there.