Highlight TableViewer cells with TM4E

[PhaserEditor2D]

Hi,

I need to create a tm4e JavaScript tokenizer to render the content of a TableViewer cell.

My tool uses Eclipse BlueSky (from @mickaelistria ) and the idea is to get the current TM theme associated with the JavaScript grammar, create a tokenizer and transform the result in an array of JFace StyleRange, then I can "syntax paint" the cell.

Any tip is welcome!

I attach here an image of the table:

[mickaelistria]

This is a very interesting and not so easy question ;)

Have you seen some occurences of other languages (like Java with JDT) which managed to reuse the syntax highlighting in other widgets than StyledText? If so, it could simply be a matter of mimicking their implementation, using the TMPresentationReconciler instead.
But to be honest, I'm not sure anyone ever did that. If I see something of that kind, I'll add a note here.

Another possibility is simply to find the right code from tm4e to add into the LabelProvider. It might be simple enough, but there are some risks of this taking too much time in a synchronous UI Thread operation, then causing freezes in the IDE.

[PhaserEditor2D]

I just did it! :)

I can go home now. Tomorrow I will work more on this. This is the important code:

private StyleRange[] parseCode(String line) {
	List<StyleRange> ranges = new ArrayList<>();

	ITokenizeLineResult result = _grammar.tokenizeLine(line);
	for (IToken token : result.getTokens()) {
		for (String scope : token.getScopes()) {
			org.eclipse.jface.text.rules.IToken themeToken = _theme.getToken(scope);
			if (themeToken != null) {
				Object data = themeToken.getData();
				if (data != null && data instanceof TextAttribute) {
					TextAttribute attr = (TextAttribute) data;
					if (attr.getForeground() != null || attr.getBackground() != null) {
						StyleRange range = new StyleRange(token.getStartIndex(),
								token.getEndIndex() - token.getStartIndex(), attr.getForeground(),
								attr.getBackground());
						ranges.add(range);
					}
				}
			}
		}
	}

	return ranges.toArray(new StyleRange[ranges.size()]);
}

[PhaserEditor2D]

Ok, I close this. Now it even updates when the user changes the theme.

but there are some risks of this taking too much time in a synchronous UI Thread operation, then causing freezes in the IDE.

I guess the time-consuming part is the parsing, but I will do this in the "search thread" and not in the UI thread.

Thanks!

[mickaelistria]

Thanks for sharing this success-story and the code that goes with it! I'm pretty sure it will be helpful to a bunch of others in the future.

[angelozerr]

Sorry @PhaserEditor2D I was very busy. I have seen that you find a solution.

It might be simple enough, but there are some risks of this taking too much time in a synchronous UI Thread operation, then causing freezes in the IDE.

Parsing one line, I think it's quick.

I can go home now. Tomorrow I will work more on this. This is the important code:

At first you can find this similar code on Markdown support https://github.com/eclipse/tm4e/blob/master/org.eclipse.tm4e.markdown/src/main/java/org/eclipse/tm4e/markdown/TMHTMLRenderer.java (I'm using it in typescript.java for hover typescript JSDoc which can use Markdown)

My fear is that I would like to support like vscode-textmate #38 and it will change the API, but it's a very hard task... But it will open the door on a lot of feature like get quickly token in an offset (required for language configuration like #159 and #141), highlighjt only visble view port, manage an hover (in debug mode kind) to see token info, etc...

[PhaserEditor2D]

Hi @angelozerr

Thanks for the answer. I will take a look to the HTML renderer, I need something like that to render the Phaser docs and in the examples of code, that is a local website.

All the changes you are planning to do are very welcome. In Phaser Editor we need to get token information because we show some fancy "text hover" with animations, images, and things like that. Actually, I plan to use Code Mining to show images next to the images keys.

[PhaserEditor2D]

Now we use TM4E to syntax coloring of the Phaser API docs:

At a certain point, we should implement it also for LSP4E docs.

[angelozerr]

You mean hover which is described with markdon syntax and github syntax for code like---xml. If it That i had done inside typescript ide. But it doesn t Work with lsp4e Le samedi 9 février 2019, Phaser Editor <[email protected]> a écrit :

…

Now we use TM4E to syntax coloring of the Phaser API docs: [image: image] <https://user-images.githubusercontent.com/12187299/52516124-82062980-2bf3-11e9-9b88-69c4c7c96ebf.png> At a certain point, we should implement it also for LSP4E docs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB17s9ulHDURBhJpyKlpRDqsl2cCoMP4ks5vLkT5gaJpZM4U2kBc> .

[PhaserEditor2D]

I should check TS IDE.

[angelozerr]

Ts ide uses the markdown support from tm4e. Imho lsp4e should give the capability to choose hover render.today it uses markdown support from mylin if i remember which dorent support syntax color for code Le samedi 9 février 2019, Phaser Editor <[email protected]> a écrit :

…

I should check TS IDE. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB17sxUrrevLLSYrfeoTXTSU7JAjWVVxks5vLwzWgaJpZM4U2kBc> .

[mickaelistria]

I don't think LSP4E should be at this level of details of making adopters chose their matkdown renderer. It seems overkill and far from the initial goal to work as best in all cases without need for specific configuration beyond binding the LS. But LSP4E could adopt another better markdown renderer if it exists and is compliant with licence. LSP4E is not committed to using Mylyn wikitext. LSP4E isn't neither supposed to be the project where to implement a markdown renderer and will only reuse existing ones from other projects. One possibility, probably the best for the overall Eclipse IDE offering would be to improve Mylyn Wikitext directly so that it could be extended by tm4e or others to render some advanced blocks.

…

-- Mickael Istria Eclipse IDE <https://www.eclipse.org/downloads/eclipse-packages/> developer, for Red Hat Developers <https://developers.redhat.com/>

[PhaserEditor2D]

If LSP4E uses mylyn for markdown then maybe the better is to implement syntax coloring i the mylyn project. Markdown has this format to say that a block is of certain language, so mylyn can parse it and color it, or maybe, create an extension point to load a third-party highlighter. So custom products can register highlighters based on TM4E or any other. In my case, I like to use the same TM4E theme for editors and documentation.