Use a templating system for the HTML code

[teolemon]

What

Extract all the HTML code from CGI files and put them in a separate template, using a standard templating system

Why

As a Frontend developper, I don't want to think of the underlying Perl code when I improve the OFF interface

Tasks

• HTML Template for Nutrition Score Calculation Details #3502
• HTML Templatization of Product page #3846
• NutritionScore Calculation Details HTML Template #3503
• Nutrition Facts Table templatization #3631
• Nutrient Levels Templatization #3728
• Search and display products templatization  #3802
• display_new() templatization #3845
• Search Form Templatization #3858
• User Form Templatization #3897
• Ingredient Analysis Details Templatization #3898
• Ingredients Analysis Templatization #3902
• Search Form Template #3976
• Top Translators using Template::Toolkit #3985
• Spellcheck test Templatization #3995
• Reset Password Templatization #4003
• Product Image Templatization #4021
• Templatization of display_recent_changes() in Display.pm #5530
• Templatization for display_list_of_tags_translate() in Display.pm #5468

Part of

• Refactor Product Opener #5527
• 🎨 Look & Feel - Design (Tracker) #5513

[stephanegigandet]

Note: doing this is quite complex. There is a lot of logic in the CGI that generate the HTML code. Most of the logic probably needs to remain there (otherwise if we move all the if / for loops etc. to the templating system, we won't gain much).

We need a way to start small. e.g. before going through the CGI scripts, it may be easier and more useful to first use a template for the code that generates the structure and navigation of all HTML pages, which is in Display.pm display_new() function.

[stephanegigandet]

We also need to discuss which template system to use. See also bug #2361 from @zigouras .
https://metacpan.org/pod/HTML::Template is a good option.

[hangy]

https://metacpan.org/pod/HTML::Template is a good option.

Other maintained candidates that I found during research include https://metacpan.org/pod/Text::Xslate and https://metacpan.org/pod/Template::Toolkit

[areeshatariq]

@stephanegigandet @hangy I would like to start working on this issue. Need your guidance on which template to use and which files to start with

[stephanegigandet]

@areeshatariq : @teolemon is suggesting to start with Made Near Me, which is separate from the main product opener code, and is already a bit templated: https://github.com/openfoodfacts/openfoodfacts-server/tree/master/madenearme
It would be a good thing to try to use HTML::Template or Template::Toolkit to generate it.

See https://madenear.me

The corresponding code is in /cgi/madenearme.pl

[areeshatariq]

@stephanegigandet I'll start working on it. Thanks

[areeshatariq]

The corresponding code is in /cgi/madenearme.pl

@stephanegigandet Can't find this file

Is it /scripts/generate_madenearme_page.pl ?

[CharlesNepote]

https://metacpan.org/pod/HTML::Template is a good option.

Other maintained candidates that I found during research include https://metacpan.org/pod/Text::Xslate and https://metacpan.org/pod/Template::Toolkit

I also love this one, with a great separation between design and logic : https://metacpan.org/pod/distribution/Template-Semantic/lib/Template/Semantic/Cookbook.pod
The html designer can design a real example of the resulting page, and the dev can create his logic on top of it.
And you don't have to learn a cryptic mini-templating-language: it's based on css selectors (or xpath), technologies already used in front end dev.

[stephanegigandet]

Is it /scripts/generate_madenearme_page.pl ?

Hi @areeshatariq , sorry for the late reply, you are right, it's this file.

[stephanegigandet]

The html designer can design a real example of the resulting page, and the dev can create his logic on top of it.

It's a bit weird to have the selectors in the Perl code though. A designer cannot deeply change the template just by looking at the template.

The Template::Semantic module last commit was in 2012 and there's no maintainer and no reply to any of the issues on github. I suggest we use a more known / used module.

[CharlesNepote]

The html designer can design a real example of the resulting page, and the dev can create his logic on top of it.

It's a bit weird to have the selectors in the Perl code though. A designer cannot deeply change the template just by looking at the template.

I think the contrary: the designer could be totally unaware of what logic exist or not on the backend; he just have to design the UI and tell what he wants in a user perspective. HTML ids help designer, javascript frontend and perl backend developpers to share common objects. This is one of the problem I'm facing in Power User Script dev: the lack of ids to create UI features without asking perl devs.

For example, for the moment there is no simple way to play with the user id. I had to write very bad code to handle it:

var editor_id = $(".side-nav > li > a").attr("href"); // /editor/charlesnepote
var user_id = (/\/(.*?)\/(.*)/).exec(editor_id)[2];

With Template-Semantic, the designer writes:
<h3 id="welcome">Hello <span id="user_id" data="charlesnepote">Charles Nepote</span></h3>

The frontend developer then can writes:
var editor_id = $("#user_id").attr("data");

And the backend dev fill the different values needed in the HTML content and attributes corresponding to each id.

The Template::Semantic module last commit was in 2012 and there's no maintainer and no reply to any of the issues on github. I suggest we use a more known / used module.

Of course, I understand it's a very important point, that probably disqualifies this solution.

[teolemon]

@stephanegigandet @hangy
In Django (python), the views.py passes all the variables likeso:

from django.template.loader import render_to_string
rendered = render_to_string('my_template.html', {'foo': 'bar'})

<html><head><title>{{foo}}</title></head></html>

The html can be fragmented in includable, reusable bits (product.html, header.html…). It should be as dumb as possible, with the objects, computations, and mapping happening in the view (eg the CGI)

[CharlesNepote]

How can we go further? Would it help if I start benchmarking templating systems?

[VaiTon]

How can we go further? Would it help if I start benchmarking templating systems?

Yup, you can start with the most popular one. HTML::Template. 😄

[stephanegigandet]

I looked at both HTML::Template and Template::Toolkit. I think Template::Toolkit would be easier to implement, as we can bind it more easily to the Perl code and data. It would be in particular very useful for the internationalization, as we can include subroutine calls to our lang() function. e.g. [% lang("some_field_description") %], without having to first declare all needed values from the Perl code. If the designer wants to use another string from the .po file, that's fine, there's no need to change the Perl code, just the template.

[CharlesNepote]

I've done some investigations also, from a front-end dev perspective, and want to share it here, but I need more time. Could you wait a little bit more?

[CharlesNepote]

Disclaimer: I'm not a professional dev and I don't pretend to suggest the best solution. I would only want to let you think about the impacts of the chosen templating system and the opportunities that we can open or not. Sorry for the long post (with probably many mistakes), at least if I'm dreaming of something that is impossible.

So, why would we like to use templates? I think it's important that we agree with the reasons why.

1. For all people contributing to OFF code, templating should be good because it can provide a clear separation between logic and presentation; it makes dev easier because, theoretically, when changing presentation, it shouldn't provide side effects on logic.
2. For everyone, concretely, having separate files for presentation, is a first very good step: front-end devs don't have to deal any more with the rich (and scarry) logic of perl code (that, probably, only @stephanegigandet and @hangy understand very well). Etc. I'm sure we all agree with the main benefits of templates.

Then, we should ask ourselves WHO will work on these templates.

1. It can be a few devs as @stephanegigandet, @hangy, @VaiTon that already understand how Product Opener works. It would be a good thing if templates could help them to better work on the code. Maybe they could have a little help from other people, but is it really a important/realistic goal? In this case I think they should make the templating system choice and I won't have any concerns about it -- and you can stop reading.
2. On the other hand, far more people could work on the presentation:

• amateur devs such as @teolemon and I
• professional devs which don't know Perl and Perl templating systems (such as @aleene, @raphael0202, and so on)
• and, last but not least, any developer with skills in front-end development: those are legions and the more templating is easier, the more we should be able to have people helping us.

Having a very easy-to-use templating system could lead to many use cases:

• allow school projects in front end dev
• allow highly skilled front-end devs to help: designers, JS devs...
• allow to organize open competitions and test live proposals
• allow to ask help from communities (stackoverflow...)
• and so on.

If we are interested in this use cases, we have to ask ourselves what is an easy-to-use templating system for all these cases.

In these cases, classical template engines has major downsides:

• the tags, which allow to present data from Perl logic in an HTML file like [% something %], creates many issues, from my point of view:
  • it is not rendering a realistic view in a browser
  • it might not render correct HTML
  • last, but not least, devs HAVE TO install a running instance of Product Opener (which takes at least hours, and is sometimes very hard; I have many examples of devs who abandoned due to that; it's probably the biggest barrier to contribute OFF dev)
• then each templating system has its own way to add logic [% IF blahblah %]; I found this very bad:
  • it is even more not rendering a realistic view in a browser
  • it's an invitation to start writing code soup
  • it throws away many devs if they don't already know how to deal with this logic

I know it can be hard to follow, but I would like to ask you all to (re-)consider pure HTML templates. It's not commonly used but there are many projects working with this idea.

Outside of Perl (you can directly jump to Perl section):

1. Plates, for Node.js and the browser
   https://github.com/flatiron/plates
   "Light-weight, logic-less, DSL-free, templates for all javascript environments! Plates (short for templates) binds data to markup. Plates has NO special syntax. It works in the browser and in Node.js".

• well documented
• claims not to be DOM templatind: "DOM templating is SLOW."

2. Pure JS
   https://pure-js.com/
   pure.js can be used where Javascript and DOM are available
   It can be run server side, but shines client side.

3. Jquery-templating
   https://codepb.github.io/jquery-template/
   "jQuery Templating is a jQuery plugin that makes using templates easy and quick. The plugin supports loading HTML files as templates, or taking a jQuery object as the template (usually using script tags to hold the template). Designed to be simple, the plugin uses valid HTML5 rather than a proprietry templating language, and templates are easily populated by calling jQuery.loadTemplate".

4. Transparency
   http://leonidas.github.io/transparency/
   Transparency is a minimal template engine for jQuery. It maps JSON objects to DOM elements with zero configuration. Just call .render().

5. Apache Wicket
   https://wicket.apache.org/
   Wicket is an open source, component oriented, serverside, Java web application framework.
   Leverage what you know about Java or your favourite HTML editor to write Wicket applications. With pure Java code and HTML markup Wicket is the go-to web framework for purists.

6. DOM templates in Polymer (not sure)
   https://polymer-library.polymer-project.org/2.0/docs/devguide/dom-template

7. Kwartz
   http://www.kuwata-lab.com/kwartz/
   Kwartz is a template system which realized the concept of 'Independence of Presentation Logic'(IoPL).
   Supports multiple programing languages: Ruby(eRuby), PHP, Java(JSP)

8. Tapestry
   https://tapestry.apache.org/tapestry4.1/usersguide/template.html
   Tapestry templates are designed to look like valid HTML files (component HTML templates will just be snippets of HTML rather than complete pages). Tapestry "hides" its extensions into special attributes of ordinary HTML elements.

Etc. You can find more!

What about Perl?

I understand a good and maintained implementation is important for Perl.

I only identified two projects. They are a bit old (2011-2012), but some major perl templating systems major release are even old.

Template::Semantic
I've already wrote about it.
https://metacpan.org/pod/release/TOMITA/Template-Semantic-0.09/lib/Template/Semantic.pm
Template::Semantic is a template engine for XHTML/XML based on XML::LibXML that doesn't use any template syntax. This module takes pure XHTML/XML as a template, and uses XPath or CSS selectors to assign values. [...] Use XPath expression or CSS selector as a selector.

PROs:

• complete separation between HTML and Perl
• use CSS selectors (very familiars to HTML/CSS and JS developers)
• very easy to use:

print Template::Semantic->process('template.html', {
	'title, h1' => 'Naoki Tomita',
	'ul.urls li' => [
		{ 'a' => 'Profile & Contacts', 'a@href' => 'http://e8y.net/', },
		{ 'a' => 'Twitter',            'a@href' => 'http://twitter.com/tomita/', },
	],
});

CONs:

• search for a new maintainer
• last release in 2012
• @stephane doesn't seem to feel it (and I think it's important that he feels comfortable with the solution)

HTML::Seamstress
https://metacpan.org/pod/HTML::Seamstress
Welcome to push-style dynamic HTML generation!
When looking at HTML::Seamstress, we are looking at a uniquely positioned 4th-generation HTML generator. Seamstress offers two sets of advantages: those common to all 4th generation htmlgens and those common to a subclass of HTML::Tree.

PROs:

• replace content, so we can make HTML templates with false text (Lorem ipsum...) and just replace it
• easy to use ; no need to know CSS selectors (only ids and pure perl):

my $tree = html::hello_world->new; 
$tree->look_down(id => name)->replace_content('terrence brannon');
$tree->look_down(id => date)->replace_content('5/11/1969');
print $tree->as_HTML;

CONs:

• last changes in 2011
• needs to previously create a compiled template

I would really appreciate your feedbacks about all that, and your ideas about opportunities to have other devs joining thanks to templates. That said, I would perfectly understand if we choose something more classical. I think Template::Toolkit is the best one, though, in particular because it is also available in Python and Javascript.

[aleene]

I like the usage of a templating system. I have been using Expression Engine for a while.

[stephanegigandet]

Thank you very much @CharlesNepote for the detailed writeup!

I think we clearly share the goals you outline, and you have a good point that it would be great that designers are able to test templates without having to install an instance of Product Opener.

On the other hand, (it's of course a matter of personal taste), I'm very unconfortable with solutions that do the rendering on the client side with Javascript.

If we do the rendering on the server side, we could also try to implement a way for designers to test templates (e.g. by allowing them to store personal templates).

[hangy]

Thanks @CharlesNepote! I agree that JS based templating probably a good option, but would shy away from jQuery if possible. An API based website #80 based on Vue, react, or angular (or other popular libs) would likely attract more frontend developers, but is a larger overhaul.

[raphael0202]

I totally support the usage of a templating system for Product Opener, for all the reasons detailed by @CharlesNepote.
However, I'm not as convinced as Charles that classical templating system are worse than pure HTML templates (without logic). We're not necessarily dependent on Product Opener to render the template, as long as the data format is fixed. Furthermore, I think the flexibility of being able to incorporate simple logic (if/else, for loop) is worth the cost.

[hangy]

However, I'm not as convinced as Charles that classical templating system are worse than pure HTML templates (without logic). We're not necessarily dependent on Product Opener to render the template, as long as the data format is fixed. Furthermore, I think the flexibility of being able to incorporate simple logic (if/else, for loop) is worth the cost.

I'm inclined to agree. The logic needs to be somewhere, and really small partial templates can be difficult to understand when doing frontend development and l10n. However, designer support of special templating languages can be tricky. But that can work, if we look at the attribute based logic that angular uses. One advantage of pure HTML though could be that they could be translated in crowdin "as-is".

[CharlesNepote]

On the other hand, (it's of course a matter of personal taste), I'm very unconfortable with solutions that do the rendering on the client side with Javascript.

I agree with you. I was only giving examples. My point was not about client side vs server side but pure HTML vs template language/tags.

If we do the rendering on the server side, we could also try to implement a way for designers to test templates (e.g. by allowing them to store personal templates).

Interesting :-) 👍

[stephanegigandet]

Hi Everyone!

Thanks a lot for all the ideas and feedback. It shows there's clearly a big need and big interest.

@areeshatariq is going to work on the templating system, as part of an internship through Outreachy, and thanks to the Perl Foundation.

There are many different ways that have been suggested, all with very interesting reasons. And there are some that we definitely should explore for the longer term (like what is suggested by @hangy and @teolemon in #80 ). Those have huge potential, but they are also very risky (it's always difficult to know which technology will be still hot 5 or 10 years from now), and require a lot of work (e.g. if we want to do API based rendering, we should probably overhaul the API as well, to have better handling of taxonomies, internationalisation etc. in the API itself).

So I think our safest bet, to fix or immediate problem (HTML and CSS code embedded in the Perl code), is to start simple, in an incremental way.

I asked Areesha to try to do the templating for a couple of things first (the popup that contains the Nutri-Score computation details with the points etc., and then the nutrition facts table), to see how we can do it concretly (e.g. dealing with the translations of the .po files), and so that we can all see what the template code looks like for actual OFF HTML generation.

We will start with Template::Toolkit first (it's popular and has the features we need), and we can share the results (how the Perl code was changed, and what the template looks like). If it works well, then we can gradually extend what code is generated by Template::Toolkit. If there are issues / concerns, we could try to experiment with a few more templating engines. Once we have done the templating support for Template::Toolkit, it should be relatively easy to try another templating engine.

[teolemon]

Can we consider this as fixed @stephanegigandet ?

[stephanegigandet]

Closing this bug. We still have some HTML in the code, but the bulk of it has been removed.