REST API /search return result with HTML tags and formating #2612

wy193777 · 2019-01-04T20:02:09Z

The /search REST API return results with unnecessary HTML tags and formatting. Is there a way to turn off HTML formatting on search results from REST API?

The text was updated successfully, but these errors were encountered:

vladak · 2019-01-04T20:08:41Z

Could you give an example ?

wy193777 · 2019-01-04T20:38:19Z

Below is an example from a instance run locally. You can see a lot of <b> been added to wrap the search term. Logically, the client using the REST API should already know the searched term, so strong them isn't necessary. You can also see & been substituted to &.

{
  "time": 68,
  "resultCount": 4,
  "startDocument": 0,
  "endDocument": 3,
  "results": {
    "/Golden-Register-2.0-Backend/package.json": [
      {
        "line": "    \"start\": \"copyfiles sql/**/*.sql build/ &amp;&amp; tsc &amp;&amp; node --<b>max-old-space-size</b>=5120 --trace-warnings -r ts-node/register build/src/index.js --pretty\",",
        "lineNumber": "13"
      }
    ],
    "/diagram-visualization/node_modules/webpack/package.json": [
      {
        "line": "    \"appveyor:test\": \"node node_modules\\\\mocha\\\\bin\\\\mocha --<b>max-old-space-size</b>=4096 --harmony test/*.test.js\",",
        "lineNumber": "129"
      },
      {
        "line": "    \"benchmark\": \"mocha --<b>max-old-space-size</b>=4096 --harmony test/*.benchmark.js -R spec\",",
        "lineNumber": "131"
      },
      {
        "line": "    \"circleci:test\": \"node node_modules/mocha/bin/mocha --<b>max-old-space-size</b>=4096 --harmony test/*.test.js\",",
        "lineNumber": "134"
      },
      {
        "line": "    \"cover\": \"node --<b>max-old-space-size</b>=4096 --harmony ./node_modules/istanbul/lib/cli.js cover -x '**/*.runtime.js' node_modules/mocha/bin/_mocha -- test/*.test.js\",",
        "lineNumber": "135"
      },
      {
        "line": "    \"cover:min\": \"node --<b>max-old-space-size</b>=4096 --harmony ./node_modules/istanbul/lib/cli.js cover -x '**/*.runtime.js' --report lcovonly node_modules/mocha/bin/_mocha -- test/*.test.js\",",
        "lineNumber": "136"
      },
      {
        "line": "    \"test\": \"mocha test/*.test.js --<b>max-old-space-size</b>=4096 --harmony --check-leaks\",",
        "lineNumber": "143"
      }
    ],
    "/Patching-Tool-Client/node_modules/webpack/package.json": [
      {
        "line": "    \"benchmark\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.benchmark.js\\\" --runInBand\",",
        "lineNumber": "200"
      },
      {
        "line": "    \"cover:all\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --coverage\",",
        "lineNumber": "204"
      },
      {
        "line": "    \"cover:integration\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.test.js\\\" --coverage\",",
        "lineNumber": "206"
      },
      {
        "line": "    \"cover:unit\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.unittest.js\\\" --coverage\",",
        "lineNumber": "208"
      },
      {
        "line": "    \"schema-lint\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.lint.js\\\" --no-verbose\",",
        "lineNumber": "214"
      },
      {
        "line": "    \"test\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest\",",
        "lineNumber": "218"
      },
      {
        "line": "    \"test:basic\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/{
          TestCasesNormal,
          StatsTestCases,
          ConfigTestCases
        }.test.js\\\"\",",
        "lineNumber": "219"
      },
      {
        "line": "    \"test:integration\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.test.js\\\"\",",
        "lineNumber": "220"
      },
      {
        "line": "    \"test:unit\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.unittest.js\\\"\",",
        "lineNumber": "221"
      }
    ],
    "/Golden-Register-2.0-Frontend/node_modules/webpack/package.json": [
      {
        "line": "    \"benchmark\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.benchmark.js\\\" --runInBand\",",
        "lineNumber": "244"
      },
      {
        "line": "    \"cover:all\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --coverage\",",
        "lineNumber": "248"
      },
      {
        "line": "    \"cover:integration\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.test.js\\\" --coverage\",",
        "lineNumber": "250"
      },
      {
        "line": "    \"cover:unit\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.unittest.js\\\" --coverage\",",
        "lineNumber": "252"
      },
      {
        "line": "    \"schema-lint\": \"node --<b>max-old-space-size</b>=4096 node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.lint.js\\\" --no-verbose\",",
        "lineNumber": "258"
      },
      {
        "line": "    \"test\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest\",",
        "lineNumber": "260"
      },
      {
        "line": "    \"test:basic\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/{
          TestCasesNormal,
          StatsTestCases,
          ConfigTestCases
        }.test.js\\\"\",",
        "lineNumber": "261"
      },
      {
        "line": "    \"test:integration\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.test.js\\\"\",",
        "lineNumber": "262"
      },
      {
        "line": "    \"test:unit\": \"node --<b>max-old-space-size</b>=4096 --trace-deprecation node_modules/jest-cli/bin/jest --testMatch \\\"&lt;rootDir&gt;/test/*.unittest.js\\\"\",",
        "lineNumber": "263"
      }
    ]
  }
}

The OpenGrok's REST API doc also show this behavior.

{
  "time": 13,
  "resultCount": 35,
  "startDocument": 0,
  "endDocument": 0,
  "results": {
    "/opengrok/test/org/opensolaris/opengrok/history/hg-export-renamed.txt": [{
      "line": "# User Vladimir <b>Kotal</b> &lt;Vladimir.<b>Kotal</b>@oracle.com&gt;",
      "lineNumber": "19"
    },{
      "line": "# User Vladimir <b>Kotal</b> &lt;Vladimir.<b>Kotal</b>@oracle.com&gt;",
      "lineNumber":"29"
    }]
  }

vladak · 2019-01-04T20:48:21Z

It seems that the summarizer/highlighter kicks in.

vladak · 2019-01-04T21:50:35Z

Yes, SearchController uses SearchEngine.results() which uses Summarizer and Summary used therein is inherently HTML based. There needs to be a special version of SearchEngine.results() for the API.

wy193777 · 2019-01-04T21:55:24Z

Thanks for your quick help!

wy193777 · 2019-01-11T22:57:49Z

Have idea on when will this bug been fixed?

vladak · 2019-01-12T09:44:09Z

Not a priority, at least for me. Pull requests are welcome of course. pá 11. 1. 2019 23:57 odesílatel Shenghan Gao <notifications@github.com> napsal:

…

Have idea on when will this bug been fixed? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2612 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACzGDOhqD2nlackS_SLGQV16dsCMm2cSks5vCRbvgaJpZM4ZsfzS> .

wy193777 · 2019-03-20T00:04:46Z

The <b> actually not just comes from Summarizer. After dig into source code, I find those html tags might also comes from Context.java, which is extremely complex.....

wy193777 · 2019-03-20T00:20:11Z

OK, Context.java calls code from PlainLineTokenizer.lex, which is generated from opengrok-indexer/src/main/resources/search/context/PlainLineTokenizer.lex. This lex file seems also did things like htmlize. I guess eliminate <b> is not a simple task.

vladak · 2019-03-20T07:49:38Z

Yes, seems like some serious refactoring is needed. Dne st 20. 3. 2019 1:20 uživatel Shenghan Gao <notifications@github.com> napsal:

…

OK, Context.java calls code from PlainLineTokenizer.lex, which is generated from opengrok-indexer/src/main/resources/search/context/PlainLineTokenizer.lex. This lex file seems also did things like htmlize. I guess eliminate <b> is not a simple task. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2612 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACzGDGGrMh50N6fbWcz0Ut8V9O4kop91ks5vYX69gaJpZM4ZsfzS> .

idodeclare · 2019-03-25T01:38:23Z

It's fairly straight-forward to extend OGKUnifiedHighlighter. I raised PR #2732.

Addresses the source-code related part of oracle#2612 since `OGKUnifiedHighlighter` allows straight- forward, alternate formatters. `HistoryContext` is still a custom highlighting, so it will still return HTML-like content.

vladak added bug API labels Jan 4, 2019

idodeclare mentioned this issue Mar 25, 2019

Feature/search uhighlighter #2732

Open

vladak mentioned this issue Feb 18, 2020

API endpoints for Annotate, History, file listing #3042

Closed

vladak mentioned this issue Oct 13, 2021

search API def return more than needed #3748

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REST API /search return result with HTML tags and formating #2612

REST API /search return result with HTML tags and formating #2612

wy193777 commented Jan 4, 2019

vladak commented Jan 4, 2019

wy193777 commented Jan 4, 2019

vladak commented Jan 4, 2019 •

edited

Loading

vladak commented Jan 4, 2019

wy193777 commented Jan 4, 2019

wy193777 commented Jan 11, 2019

vladak commented Jan 12, 2019 via email

wy193777 commented Mar 20, 2019

wy193777 commented Mar 20, 2019

vladak commented Mar 20, 2019 via email

idodeclare commented Mar 25, 2019

REST API /search return result with HTML tags and formating #2612

REST API /search return result with HTML tags and formating #2612

Comments

wy193777 commented Jan 4, 2019

vladak commented Jan 4, 2019

wy193777 commented Jan 4, 2019

vladak commented Jan 4, 2019 • edited Loading

vladak commented Jan 4, 2019

wy193777 commented Jan 4, 2019

wy193777 commented Jan 11, 2019

vladak commented Jan 12, 2019 via email

wy193777 commented Mar 20, 2019

wy193777 commented Mar 20, 2019

vladak commented Mar 20, 2019 via email

idodeclare commented Mar 25, 2019

vladak commented Jan 4, 2019 •

edited

Loading