Matching comments is not supported

[Vasco-jofra]

Problem description

It would be nice if Semgrep could be used to match python comments in a simpler way. For example, I wanted to match python functions with the @route decorator, did not have the @protected_endpoint decorator, and did not have a specific comment (# Hello in the examples below).

This was the best solution I came up with (https://semgrep.dev/s/pwr0).

patterns:
  - pattern-regex: |
      # Hello
      [\s\S]*?
      def.*
      (^[ \t]+[^\n]*\n?)+
  - pattern: |
      @route($ROUTE, ...)
      def $F(...):
        ...
  - pattern-not: |
      @protected_endpoint(...)
      def $F(...):
        ...

The pattern-regex works as follows:

1. Matches the comment (# Hello)
2. Matches any decorators until it finds a def, i.e., until the beginning of a function ([\s\S]*?\ndef.*)
3. Matches any string that starts with a space or tab, i.e., until the end of the function ((^[ \t]+[^\n]*\n?)+)

Problems:

• If a match is at the bottom of a file without a new line at the end, this solution does not work.
• It is error prone to write these complicated regexes

Solution I'd like

I would like to be able to match comments with Semgrep. For example, the rule could look something like this (https://semgrep.dev/s/2Dyq):

patterns:
  - pattern: |
      @route($ROUTE, ...)
      def $F(...):
        ...
  - pattern-not: |
      @protected_endpoint(...)
      def $F(...):
        ...
  - pattern-not: |
      # Hello
      def $F(...):
        ...

[r2c-demo]

This issue is synced in Linear at https://linear.app/r2c/issue/PA-1775/matching-python-comments-is-not-well-supported. Note: this link is for r2c use only and is not accessible publicly.

[Sjord]

I think comments are generally ignored by semgrep in any language. Today I also ran into a use case where I want to match comments, but in PHP: the comments contain some documentation on parameters, and I want to cross-check that against the code with semgrep. So matching comments would be desirable.

[stale]

This issue is being marked stale because there hasn't been any activity in 14 days and either it wasn't prioritized or its priority is high. Please apply the appropriate priority:* label before removing the stale label.

[mjambon]

Hi there. It's not impossible that we could add some support for matching comments but I haven't found a simple approach that's also generic yet. Thanks for the clear use case, it's helpful.

Preliminary work to have comments in the AST is almost done: semgrep/ocaml-tree-sitter-core#48

[Pchao93]

Chiming in with another use case - catching and preventing linter exemption comments. In ruby, Rubocop rules can be disabled with comments like # rubocop:disable Some/Rule, which cannot be detected. Very interested in seeing this usage supported!

[xmo-odoo]

It's pretty much the same for Python, tooling generally supports a way to opt out (e.g. noqa, pylint: disable, ...) and while semgrep itself can be invoked with --disable-nosem that's often not the option elsewhere. Plus it might still be of value to have a pass highlighting such opt-outs in order to review them more carefully.

OTOH the command of these while parseable is generally not regular, so I think pattern-regex is sufficient, it seems to work fine enough.