Fix line vs. block comment interaction in the saw-script lexer #2191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Don't have the lexer match //.*; just match the // part, and extend the same logic that handles dropping the bits between /* and */ to also handle dropping the bits between // and EOL. This requires materializing EOL tokens instead of treating them as whitespace, but that costs ~nothing. Also fix it so we generate explicit messages when there are unclosed comments. A // comment that doesn't have an EOL before EOF generates a warning; a /* comment without a matching */ generates an error. The position for this error is the comment start. (The previous behavior was that unclosed comments commented out the EOF token and that would cause a non-specific "parse error".) Update the lexer call sites to post the messages. There should not be four separate lexer call sites, but that's a problem for a future branch. Add some tests. This includes a new test_parse_errors area akin to the existing test_type_errors.
(interactions of comments and Cryptol blocks) Note: these reflect the current behavior, which as discussed in the issue is not particularly desirable. They should be updated when/if the behavior is improved.
sauclovian-g
requested review from
RyanGlScott,
mtullsen and
mccleeary-galois
RyanGlScott
reviewed
Jan 22, 2025
RyanGlScott
approved these changes
Jan 22, 2025
src/SAWScript/Token.hs
Outdated
Comment on lines
38
to
39
| -- EOL End of line | ||
| -- EOF End of file/input |
There was a problem hiding this comment.
Suggested change
| -- EOL End of line | |
| -- EOF End of file/input | |
| -- TEOL End of line | |
| -- TEOF End of file/input |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prompted by #1225 (which is about the saw-core lexer); closes #2181.