Extract multiple failed lines by parsing Ruby

[yujinakayama]

For rspec/rspec-expectations#790

With the following failing example:

expect('RSpec').to be_a(String)
              .and start_with('R')
              .and end_with('z')

Before

Failures:

  1) failure snippet is an example
     Failure/Error: expect('RSpec').to be_a(String)
       expected "RSpec" to end with "z"
     # ./spec/test_spec.rb:3:in `block (2 levels) in <top (required)>'

After

Failures:

  1) failure snippet is an example
     Failure/Error:
       expect('RSpec').to be_a(String)
                     .and start_with('R')
                     .and end_with('z')

       expected "RSpec" to end with "z"
     # ./spec/test_spec.rb:3:in `block (2 levels) in <top (required)>'

[myronmarston]

For multiline snippets it should certainly be started on the next line but for single line snippets I think I prefer it to be on the same line like it was before. It cuts down on unnecessary churn in this PR if you don't have to change all the expectations about the failure formatting and keeping it to one line means the failure output doesn't take up as much vertical space, which is always nice.

How much effort would it be to keep single-line failures on the same line as Failure/Error:?

[yujinakayama]

Actually that's what I was unsure about. I made the snippets be always on the next line for consistency, but it's not a strong opinion. I'll make change to do so.

By the way there's another concern about putting the snippets on the next line. There's no problem when an RSpec's exception has a message starting with a blank line:

 1.1) Failure/Error:
        expect(response.status).to eq(200)

         expected: 200
              got: 404

         (compared using ==)
       # ./spec/nested_failure_aggregation_spec.rb:7

However it's hard to read when there's no blank line:

 1.2.2) Failure/Error:
          expect(response.headers).to include("Content-Length" => "21")
          expected {"Content-Type" => "text/plain"} to include {"Content-Length" => "21"}
          Diff:
          @@ -1,2 +1,2 @@
          -[{"Content-Length"=>"21"}]
          +"Content-Type" => "text/plain",
        # ./spec/nested_failure_aggregation_spec.rb:11

Maybe we should always put a blank line in ExceptionPresenter and remove the blank line in exception message?

[yujinakayama]

Oh you had already mentioned it :) #2083 (comment)

[myronmarston]

I'm really excited about this feature, @yujinakayama -- thanks for developing it! I'm not done with my review yet but need to head to bed. For now I wanted to share a few general thoughts:

• I believe the build is failing due to the issue described in Relying on a temp branch of an rspec repo can cause problems rspec-dev#137. It would probably be helpful to make the rspec-support PR and get that merged so the build failure goes away.
• CodeRay, the gem we use to provide syntax highlighting for HTML code snippets also includes a terminal encoder. IMO it would be pretty amazing if we could syntax highlight these snippets in the failure output. It would make them much easier to read. We obviously don't want to require CodeRay as a hard dependency but we can do it like we do the HTML snippets -- use it if we're able to load it, and if we're not, we can add a # Install the coderay gem to get syntax highlighting comment to the bottom of each multi-line snippet. (For single-line snippets I'd like them to stay on the Failure/Error: line as mentioned above so the comment wouldn't fit there). I definitely don't consider adding syntax highlighting to be a merge blocker but it would make this feature even more useful, IMO, and hopefully isn't difficult.
• Another idea for a future enhancement to this (and definitely not a requirement for this PR): perhaps we could add left-aligned line numbers next to the snippet? Actually, if we do that, that might sway me to put single-line snippets on their own line like you've already done.
• Is the new code you added lazily loaded only when a failure actually occurs? As much as possible I want RSpec to load as little as possible to get booted and running, and then delay the loading of optional features like this until they are actually used.
• There's a planned change I haven't yet made (discussed in Getting "Unable to find matching line from backtrace" from expectation in spec support file #1991) that could affect things here: instead of looking only for a failure snippet from a spec file (as the current logic does) I'd like it to show failure snippets from other files, too. Getting "unable to find line" is never useful. That may affect things here since spec file failures are generally single expectation expressions but errors from other files may not have quite as clear snippet boundaries. Any thoughts on how this feature might interact with that change if we ever make it?
• Are there any gotchas or situations where the snippet extractor doesn't work properly? For example, how does it handle the case where there are multiple expectations on one line but only one failed?

expect(1).to eq(1); expect(2).to eq(1); expect(2).to eq(2)

This is a contrived example obviously...

[myronmarston]

I noticed a case where this didn't work as I expect:

expect {
  expect("foo").to start_with("a").and end_with("z")
}.to fail_with(dedent <<-EOS)
  |   expected "fo" to start with "a"
  |
  |...and:
  |
  |   expected "foo" to end with "z"
EOS

(This is a slightly modified spec from rspec-expectations). The extracted snippet is:

       expect {
         expect("foo").to start_with("a").and end_with("z")
       }.to fail_with(dedent <<-EOS)

Notice it's missing the heredoc. It would be nice if that was included.

In addition, in the full output the snippet is crammed right next to the failure message:

     Failure/Error:
       expect {
         expect("foo").to start_with("a").and end_with("z")
       }.to fail_with(dedent <<-EOS)
       expected RSpec::Expectations::ExpectationNotMetError with "   expected \"fo\" to start with \"a\"\n\n...and:\n\n   expected \"foo\" to end with \"z\"", got #<RSpec::Expectations::ExpectationNotMetError:    expected "foo" to start with "a"

       ...and:

          expected "foo" to end with "z"> with backtrace:
 ...

I think it would be easier if there was an extra line break after the snippet:

     Failure/Error:
       expect {
         expect("foo").to start_with("a").and end_with("z")
       }.to fail_with(dedent <<-EOS)

       expected RSpec::Expectations::ExpectationNotMetError with "   expected \"fo\" to start with \"a\"\n\n...and:\n\n   expected \"foo\" to end with \"z\"", got #<RSpec::Expectations::ExpectationNotMetError:    expected "foo" to start with "a"

       ...and:

          expected "foo" to end with "z"> with backtrace:
 ...

Or perhaps on both sides of the snippet:

     Failure/Error:

       expect {
         expect("foo").to start_with("a").and end_with("z")
       }.to fail_with(dedent <<-EOS)

       expected RSpec::Expectations::ExpectationNotMetError with "   expected \"fo\" to start with \"a\"\n\n...and:\n\n   expected \"foo\" to end with \"z\"", got #<RSpec::Expectations::ExpectationNotMetError:    expected "foo" to start with "a"

       ...and:

          expected "foo" to end with "z"> with backtrace:
...

[myronmarston]

@yujinakayama -- I went ahead and addressed #1991 via #2088 as that will help to give 3.4 a nice "vastly improved failure output" theme when combined with this PR and rspec/rspec-expectations#859. I don't think it complicates this PR but there might be a slight merge conflict when one or the other PR gets merged.

[yujinakayama]

Now I'm trying the improve the format of message lines and thinking of it in various cases.

Here are some failure messages with the current implementation in this branch:

  1) failure message format is a single line expectation with a single line failure message
     Failure/Error: expect(1).to be_a(String)
       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  2) failure message format is a multi-line expectation with a single line failure message
     Failure/Error:
       expect(1)
         .to be_a(String)
       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  3) failure message format is a single line expectation with a multi-line failure message
     Failure/Error: expect(1).to eq(2)

       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  4) failure message format is a multi-line expectation with a multi-line failure message
     Failure/Error:
       expect(1)
         .to eq(2)

       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

The failure 2 is obviously unreadble. Also, the formats are inconsistent a bit.

To address these issues, I'm thinking of the following two formats:

A: Less lines when possible

• Put the snippet on the same line with the Failure/Error: when it's single line
• Insert a blank line after the snippet only when the it's multiline

  1) failure message format is a single line expectation with a single line failure message
     Failure/Error: expect(1).to be_a(String)
       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  2) failure message format is a multi-line expectation with a single line failure message
     Failure/Error:
       expect(1)
         .to be_a(String)

       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  3) failure message format is a single line expectation with a multi-line failure message
     Failure/Error: expect(1).to eq(2)
       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  4) failure message format is a multi-line expectation with a multi-line failure message
     Failure/Error:
       expect(1)
         .to eq(2)

       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

B: Consistent format

• Always put the snippet after the Failure/Error: line
• Always insert a blank line after the snippet

  1) failure message format is a single line expectation with a single line failure message
     Failure/Error:
       expect(1).to be_a(String)

       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  2) failure message format is a multi-line expectation with a single line failure message
     Failure/Error:
       expect(1)
         .to be_a(String)

       expected 1 to be a kind of String
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  3) failure message format is a single line expectation with a multi-line failure message
     Failure/Error:
       expect(1).to eq(2)

       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

  4) failure message format is a multi-line expectation with a multi-line failure message
     Failure/Error:
       expect(1)
         .to eq(2)

       expected: 2
            got: 1

       (compared using ==)
     # /Users/me/Projects/rspec-dev/repos/rspec-support/lib/rspec/support.rb:87:in `block in <module:Support>'

What do you think?

[myronmarston]

How about option (C):

• Put the snippet on the same line with the Failure/Error: when it's single line
• Insert a blank line between the snippet and failure message when either are multiline strings.

(In other words, the only time there wouldn't be a blank line is when both are single line strings).

[myronmarston]

BTW, I merged #2088 so you may want to rebase.

[myronmarston]

❤️ This looks great to me! It'll need a changelog entry but that can happen post-merge. Want to do the honors, @yujinakayama?

[yujinakayama]

Added changelog entries 👌

[JonRowe]

Merge when green! :)

[yujinakayama]

BTW I'll try code highlight with CodeRay and hopefully it'll be merged before 3.4 release

[myronmarston]

BTW I'll try code highlight with CodeRay and hopefully it'll be merged before 3.4 release

Sounds great! FWIW, I played around with it a bit a few days ago and I pushed my spike up in 170b28e. Feel free to use any (or none) of that in whatever you come up with.