Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Resolver] Resolver's summary suggests UNRESOLVED due to "no human reviewer" #5995

Closed
xingyaoww opened this issue Jan 3, 2025 · 3 comments · Fixed by #5996
Closed

[Resolver] Resolver's summary suggests UNRESOLVED due to "no human reviewer" #5995

xingyaoww opened this issue Jan 3, 2025 · 3 comments · Fixed by #5996
Labels
fix-me Attempt to fix this issue with OpenHands resolver Related to OpenHands Resolver

Comments

@xingyaoww
Copy link
Collaborator

xingyaoww commented Jan 3, 2025

We got something like the following -- This status overview should ideally just be a summary of changes & LLM's prediction of the task success, it should NOT say it require external human reviewer

          Status Overview:

Pending Issues:

  • Right side panels extending beyond browser window
  • Terminal panel resizing problems
  • Chat panel max-width adjustment to 50%

Current State:
While changes have been implemented, there is no reviewer confirmation that the fixes are working as intended. Multiple previous attempts to resolve these issues were unsuccessful.

Recommendation:
Await reviewer verification before considering these issues closed. Visual confirmation is needed to ensure:

  1. Panel containment within browser
  2. Proper terminal resizing
  3. Correct chat panel width

Overall Status: UNRESOLVED - Pending reviewer confirmation

Originally posted by @openhands-agent in #5966 (comment)

We likely need a prompt change to tell LLM do not rely on human for this judgment.

We should add a line in prompt files from this folder:openhands/resolver/prompts/guess_success

@enyst enyst added the resolver Related to OpenHands Resolver label Jan 3, 2025
@xingyaoww xingyaoww added the fix-me Attempt to fix this issue with OpenHands label Jan 3, 2025
@openhands-agent
Copy link
Contributor

OpenHands started fixing the issue! You can monitor the progress here.

@openhands-agent
Copy link
Contributor

A potential fix has been generated and a draft PR #5996 has been created. Please review the changes.

neubig added a commit that referenced this issue Jan 4, 2025
…e to "no human reviewer" (#5996)

Co-authored-by: Xingyao Wang <[email protected]>
Co-authored-by: Graham Neubig <[email protected]>
@tawago
Copy link

tawago commented Jan 5, 2025

@neubig

I've been trying to add openhands-resolver.yml github action workflow on my repository.
It manages to work on tasks and create a branch but always fails to make it as a PR.
openhands.resolver.send_pull_request returns something like below

This pull request fixes #XX.
While the AI agent claims to have completed the task, there is no concrete evidence provided in the message thread of actual code changes or a Pull Request being created and merged. The agent only describes what they claim to have done, but without seeing
We cannot verify that the issue has been successfully resolved. The agent's message only describes intended or claimed actions, but doesn't provide proof of implementation. To consider this resolved, we would need to see the actual code changes in a Pull Request and confirmation that the changes were successfully merged into the codebase.
A proper resolution would require evidence of the actual implementation and successful merge of these changes.

I wonder if this ticket is the fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix-me Attempt to fix this issue with OpenHands resolver Related to OpenHands Resolver
Projects
None yet
4 participants