Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Error Handling: Jinja and Workload Template Errors #447

Open
IanHoang opened this issue Jan 24, 2024 · 4 comments
Open

Improve Error Handling: Jinja and Workload Template Errors #447

IanHoang opened this issue Jan 24, 2024 · 4 comments
Assignees
@IanHoang
Labels
enhancement New feature or request Medium Priority

Comments

@IanHoang
Copy link
Collaborator

IanHoang commented Jan 24, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

Many OSB users have encountered Jinja and workload errors that are unclear, which lead many to believe that there is a bug in OSB tool itself. This happens in several situations but often happens when users make edits to workload files. Here are a few examples and their associated fixes.

Example 1:

2024-01-23 22:53:19,727 ActorAddr-(T|:58179)/PID:97098 osbenchmark.workload.loader ERROR Could not load [/Users/hoangia/Desktop/Development/create-workload/movies-with-custom-queries/workload.json].
Traceback (most recent call last):

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 984, in read
  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 72 column 18 (char 1613)

2024-01-23 22:53:19,730 ActorAddr-(T|:58179)/PID:97098 osbenchmark.workload.loader ERROR Cannot load workload [movies-with-custom-queries]


Traceback (most recent call last):

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/actor.py", line 92, in guard
    return f(self, msg, sender)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 108, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 195, in setup
    self.current_workload = workload.load_workload(self.cfg)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 193, in load_workload
    return _load_single_workload(cfg, repo, repo.workload_name)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 200, in _load_single_workload
    current_workload = reader.read(workload_name, workload_repository.workload_file(workload_name), workload_dir)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 1006, in read
    raise WorkloadSyntaxError(msg, str(e))

osbenchmark.workload.loader.WorkloadSyntaxError: Could not load '/Users/hoangia/Desktop/Development/create-workload/movies-with-custom-queries/workload.json'. The complete workload has been written to '/var/folders/yh/89c2pcg10szgzwc6qj2h04gst115h_/T/tmptoxqvjsr.json' for diagnosis.

This was because there was a missing comma in the jinja JSON template. It's not as simple to discover this issue through a JSON formatter because there are jinja template inputs and those are labeled as incorrect in a JSON formatter.

Example 2:

2024-01-23 22:53:19,730 ActorAddr-(T|:58179)/PID:97098 osbenchmark.workload.loader ERROR Cannot load workload [movies-with-custom-queries]
    self.environment.handle_exception()

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)

  File "<template>", line 57, in top-level template code

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/filters.py", line 1260, in do_tojson
    return htmlsafe_json_dumps(value, dumper=dumper, **options)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/utils.py", line 619, in htmlsafe_json_dumps
    dumper(obj, **kwargs)

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)

  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

TypeError: Object of type Undefined is not JSON serializable

This error occurred because the Jinja template inputs were missing some important parameters. However, there's no check for them. For instance, template had {{search_clients}} but needed {{search_clients | default(8)}}. Another common instance is when users have something like {{index_settings | default({})}} but needed {{index_settings | default({}) | tojson}}.

Example 3:

Traceback (most recent call last):

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/actor.py", line 92, in guard
    return f(self, msg, sender)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 108, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 195, in setup
    self.current_workload = workload.load_workload(self.cfg)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 193, in load_workload
    return _load_single_workload(cfg, repo, repo.workload_name)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 200, in _load_single_workload
    current_workload = reader.read(workload_name, workload_repository.workload_file(workload_name), workload_dir)

  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload/loader.py", line 1001, in read
    raise WorkloadSyntaxError(msg)

osbenchmark.workload.loader.WorkloadSyntaxError: Could not load '/Users/hoangia/Desktop/Development/create-workload/movies-with-custom-queries/workload.json': Expecting value: line 72 column 18 (char 1613). Lines containing the error:

    },
    {
      "operation": "index-append",
      "clients": ,
-----------------^ Error is here
      "ignore-response-error-level": ""
    },

The complete workload has been written to '/var/folders/yh/89c2pcg10szgzwc6qj2h04gst115h_/T/tmp08a_mtof.json' for diagnosis.

This error was more helpful than others but still lacked clarity. This eventually led developers to realize that there's something wrong with the jinja formatting rather than the JSON file.

Even errors that include The complete workload has been written to '/var/folders/yh/89c2pcg10szgzwc6qj2h04gst115h_/T/tmptoxqvjsr.json' for diagnosis in the error message are not helpful since the file for diagnosis is usually empty or unclear.

Describe the solution you'd like

We should include clearer error messages for reading Jinja and workload templates.

Describe alternatives you've considered

We could have another file that checks for specific issues but that might be unwieldy. The simplest solution would be to add better error handling first.

Additional context
@IanHoang IanHoang added the enhancement New feature or request label Jan 24, 2024
@IanHoang
Copy link
Collaborator Author

Error also occurs for create-workload feature when there's an error with workload template. This will need to have additional error handling.

2024-01-23 21:23:27,418 -not-actor-/PID:81790 osbenchmark.benchmark ERROR A fatal error occurred while running subcommand [create-workload].
Traceback (most recent call last):
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/benchmark.py", line 889, in dispatch_sub_command
    workload_generator.create_workload(cfg)
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 193, in create_workload
    render_templates(
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 123, in render_templates
    write_template(template_vars, templates_path, test_procedures_path, "custom-test-procedures")
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 107, in write_template
    f.write(template.render(template_vars))
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/resources/custom-test-procedures.json.j2", line 35, in top-level template code
    {%- block queries -%}
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/resources/custom-test-procedures.json.j2", line 44, in block "queries"
    "target-throughput": {{ query.target_throughput | tojson }}
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/filters.py", line 1260, in do_tojson
    return htmlsafe_json_dumps(value, dumper=dumper, **options)
  File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/utils.py", line 619, in htmlsafe_json_dumps
    dumper(obj, **kwargs)
  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Undefined is not JSON serializable

@IanHoang IanHoang mentioned this issue Jan 26, 2024
Merged
1 task
@IanHoang
Copy link
Collaborator Author

Note: These are not the only issues that are encountered, just a few unclear ones.

@IanHoang
Copy link
Collaborator Author

IanHoang commented Feb 2, 2024

Will address this in a separate PR #448 (comment)

@gkamat
Copy link
Collaborator

gkamat commented Oct 15, 2024

@IanHoang it appears that there has been at least one relevant change merged, but there are still areas where Jinja error messages can be improved. Therefore, leaving this open for now.

@IanHoang IanHoang self-assigned this Dec 11, 2024
@IanHoang IanHoang moved this from 🆕 New to 📦 Backlog in Engineering Effectiveness Board Dec 11, 2024
@IanHoang IanHoang mentioned this issue Dec 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@IanHoang IanHoang

Labels
enhancement New feature or request Medium Priority
Projects
Engineering Effectiveness Board
Status: 📦 Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gkamat @IanHoang