Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor benchmarks #2722

Conversation

officialasishkumar
Copy link
Member

📝 Description

Type: 🪲 bugfix

  • fix the benchmark action of PR
  • changed the name of the classes and files
  • removed the benchmarks which are not important.

📌 Resources

Examples, notebooks, and links to useful references.

🚦 Testing

How did you test these changes?

  • Testing pipeline
  • Other method (describe)
  • My changes can't be tested (explain why)

☑️ Checklist

  • I requested two reviewers for this pull request
  • I updated the documentation according to my changes
  • I built the documentation by applying the build_docs label

Note: If you are not allowed to perform any of these actions, ping (@) a contributor.

Copy link

codecov bot commented Jul 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 36.58%. Comparing base (33b5ad9) to head (4d73d7c).
Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2722      +/-   ##
==========================================
- Coverage   36.61%   36.58%   -0.04%     
==========================================
  Files         186      186              
  Lines       14837    14859      +22     
==========================================
+ Hits         5433     5436       +3     
- Misses       9404     9423      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

},
{
"nu": 0,
"temperature": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't physical, I don't think they are worth parametrizing

},
{
"nu": 1,
"temperature": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

"Parameters": [
{
"nu": 1e14,
"temperature": 1e4,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is okay

Signed-off-by: Asish Kumar <[email protected]>
@officialasishkumar officialasishkumar changed the title Refactor benchmarks [WIP] Refactor benchmarks Jul 22, 2024
Signed-off-by: Asish Kumar <[email protected]>
@officialasishkumar
Copy link
Member Author

@tardis-bot
Copy link
Contributor

*beep* *bop*
Hi human,
I ran benchmarks as you asked comparing master (fea0ff0) and the latest commit (a5cbb72).
Here are the logs produced by ASV.
Results can also be downloaded as artifacts here.

Significantly changed benchmarks:

All benchmarks:

All benchmarks:

| Change   | Before [fea0ff01]    | After [fea0ff01]    |   Ratio | Benchmark (Parameter)                                                                                                                                                                                                 |
|----------|----------------------|---------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|          | 38.2±0.1s            | 38.2±0.1s           |       1 | run_tardis.BenchmarkRunTardis.time_run_tardis                                                                                                                                                                         |
|          | 27.3±10μs            | 27.3±10μs           |       1 | transport_geometry_calculate_distances.BenchmarkTransportGeometryCalculateDistances.time_calculate_distance_boundary                                                                                                  |
|          | 44.0±4μs             | 44.0±4μs            |       1 | transport_geometry_calculate_distances.BenchmarkTransportGeometryCalculateDistances.time_calculate_distance_line({'packet': {'nu_line': 0.1, 'is_last_line': True}, 'enable_full_relativity': True})                  |
|          | 43.6±4μs             | 43.6±4μs            |       1 | transport_geometry_calculate_distances.BenchmarkTransportGeometryCalculateDistances.time_calculate_distance_line({'packet': {'nu_line': 0.2, 'is_last_line': False}, 'enable_full_relativity': True})                 |
|          | 7.68±0.01s           | 7.68±0.01s          |       1 | transport_montecarlo_interaction.BenchmarkMontecarloMontecarloNumbaInteraction.time_line_emission({'mu': 0.8599443103322428, 'emission_line_id': 1000, 'energy': 0.9114437898710559})                                 |
|          | 7.73±0.03s           | 7.73±0.03s          |       1 | transport_montecarlo_interaction.BenchmarkMontecarloMontecarloNumbaInteraction.time_line_scatter(<LineInteractionType.MACROATOM: 2>)                                                                                  |
|          | 7.68±0s              | 7.68±0s             |       1 | transport_montecarlo_interaction.BenchmarkMontecarloMontecarloNumbaInteraction.time_line_scatter(<LineInteractionType.SCATTER: 0>)                                                                                    |
|          | 3.49±0.03s           | 3.49±0.03s          |       1 | transport_montecarlo_interaction.BenchmarkMontecarloMontecarloNumbaInteraction.time_thomson_scatter                                                                                                                   |
|          | 4.35±0.03s           | 4.35±0.03s          |       1 | transport_montecarlo_main_loop.BenchmarkTransportMontecarloMainLoop.time_montecarlo_main_loop                                                                                                                         |
|          | 2.14±0m              | 2.14±0m             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_FormalIntegrator_functions                                                                                    |
|          | 1.61±0.6μs           | 1.61±0.6μs          |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_p_values(100)                                                                                       |
|          | 2.49±0.6μs           | 2.49±0.6μs          |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_p_values(1000)                                                                                      |
|          | 9.83±0.6μs           | 9.83±0.6μs          |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_p_values(10000)                                                                                     |
|          | 749±4ms              | 749±4ms             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(0.0, array([0. , 0.5, 1. ]))                                                                      |
|          | 753±4ms              | 753±4ms             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(0.0, array([1. , 1.5, 2. ]))                                                                      |
|          | 739±0.5ms            | 739±0.5ms           |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(0.5, array([0. , 0.5, 1. ]))                                                                      |
|          | 739±2ms              | 739±2ms             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(0.5, array([1. , 1.5, 2. ]))                                                                      |
|          | 735±9ms              | 735±9ms             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(1.0, array([0. , 0.5, 1. ]))                                                                      |
|          | 757±3ms              | 757±3ms             |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_calculate_z(1.0, array([1. , 1.5, 2. ]))                                                                      |
|          | 651±200ns            | 651±200ns           |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_intensity_black_body({'nu': 0, 'temperature': 1})                                                             |
|          | 691±200ns            | 691±200ns           |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_intensity_black_body({'nu': 1, 'temperature': 1})                                                             |
|          | 721±300ns            | 721±300ns           |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_intensity_black_body({'nu': 100000000000000.0, 'temperature': 10000.0})                                       |
|          | 3.52±0.9μs           | 3.52±0.9μs          |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_trapezoid_integration(100.0)                                                                                  |
|          | 9.16±1μs             | 9.16±1μs            |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_trapezoid_integration(1000.0)                                                                                 |
|          | 64.9±1μs             | 64.9±1μs            |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_trapezoid_integration(10000.0)                                                                                |
|          | 651±10μs             | 651±10μs            |       1 | transport_montecarlo_numba_formal_integral_p.BenchmarkMontecarloMontecarloNumbaNumbaFormalIntegral.time_trapezoid_integration(100000.0)                                                                               |
|          | 1.46±0.01s           | 1.46±0.01s          |       1 | transport_montecarlo_numba_interface.BenchmarkMontecarloMontecarloNumbaNumbaInterface.time_opacity_state_initialize('macroatom')                                                                                      |
|          | 1.46±0s              | 1.46±0s             |       1 | transport_montecarlo_numba_interface.BenchmarkMontecarloMontecarloNumbaNumbaInterface.time_opacity_state_initialize('scatter')                                                                                        |
|          | 667±200ns            | 667±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000.0, 255.5)                                                                                          |
|          | 621±200ns            | 621±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000.0, 511.0)                                                                                          |
|          | 636±200ns            | 636±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000.0, 5110000000.0)                                                                                   |
|          | 606±200ns            | 606±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000000000.0, 255.5)                                                                                    |
|          | 626±200ns            | 626±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000000000.0, 511.0)                                                                                    |
|          | 641±200ns            | 641±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(100000000000.0, 5110000000.0)                                                                             |
|          | 617±200ns            | 617±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(1000000000000000.0, 255.5)                                                                                |
|          | 631±200ns            | 631±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(1000000000000000.0, 511.0)                                                                                |
|          | 646±200ns            | 646±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation(1000000000000000.0, 5110000000.0)                                                                         |
|          | 696±200ns            | 696±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 255.5, 'Iron_group_fraction': 0.0})                              |
|          | 706±300ns            | 706±300ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 255.5, 'Iron_group_fraction': 0.5})                              |
|          | 691±200ns            | 691±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 511.0, 'Iron_group_fraction': 0.5})                              |
|          | 722±200ns            | 722±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 5110000000.0, 'Iron_group_fraction': 0.0})                       |
|          | 702±200ns            | 702±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation({'Ejecta_density': 100000.0, 'Energy': 255.5, 'Iron_group_fraction': 1.0})                          |
|          | 681±200ns            | 681±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 255.5, 'Iron_group_fraction': 0.0})                            |
|          | 741±200ns            | 741±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 255.5, 'Iron_group_fraction': 0.5})                            |
|          | 726±200ns            | 726±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 511.0, 'Iron_group_fraction': 0.5})                            |
|          | 751±200ns            | 751±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation({'Ejecta_density': 0.01, 'Energy': 5110000000.0, 'Iron_group_fraction': 0.0})                     |
|          | 701±200ns            | 701±200ns           |       1 | transport_montecarlo_opacities.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation({'Ejecta_density': 100000.0, 'Energy': 255.5, 'Iron_group_fraction': 1.0})                        |
|          | 711±200ns            | 711±200ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_distance_electron({'electron_density': 1.0, 'tua_event': 10000000000.0})                                                          |
|          | 667±200ns            | 667±200ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_distance_electron({'electron_density': 1e-05, 'tua_event': 10000000000.0})                                                        |
|          | 677±200ns            | 677±200ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_tau_electron({'electron_density': -1, 'distance': 0})                                                                             |
|          | 721±300ns            | 721±300ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_tau_electron({'electron_density': -10000000000.0, 'distance': -10000000000.0})                                                    |
|          | 697±200ns            | 697±200ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_tau_electron({'electron_density': 10000000000.0, 'distance': 10000000000.0})                                                      |
|          | 696±200ns            | 696±200ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_calculate_tau_electron({'electron_density': 1e-05, 'distance': 1.0})                                                                        |
|          | 360±100ns            | 360±100ns           |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_get_random_mu                                                                                                                               |
|          | 2.15±0s              | 2.15±0s             |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_emitted({'current_shell_id': 132, 'delta_shell': 11, 'no_of_shells': 132})                                |
|          | 2.20±0.01s           | 2.20±0.01s          |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_emitted({'current_shell_id': 132, 'delta_shell': 2, 'no_of_shells': 133})                                 |
|          | 2.17±0.04s           | 2.17±0.04s          |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_increment({'current_shell_id': 132, 'delta_shell': -1, 'no_of_shells': 199})                              |
|          | 2.19±0.04s           | 2.19±0.04s          |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_increment({'current_shell_id': 132, 'delta_shell': 0, 'no_of_shells': 132})                               |
|          | 2.20±0.02s           | 2.20±0.02s          |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_reabsorbed({'current_shell_id': 132, 'delta_shell': 132, 'no_of_shells': 132})                            |
|          | 2.15±0s              | 2.15±0s             |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_move_packet_across_shell_boundary_reabsorbed({'current_shell_id': 132, 'delta_shell': 133, 'no_of_shells': 133})                            |
|          | 37.6±5μs             | 37.6±5μs            |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_update_line_estimators({'cur_line_id': 0, 'distance_trace': 0, 'time_explosion': 52000000.0, 'enable_full_relativity': True})               |
|          | 37.2±5μs             | 37.2±5μs            |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_update_line_estimators({'cur_line_id': 0, 'distance_trace': 1000000000000.0, 'time_explosion': 52000000.0, 'enable_full_relativity': True}) |
|          | 38.4±5μs             | 38.4±5μs            |       1 | transport_montecarlo_packet.BenchmarkMontecarloMontecarloNumbaPacket.time_update_line_estimators({'cur_line_id': 1, 'distance_trace': 100000.0, 'time_explosion': 10000000000.0, 'enable_full_relativity': False})    |
|          | 1.43±0.01s           | 1.43±0.01s          |       1 | transport_montecarlo_r_packet.BenchmarkMontecarloMontecarloNumbaRPacket.time_rpacket_trackers_to_dataframe                                                                                                            |
|          | 6.37±0.02s           | 6.37±0.02s          |       1 | transport_montecarlo_single_packet_loop.BenchmarkMontecarloMontecarloNumbaVpacket.time_single_packet_loop                                                                                                             |
|          | 5.02±0.02s           | 5.02±0.02s          |       1 | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_bad_vpacket                                                                                                                         |
|          | 5.07±0.02s           | 5.07±0.02s          |       1 | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket                                                                                                                             |
|          | 4.96±0.03s           | 4.96±0.03s          |       1 | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket_volley({'tau_russian': 10.0, 'survival_possibility': 0.0})                                                                  |
|          | 5.01±0.01s           | 5.01±0.01s          |       1 | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket_volley({'tau_russian': 15.0, 'survival_possibility': 0.1})                                                                  |
|          | 3.74±0.02s           | 3.74±0.02s          |       1 | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket_within_shell                                                                                                                |

If you want to see the graph of the results, you can check it here

@officialasishkumar officialasishkumar changed the title [WIP] Refactor benchmarks Refactor benchmarks Jul 22, 2024
@officialasishkumar officialasishkumar marked this pull request as ready for review July 22, 2024 15:37
@officialasishkumar
Copy link
Member Author

The benchmark run is successful in these changes.

link: https://paste.opensuse.org/pastes/0e80fd45e5c1

@andrewfullard
Copy link
Contributor

I notice the before and after commits are still the same in the bot comment. Should that have been resolved here?

@officialasishkumar
Copy link
Member Author

new action run on the existing benchmarks.yml -> https://github.com/officialasishkumar/tardis/actions/runs/10077232306/job/27859281972

@officialasishkumar
Copy link
Member Author

I have ironed out all the bugs. So this can be merged to get the benchmarks working.

cc: @andrewfullard

Copy link
Contributor

@andrewfullard andrewfullard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job fixing the benchmarks, please remove the test items

@@ -61,6 +62,9 @@ def single_packet_loop(
This function does not return anything but changes the r_packet object
and if virtual packets are requested - also updates the vpacket_collection
"""

for i in range(200):
print("hello")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, my bad. I will fix it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@@ -30,6 +30,7 @@
)
from tardis.transport.montecarlo.vpacket import trace_vpacket_volley

import time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Signed-off-by: Asish Kumar <[email protected]>
@officialasishkumar officialasishkumar force-pushed the change-names-of-benchmark-classes branch from fcf70f5 to 4d73d7c Compare July 24, 2024 13:50
Copy link
Member

@atharva-2001 atharva-2001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me too!

@andrewfullard andrewfullard merged commit 051effd into tardis-sn:master Jul 25, 2024
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants