Integrated correctionlib developments #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…opagation (cms-btv-pog#39) * feat : add PU files - add pu files, adapt correction will available dicts - add Muon_RunDv1 to data json files - improve file splitting for dask/lxplus - add autorebin for plotting tools - remove redudant puwei file fix : python3.7->3.10 in pipeline * fix : minor fix - update readme - bugfix for dileptt,semileptt workflow - add init to PU directory - update descriptions * minor fixes for runner.py and comparison.py * formatting with black * fix: add JEC to MET and plot format - add JEC to MET - plot format * fix: set tail of PU to 1 * fix: minor fix on runner.py Co-authored-by: Ming-Yan <[email protected]>
* patch for the daskoutput name * reformatted runner (hope with the right black version
* feat : improve plot scripts and add new variables - Add PFCands to defintions - Correct the uncertainty calculations for ratio panels - include MC stat error band - improve DeepJet variable binning and labels * feat : add track related info - add track related info - fix minor comparison * fix : minor fixs - modify name for track variables - fix jet pt axis - fix cut in ctag_semileptt_sf and ttcom workflow - move default errtype to plot_util * fix : fix the file merging in the plots and other minor fixes - move black and flake8 to python 3.10 - change the condition PFCands variables storing supported with higher version(not Reco17_94X) - minor fixes on plotting scripts * feat : minor fixes * fix: update python version in setup.cfg
* fix setup.cfg * fix: move setup to coffea0.7.20
…g#43) * fix : MC merging in plotting scripts and move to PuppiMET - move PFMET to Puppi MET for run3 - extend xrootdtimeout for local executor - fix plotting scripts with merging MC files with same dataset name - fix axes label - fix comparison scripts panel - splitted OS-SS for Wc in plotting scripts * feat : auto-scale x-axis - update x-axis range for plots - fix missing variables in defintion - change lepton dz range, dr range * fix: minor fix for plotting scripts
* fix : fix MET correction and plotting scripts - add MetUnclustEnUpDeltaY,MetUnclustEnUpDeltaX to PuppiMET for MET correction - fix dxy/dz axes for leptons, have zooming version for lepton with QCD veto - fix spacing for the axis labels * fix:minor fix
…tv-pog#45) * fix: plotting style improvements; took care of future warnings in runner.py; added link to original code in runner.py * removed deprecated `extra` in lxplus setup * liniting with black * fixes for plotting * fix label for btag discriminator; added prob distributions * protection for not adding SF axis to prob distributions * add printout for produced plot * flake8 fix * fix: add missing xsection & add protection for missing xsection Co-authored-by: Ming-Yan Lee <[email protected]>
- add additional cross-section rescaler for sample set - add mergemap for comparison script - minor fix for debug mode
Conflicts: plotting/comparison.py src/BTVNanoCommissioning/utils/plot_utils.py testfile/btv_compare.yml
fix : plotting CI, polish readme, reformat configurator - create reproducible config file (`.py`) under output directory - add default `run_options` in configurator - move systematic as common methods - polish readme - fix plotting CI name
fix: limit should be None in default
…tion files) directory - wrap jsonpog as submodule - rename directory name of data/ directory
- utils: make SFs loading compatible with correctionlib and custom inputs - utils: add example using correctionlib in AK4_parameter - utils: rename JEC producer - runner: limit campaign with particular choices, add isSyst flag - doc: - workflows: implement correctionlib, reduce weights implementation on selected events only - workflows: make conditions for branch not exists in files, skip selection/not filled histogram - workflows: load SFs, lumimask in __init__ - workflows(Wc): fix selections same as VHccSF(https://github.com/mondalspandan/VHcc-cTagSF/blob/UL/Analyzer/WcSelection.py) - utils : correct jetid selection for UL - utils : fix Wc soft lepton pT axis - env : avoid python 3.11 for setting enviroment - plotting : fix error band on upper panel - CI: add submodules, ignore black in jsonpog-integration
* updated 13 TeV single top and ttbar cross sections * black v23 does not like formatting of black v22
- reformat with black v23 - restrict setuptools version - wf: fix mamba error, conda setup errors - wf: add submodules - wf: add correctionlib workflow - wf: run workflow with futures
- improve documentation & implemented suggestion from Denise - make ctag_ttsemilep selection sync with Wc
- remove UL lepton SFs, PU, btag SFs
- fix bug with filled discriminant for NanoAOD
- runner: modify memory, disk usages adapt parsl/dask interfaces - plot: fixing error band - doc: minor updates on choices of campaign - data/utils: implemented low pT muon corrections, remove fill_none - env: change to <3.11 for python version, add line for test_env.yml - workflows: fix bug for reading SFs, put HLT flag for triggered lepton - utils: add flag for triggered leptons - fix: remove correction for ttbar
…tions (cms-btv-pog#50) * feat : add correctionlib (jsonpog-integration) and rename data(correction files) directory - wrap jsonpog as submodule - rename directory name of data/ directory * feat : correctionlib implementation & minor fixes - utils: make SFs loading compatible with correctionlib and custom inputs - utils: add example using correctionlib in AK4_parameter - utils: rename JEC producer - runner: limit campaign with particular choices, add isSyst flag - doc: - workflows: implement correctionlib, reduce weights implementation on selected events only - workflows: make conditions for branch not exists in files, skip selection/not filled histogram - workflows: load SFs, lumimask in __init__ - workflows(Wc): fix selections same as VHccSF(https://github.com/mondalspandan/VHcc-cTagSF/blob/UL/Analyzer/WcSelection.py) - utils : correct jetid selection for UL - utils : fix Wc soft lepton pT axis - env : avoid python 3.11 for setting enviroment - plotting : fix error band on upper panel - CI: add submodules, ignore black in jsonpog-integration * fix : fix issues for action updates - reformat with black v23 - restrict setuptools version - wf: fix mamba error, conda setup errors - wf: add submodules - wf: add correctionlib workflow - wf: run workflow with futures * fix: suggestions from Denise & fix semilep selection - improve documentation & implemented suggestion from Denise - make ctag_ttsemilep selection sync with Wc * fix : remove UL files & fix bug for NanoAOD - remove UL lepton SFs, PU, btag SFs - fix bug with filled discriminant for NanoAOD * feat : suggestions from Annika - runner: modify memory, disk usages adapt parsl/dask interfaces - plot: fixing error band - doc: minor updates on choices of campaign - data/utils: implemented low pT muon corrections, remove fill_none - env: change to <3.11 for python version, add line for test_env.yml - workflows: fix bug for reading SFs, put HLT flag for triggered lepton - utils: add flag for triggered leptons - fix: remove correction for ttbar * feat : fixes on plotting code - plot: fix variances bug, additional text in the plot - plot: add custom xrange - util: fix some range of histograms - doc: add new features for plot * feat : fixes dr selection, njet & add under/overflow bin - plot: add plotting under/overflow bin options - plot: wrap errorband to a function - plot: add non-uniform rebinning - workflow: fix dr selection when using ak.all,masked identity is required - workflow: add nmujet & nsoftmu in ctag SFs workflows - worlfow : store njet correctly * fix : fix mc weight scaler & other minor bug fix - plot,utils: change the mcwei implementation - utils: change dr range & bugfix * feat : minor fixes plot: comment out flow option before new mplhep release plot: add custom ylabel & xerror workflow: return corrected jets utils: remove repeated puwei, correct MET filter * feat : minor suggestions from Denise CI,env: remove python 3.8 plot: add ylabel, xerr * fix: CI python version
- utils: fixes to adapt systematics - plot_util : fix for multiple coffea files - data: remove redundant files
…into MY_dev - remove redudant files, workflows - fix plotting code - update readme, examples for new developements Conflicts: .github/workflows/ctag_Wc_workflow.yml .github/workflows/test_workflow.yml .github/workflows/ttbar_SL_DL_workflow.yml .github/workflows/ttbar_workflow.yml .github/workflows/validation_workflow.yml .gitignore README.md env.yml plotting/comparison.py plotting/plotdataMC.py runner.py src/BTVNanoCommissioning/data/JME/2017_UL/jec_compiled.pkl.gz src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_RunBCD_V7_DATA_L3Absolute_AK4PFPuppi.jec.txt src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_V7_MC_L2L3Residual_AK4PFPuppi.jec.txt src/BTVNanoCommissioning/data/JME/Winter22Run3/jec_compiled.pkl.gz src/BTVNanoCommissioning/utils/AK4_parameters.py src/BTVNanoCommissioning/utils/correction.py src/BTVNanoCommissioning/utils/histogrammer.py src/BTVNanoCommissioning/utils/plot_utils.py src/BTVNanoCommissioning/utils/selection.py src/BTVNanoCommissioning/utils/xs_scaler.py src/BTVNanoCommissioning/workflows/ctag_DY_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_Wc_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_dileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_eDY_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_eWc_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_edileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_emdileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_ettsemilep_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_semileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ttbar_validation.py src/BTVNanoCommissioning/workflows/ttdilep_valid_sf.py src/BTVNanoCommissioning/workflows/ttsemilep_valid_sf.py src/BTVNanoCommissioning/workflows/validation.py
Conflicts: .github/workflows/plotting_test.yml .github/workflows/test_workflow.yml README.md env.yml plotting/comparison.py plotting/plotdataMC.py runner.py src/BTVNanoCommissioning/helpers/xsection.py src/BTVNanoCommissioning/utils/Configurator.py src/BTVNanoCommissioning/utils/plot_utils.py src/BTVNanoCommissioning/utils/xs_scaler.py testfile/btv_datamc.yml
|
Before merging this one, I merged the other PR #16 . Now there are a few minor conflicts with this one. Could you please resolve those, before I can merge. |
Conflicts: .github/workflows/ctag_DY_workflow.yml .github/workflows/ctag_Wc_workflow.yml .github/workflows/test_workflow.yml .github/workflows/ttbar_SL_DL_workflow.yml .github/workflows/ttbar_workflow.yml .gitignore README.md env.yml plotting/comparison.py plotting/plotdataMC.py runner.py src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_RunBCD_V7_DATA_L3Absolute_AK4PFPuppi.jec.txt src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_V7_MC_L2L3Residual_AK4PFPuppi.jec.txt src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_V7_MC_L2L3Residual_AK4PFchs.jec.txt src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_V7_MC_L2L3Residual_AK8PFPuppi.jec.txt src/BTVNanoCommissioning/data/JME/UL16/Summer19UL16APV_V7_MC_L2L3Residual_AK8PFchs.jec.txt src/BTVNanoCommissioning/data/JME/UL17/jec_compiled.pkl.gz src/BTVNanoCommissioning/data/JME/Winter22Run3/jec_compiled.pkl.gz src/BTVNanoCommissioning/data/PU/Winter22Run3/__init__.py src/BTVNanoCommissioning/helpers/definitions.py src/BTVNanoCommissioning/helpers/func.py src/BTVNanoCommissioning/utils/AK4_parameters.py src/BTVNanoCommissioning/utils/compile_jec.py src/BTVNanoCommissioning/utils/correction.py src/BTVNanoCommissioning/utils/histogrammer.py src/BTVNanoCommissioning/utils/plot_utils.py src/BTVNanoCommissioning/utils/selection.py src/BTVNanoCommissioning/workflows/ctag_DY_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_Wc_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_dileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_eDY_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_eWc_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_edileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_emdileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_ettsemilep_valid_sf.py src/BTVNanoCommissioning/workflows/ctag_semileptt_valid_sf.py src/BTVNanoCommissioning/workflows/ttbar_validation.py src/BTVNanoCommissioning/workflows/ttdilep_valid_sf.py src/BTVNanoCommissioning/workflows/ttsemilep_valid_sf.py src/BTVNanoCommissioning/workflows/validation.py
|
@andreypz ,apologize I merge with the squash and merge button... In case there's anything to be fix in this PR, let's have it in the following PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
integrate changes in feat: correctionlib(jsonpog-integration) implementation & fixes on actions cms-btv-pog/BTVNanoCommissioning#50 (plotting config, correctionlib developements)
utils: modify code to adapt the configuration for CoffeaRunner
utils: add systematic uncertainties
General plotting scripts #3 non-uniform binning is done
plots for under/overflow bins still in progress with mplhep, ratioplot would also integrated at some point feat : implement under/overflow function scikit-hep/mplhep#413