Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: correctionlib(jsonpog-integration) implementation & fixes on actions #50

Merged
merged 13 commits into from
Mar 15, 2023
Merged

feat: correctionlib(jsonpog-integration) implementation & fixes on actions #50

merged 13 commits into from
Mar 15, 2023

Conversation

Ming-Yan
Copy link
Collaborator

@Ming-Yan Ming-Yan commented Feb 2, 2023 •
edited
Loading

Correctionlib implementation

  • utils: make SFs loading compatible with correctionlib and custom inputs
  • utils: add example using correctionlib in AK4_parameter.py
  • utils: rename JEC producer
  • data: rename name of dictionaries fit with jsonpog-integration scheme
  • runner: limit campaign with particular choices, add isSyst flag for SF errors
  • utils&data: correction of low pT muon implemented(custom & correctionlib could use at the same time)
  • utils&workflows: add isHLT flag for triggered lepton with SFs

Workflows

  • workflows: implement correctionlib, reduce weights implementation on selected events only
  • workflows: make conditions for branch not exists in files, skip selection/not filled histogram(not filled the histograms if the branches not exist without error)
  • workflows: load SFs, lumimask in init
  • workflows(Wc/ctag_ttsemilep): fix selections same as VHccSF
  • workflow: fix dr selection when using ak.all,masked identity is required
  • workflow: add nmujet & nsoftmu in ctag SFs workflows
  • worlfow : store njet correctly
  • workflows: fix bug for reading SFs, put HLT flag for triggered lepton

Plot

  • plotting : fix error band on upper panel
  • plot: fix variances bug, additional text in the plot
  • plot: add custom xrange
  • plot: add plotting under/overflow bin options, include changes from histplot in mplhep
  • plot: wrap errorband to a function
  • plot: add non-uniform rebinning
  • plot,utils: change the mcwei implementation
  • plot: add cutstom y-label,xerror from Denise

Others

  • doc: add --recursive for git clone, add an example, add choices, new plotting options
  • doc: minor updates on choices of campaign
  • util : correct jetid selection for ULVHccSF
  • util: fix some range of histograms
  • util: change dr range & bugfix
  • utils: add flag for triggered leptons
  • runner: change the default scaleout workers with 3
  • runner: modify memory, disk usages adapt parsl/dask interfaces
  • data/utils: implemented low pT muon corrections, remove fill_none
  • fix: remove correction for ttbar

Environment & actions

Please check the actions in forked directory

@Ming-Yan
feat : add correctionlib (jsonpog-integration) and rename data(correc…
c79bf5f 
…tion files) directory

- wrap jsonpog as submodule
- rename directory name of data/ directory
@Ming-Yan
feat : correctionlib implementation & minor fixes
eefbe9b 
- utils: make SFs loading compatible with correctionlib and custom inputs
- utils: add example using correctionlib in AK4_parameter
- utils: rename JEC producer
- runner: limit campaign with particular choices, add isSyst flag
- doc:
- workflows: implement correctionlib, reduce weights implementation on selected events only
- workflows: make conditions for branch not exists in files, skip selection/not filled histogram
- workflows: load SFs, lumimask in __init__
- workflows(Wc): fix selections same as VHccSF(https://github.com/mondalspandan/VHcc-cTagSF/blob/UL/Analyzer/WcSelection.py)
- utils : correct jetid selection for UL
- utils : fix Wc soft lepton pT axis
- env : avoid python 3.11 for setting enviroment
- plotting : fix error band on upper panel
- CI: add submodules, ignore black in jsonpog-integration
@Ming-Yan
fix : fix issues for action updates
4073b94 
- reformat with black v23
- restrict setuptools version
- wf: fix mamba error, conda setup errors
- wf: add submodules
- wf: add correctionlib workflow
- wf: run workflow with futures
@Ming-Yan
merge with master
5ca1048
@Ming-Yan Ming-Yan requested a review from demuller February 2, 2023 14:57
@demuller demuller requested a review from AnnikaStein February 2, 2023 14:58
demuller
demuller suggested changes Feb 2, 2023
View reviewed changes
Copy link
Contributor

@demuller demuller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Ming-Yan thanks a lot for your PR! Besides the comments in-line, I have some more remarks:

  • It seems like some JEC files got lost/deleted in the renaming procedure of the UL folders?
  • Are we still in need of the UL BTV csv files and UL SF root files? As they should all now be taken from jsonpog?

.github/workflows/ctag_DY_workflow.yml Show resolved Hide resolved
.github/workflows/ttbar_SL_DL_workflow.yml Show resolved Hide resolved
.github/workflows/validation_workflow.yml Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Show resolved Hide resolved
setup.cfg Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/AK4_parameters.py Show resolved Hide resolved
src/BTVNanoCommissioning/utils/correction.py Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/correction.py Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/correction.py Show resolved Hide resolved
@Ming-Yan
fix: suggestions from Denise & fix semilep selection
776e60e 
- improve documentation & implemented suggestion from Denise
- make ctag_ttsemilep selection sync with Wc
AnnikaStein
AnnikaStein reviewed Feb 6, 2023
View reviewed changes
.github/workflows/validation_workflow.yml Outdated
@@ -13,14 +13,21 @@ jobs:
strategy:
max-parallel: 4
matrix:
python-version: ["3.8", "3.9", "3.10"]
python-version: [ "3.10", "3.8","3.9"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A quick heads-up on the different python versions, do we have a timeline for how long we want to support each version? Or if we always want to test all three currently? Can this be simplified after figuring out which sites are used by those running the workflows actually (like, with a survey) such that one does not need to provide backwards-compatibility to e.g. 3.8? I guess (from our other discussions) 3.9 seems to be the go-to version which also runs at lxplus.
This is not a strict requirement for this PR, but rather a comment (as it's listed here and we always try to optimize things, I thought I'd bring this up).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be best to stick with one python version indeed. Concerning lxplus: the BTVNanoCommissioning setup runs there successfully with python 3.10. I do not have a strong opinion on whether to support 3.9 or 3.10, python 3.8 might become too old soon though. We should also make sure that the installation readme mentions the correct python version for the setup

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, we could also just focus on two of the three versions (3.9/3.10), save 1/3 of the tests (time) and related to that save person power to make the setup compatible with 3.8, which could be spend on more detailed studies with the state-of-the-art versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ming-Yan please restrict the support to python 3.9 and 3.10 only as discussed above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and adjust the README accordingly (for setting up the conda environment)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have specific issues with py38 compatibility? Dropping support for versions before they reach EOL is rather radical and should have a good justification. Otherwise, if it's necessary to save core hours on actions dropping the intermediate versions is better because it's a fair assumption the code will work if it works on both the previous and following versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andrzejnovak! We do not have issues with python 3.8, but the idea was to only support 3.9 and 3.10 such that the amount of CI pipeline tests is reduced. I am fine with dropping 3.9 instead

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand the issue with multiple tests unless we do actually manage to hit some GHA limit. They run concurrently no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they do run in parallel, but sometimes, one of the 3 versions takes longer, which might be a hint for some GHA limit being reached?

@demuller
Copy link
Contributor

demuller commented Feb 7, 2023

Hi @Ming-Yan thanks a lot for your PR! Besides the comments in-line, I have some more remarks:

* It seems like some JEC files got lost/deleted in the renaming procedure of the UL folders?

* Are we still in need of the UL BTV csv files and UL SF root files? As they should all now be taken from jsonpog?

Hi @Ming-Yan, concerning the deleted JEC and JER files: is the deletion of those files in folder UL17_106X intended or is it the result of failed renaming?
We should also consider removing the old BTV csv files and lepton SF root files, wherever we have the correctionlib json available (which should be the case for UL)

@Ming-Yan
Copy link
Collaborator Author

Ming-Yan commented Feb 7, 2023 •
edited
Loading

Hi @Ming-Yan, concerning the deleted JEC and JER files: is the deletion of those files in folder UL17_106X intended or is it the result of failed renaming? We should also consider removing the old BTV csv files and lepton SF root files, wherever we have the correctionlib json available (which should be the case for UL)

Hi @demuller, indeed the UL17_106X and UL17 should have the same contents, it duplicates in the beginning.
But I think I can just remove it and leave the custom for Rereco17 and Winter22Run3 for other correction files(lepSFs, puweight, BTV SFs)

@Ming-Yan
fix : remove UL files & fix bug for NanoAOD
0cfa32d 
    - remove UL lepton SFs, PU, btag SFs
    - fix bug with filled discriminant for NanoAOD
AnnikaStein
AnnikaStein suggested changes Feb 8, 2023
View reviewed changes
Copy link
Contributor

@AnnikaStein AnnikaStein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I think these are all my comments for now, and of course thanks for the PR!
Some of the change requests I believe are really crucial to follow-up on, as they concern the SF application where we need accurate treatment across different taggers and flavours. Once these changes have been clearly cross-checked as proposed at the respective places and modifications documented (also for the others using this repo), I'll have another look.
Best,
A.

plotting/plotdataMC.py Outdated Show resolved Hide resolved
runner.py Show resolved Hide resolved
runner.py Outdated Show resolved Hide resolved
setup.cfg Outdated Show resolved Hide resolved
setup.cfg Outdated Show resolved Hide resolved
test_env.yml Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/correction.py Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/correction.py Outdated Show resolved Hide resolved
src/BTVNanoCommissioning/utils/selection.py Show resolved Hide resolved
.github/workflows/validation_workflow.yml Outdated
@@ -13,14 +13,21 @@ jobs:
strategy:
max-parallel: 4
matrix:
python-version: ["3.8", "3.9", "3.10"]
python-version: [ "3.10", "3.8","3.9"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, we could also just focus on two of the three versions (3.9/3.10), save 1/3 of the tests (time) and related to that save person power to make the setup compatible with 3.8, which could be spend on more detailed studies with the state-of-the-art versions.

@Ming-Yan
feat : suggestions from Annika
8f0dd07 
- runner: modify memory, disk usages adapt parsl/dask interfaces
- plot: fixing error band
- doc: minor updates on choices of campaign
- data/utils: implemented low pT muon corrections, remove fill_none
- env: change to <3.11 for python version, add line for test_env.yml
- workflows: fix bug for reading SFs, put HLT flag for triggered lepton
- utils: add flag for triggered leptons

- fix: remove correction for ttbar
@Ming-Yan
feat : fixes on plotting code
631659c 
- plot: fix variances bug, additional text in the plot
- plot: add custom xrange
- util: fix some range of histograms
- doc: add new features for plot
@Ming-Yan
feat : fixes dr selection, njet & add under/overflow bin
9a8f7fe 
- plot: add plotting under/overflow bin options
- plot: wrap errorband to a function
- plot: add non-uniform rebinning
- workflow: fix dr selection when using ak.all,masked identity is required
- workflow: add nmujet & nsoftmu in ctag SFs workflows
- worlfow : store njet correctly
@Ming-Yan
fix : fix mc weight scaler & other minor bug fix
5b2b1d8 
- plot,utils: change the mcwei implementation
- utils: change dr range & bugfix
@Ming-Yan
Copy link
Collaborator Author

Ming-Yan commented Mar 6, 2023

Demonstration of unequal binning

python plotting/plotdataMC.py -i "hist*.coffea" --com 13.6 --lumi 7650 -p ttsemilep_sf --ext "test" -v DeepJet_nsv --flow "show" --autorebin 0,1,2,5,8,10

image

@Ming-Yan
feat : minor fixes
1875eda 
plot: comment out flow option before new mplhep release
plot: add custom ylabel & xerror
workflow: return corrected jets
utils: remove repeated puwei, correct MET filter
demuller
demuller reviewed Mar 14, 2023
View reviewed changes
README.md Outdated Show resolved Hide resolved
demuller
demuller reviewed Mar 14, 2023
View reviewed changes
setup.cfg Show resolved Hide resolved
@demuller
Copy link
Contributor

Hi @AnnikaStein are there any open comments from your side concerning your review?

@AnnikaStein
Copy link
Contributor

Hi @AnnikaStein are there any open comments from your side concerning your review?

Just resolved all open items, looks good

@Ming-Yan
feat : minor suggestions from Denise
ed548f3 
CI,env: remove python 3.8
plot: add ylabel, xerr
@demuller demuller merged commit ca74d50 into cms-btv-pog:master Mar 15, 2023
@Ming-Yan Ming-Yan mentioned this pull request Mar 22, 2023
Merged
Ming-Yan added a commit to cms-rwth/CoffeaRunner that referenced this pull request Mar 23, 2023
@Ming-Yan @demuller
@pmatorras @andreypz
Merge commits
f7226d6 
* fix : minor fixes for runner.py and comparison.py; add JERC to MET propagation (cms-btv-pog#39)

* feat : add PU files

- add pu files, adapt correction will available dicts
- add Muon_RunDv1 to data json files
- improve file splitting for dask/lxplus
- add autorebin for plotting tools
- remove redudant puwei file

fix : python3.7->3.10 in pipeline

* fix : minor fix

- update readme
- bugfix for dileptt,semileptt workflow
- add init to PU directory
- update descriptions

* minor fixes for runner.py and comparison.py

* formatting with black

* fix: add JEC to MET and plot format

- add JEC to MET
- plot format

* fix: set tail of PU to 1

* fix: minor fix on runner.py

Co-authored-by: Ming-Yan <[email protected]>

* patch for the daskoutput name (cms-btv-pog#40)

* patch for the daskoutput name

* reformatted runner (hope with the right black version

* feat : update plotting scripts and add track variables (cms-btv-pog#41)

* feat : improve plot scripts and add new variables

- Add PFCands to defintions
- Correct the uncertainty calculations for ratio panels
- include MC stat error band
- improve DeepJet variable binning and labels

* feat : add track related info

- add track related info
- fix minor comparison

* fix :  minor fixs

- modify name for track variables
- fix jet pt axis
- fix cut in ctag_semileptt_sf and ttcom workflow
- move default errtype to plot_util

* fix : fix the file merging in the plots and other minor fixes

- move black and flake8 to python 3.10
- change the condition PFCands variables storing supported with higher version(not Reco17_94X)
- minor fixes on plotting scripts

* feat : minor fixes

* fix: update python version in setup.cfg

* fix: fix setup.cfg for CI pipelines (cms-btv-pog#42)

* fix setup.cfg

* fix: move setup to coffea0.7.20

* local changes

* Fix : MC merging in plotting scripts and move to PuppiMET (cms-btv-pog#43)

* fix : MC merging in plotting scripts and move to PuppiMET
- move PFMET to Puppi MET for run3
- extend xrootdtimeout for local executor
- fix plotting scripts with merging MC files with same dataset name
- fix axes label
- fix comparison scripts panel
- splitted OS-SS for Wc in plotting scripts

* feat : auto-scale x-axis

- update x-axis range for plots
- fix missing variables in defintion
- change lepton dz range, dr range

* fix: minor fix for plotting scripts

* fix : fix MET correction and plotting scripts (cms-btv-pog#44)

* fix : fix MET correction and plotting scripts

- add MetUnclustEnUpDeltaY,MetUnclustEnUpDeltaX to PuppiMET for MET correction
- fix dxy/dz axes for leptons, have zooming version for lepton with QCD veto
- fix spacing for the axis labels

* fix:minor fix

* my changes to compare script

* Remove stuff that does not work

* fix: plotting style improvements; future warnings in runner.py (cms-btv-pog#45)

* fix: plotting style improvements; took care of future warnings in runner.py; added link to original code in runner.py

* removed deprecated `extra` in lxplus setup

* liniting with black

* fixes for plotting

* fix label for btag discriminator; added prob distributions

* protection for not adding SF axis to prob distributions

* add printout for produced plot

* flake8 fix

* fix: add missing xsection & add protection for missing xsection

Co-authored-by: Ming-Yan Lee <[email protected]>

* added prob var protection to second if condition in ctag ttbar workflows (cms-btv-pog#46)

* bug fix for ctag emu ttbar dilep workflow (cms-btv-pog#47)

* add cross section

* trying to fix the plotting scripts

* Run black and change env.yml file name

* minor fixes

* worflow tester

* fix: setup debug mode for plotting script

* feat : more options for plotting scripts

- add additional cross-section rescaler for sample set
- add mergemap for comparison script
- minor fix for debug mode

* re-added worker port condition for lxplus; bug for xlabel in comparison.py (cms-btv-pog#48)

* linting runner.py

* linting comparison.py

* bugfix jet to PFcand matching for ttbar dilep

* feat: add plotting yaml

fix : plotting CI, polish readme, reformat configurator

- create reproducible config file (`.py`) under output directory
- add default `run_options` in configurator
- move systematic as common methods
- polish readme
- fix plotting CI name

* fix : fix additional scaler bug

fix: limit should be None in default

* feat: add split

* feat : add correctionlib (jsonpog-integration) and rename data(correction files) directory

- wrap jsonpog as submodule
- rename directory name of data/ directory

* feat : correctionlib implementation & minor fixes

- utils: make SFs loading compatible with correctionlib and custom inputs
- utils: add example using correctionlib in AK4_parameter
- utils: rename JEC producer
- runner: limit campaign with particular choices, add isSyst flag
- doc:
- workflows: implement correctionlib, reduce weights implementation on selected events only
- workflows: make conditions for branch not exists in files, skip selection/not filled histogram
- workflows: load SFs, lumimask in __init__
- workflows(Wc): fix selections same as VHccSF(https://github.com/mondalspandan/VHcc-cTagSF/blob/UL/Analyzer/WcSelection.py)
- utils : correct jetid selection for UL
- utils : fix Wc soft lepton pT axis
- env : avoid python 3.11 for setting enviroment
- plotting : fix error band on upper panel
- CI: add submodules, ignore black in jsonpog-integration

* fix: updated 13 TeV single top and ttbar cross sections (cms-btv-pog#49)

* updated 13 TeV single top and ttbar cross sections

* black v23 does not like formatting of black v22

* fix : fix issues for action updates

- reformat with black v23
- restrict setuptools version
- wf: fix mamba error, conda setup errors
- wf: add submodules
- wf: add correctionlib workflow
- wf: run workflow with futures

* fix: suggestions from Denise & fix semilep selection

- improve documentation & implemented suggestion from Denise
- make ctag_ttsemilep selection sync with Wc

* fix : remove UL files & fix bug for NanoAOD

    - remove UL lepton SFs, PU, btag SFs
    - fix bug with filled discriminant for NanoAOD

* feat : suggestions from Annika

- runner: modify memory, disk usages adapt parsl/dask interfaces
- plot: fixing error band
- doc: minor updates on choices of campaign
- data/utils: implemented low pT muon corrections, remove fill_none
- env: change to <3.11 for python version, add line for test_env.yml
- workflows: fix bug for reading SFs, put HLT flag for triggered lepton
- utils: add flag for triggered leptons

- fix: remove correction for ttbar

* feat: add ignore

* feat:fixes

* feat: correctionlib(jsonpog-integration) implementation & fixes on actions (cms-btv-pog#50)

* feat : add correctionlib (jsonpog-integration) and rename data(correction files) directory

- wrap jsonpog as submodule
- rename directory name of data/ directory

* feat : correctionlib implementation & minor fixes

- utils: make SFs loading compatible with correctionlib and custom inputs
- utils: add example using correctionlib in AK4_parameter
- utils: rename JEC producer
- runner: limit campaign with particular choices, add isSyst flag
- doc:
- workflows: implement correctionlib, reduce weights implementation on selected events only
- workflows: make conditions for branch not exists in files, skip selection/not filled histogram
- workflows: load SFs, lumimask in __init__
- workflows(Wc): fix selections same as VHccSF(https://github.com/mondalspandan/VHcc-cTagSF/blob/UL/Analyzer/WcSelection.py)
- utils : correct jetid selection for UL
- utils : fix Wc soft lepton pT axis
- env : avoid python 3.11 for setting enviroment
- plotting : fix error band on upper panel
- CI: add submodules, ignore black in jsonpog-integration

* fix : fix issues for action updates

- reformat with black v23
- restrict setuptools version
- wf: fix mamba error, conda setup errors
- wf: add submodules
- wf: add correctionlib workflow
- wf: run workflow with futures

* fix: suggestions from Denise & fix semilep selection

- improve documentation & implemented suggestion from Denise
- make ctag_ttsemilep selection sync with Wc

* fix : remove UL files & fix bug for NanoAOD

    - remove UL lepton SFs, PU, btag SFs
    - fix bug with filled discriminant for NanoAOD

* feat : suggestions from Annika

- runner: modify memory, disk usages adapt parsl/dask interfaces
- plot: fixing error band
- doc: minor updates on choices of campaign
- data/utils: implemented low pT muon corrections, remove fill_none
- env: change to <3.11 for python version, add line for test_env.yml
- workflows: fix bug for reading SFs, put HLT flag for triggered lepton
- utils: add flag for triggered leptons

- fix: remove correction for ttbar

* feat : fixes on plotting code

- plot: fix variances bug, additional text in the plot
- plot: add custom xrange
- util: fix some range of histograms
- doc: add new features for plot

* feat : fixes dr selection, njet & add under/overflow bin

- plot: add plotting under/overflow bin options
- plot: wrap errorband to a function
- plot: add non-uniform rebinning
- workflow: fix dr selection when using ak.all,masked identity is required
- workflow: add nmujet & nsoftmu in ctag SFs workflows
- worlfow : store njet correctly

* fix : fix mc weight scaler & other minor bug fix
- plot,utils: change the mcwei implementation
- utils: change dr range & bugfix

* feat : minor fixes

plot: comment out flow option before new mplhep release
plot: add custom ylabel & xerror
workflow: return corrected jets
utils: remove repeated puwei, correct MET filter

* feat : minor suggestions from Denise

CI,env: remove python 3.8
plot: add ylabel, xerr

* fix: CI python version

* fix: minor fixes for plot

---------

Co-authored-by: Denise Müller <[email protected]>
Co-authored-by: Pablo Matorras <[email protected]>
Co-authored-by: Andrey Pozdnyakov <[email protected]>
Ming-Yan pushed a commit to cms-rwth/CoffeaRunner that referenced this pull request May 9, 2023
@andreypz @Ming-Yan
feat : add full run2 camapign & several fixes on SFs
7ad5ac7 
- runner: add test option(run iterative with 1 file, 1 chunk)
- runner: add begin/ stop for splited file case
- config: add full run2 config example from Hc analysis
- example: add example workflow (test_wf) with config/example.py wrapped latest corrections & systematics implementation
- scripts: update fetch.py script (avoid merge data into one dict) & put error for empty list
- data: add run3 files back
- utils: add JEC shifts, rochester muon correction ttbar pT reweighting, pileup SFs
- utils: make SFs reader capable for more than 1 object
- utils: update collated scripts
- helper: add xsection for H+c

=============old commits==================
 # This is a combination of 2 commits.

* fix: previous fixes

- plot: add additional cross-section rescaler for sample set,  add mergemap for comparison script
- plot : fixes from Andrey in #16
- utils: add cross section template
- utils: trying to fix the plotting scripts
- utils: setup debug mode for plotting script

* Development from BTV

- minor fixes for runner.py and comparison.py; add JERC to MET propagation (cms-btv-pog#39)
- patch for the daskoutput name (cms-btv-pog#40)
- update plotting scripts and add track variables (cms-btv-pog#41)
- fix setup.cfg for CI pipelines (cms-btv-pog#42)
- MC merging in plotting scripts and move to PuppiMET (cms-btv-pog#43)
- plotting style improvements; future warnings in runner.py (cms-btv-pog#45)
- re-added worker port condition for lxplus; bug for xlabel in comparison.py (cms-btv-pog#48)
- updated 13 TeV single top and ttbar cross sections (cms-btv-pog#49)
- correctionlib(jsonpog-integration) implementation & fixes on actions (cms-btv-pog#50)
Ming-Yan pushed a commit to cms-rwth/CoffeaRunner that referenced this pull request May 9, 2023
@andreypz @Ming-Yan
feat : add full run2 camapign & several fixes on SFs
312a8ec 
- runner: add test option(run iterative with 1 file, 1 chunk)
- runner: add begin/ stop for splited file case
- config: add full run2 config example from Hc analysis
- example: add example workflow (test_wf) with config/example.py wrapped latest corrections & systematics implementation
- scripts: update fetch.py script (avoid merge data into one dict) & put error for empty list
- data: add run3 files back
- utils: add JEC shifts, rochester muon correction ttbar pT reweighting, pileup SFs
- utils: make SFs reader capable for more than 1 object
- utils: update collated scripts
- helper: add xsection for H+c

=============old commits==================
 # This is a combination of 2 commits.

* fix: previous fixes

- plot: add additional cross-section rescaler for sample set,  add mergemap for comparison script
- plot : fixes from Andrey in #16
- utils: add cross section template
- utils: trying to fix the plotting scripts
- utils: setup debug mode for plotting script

* Development from BTV

- minor fixes for runner.py and comparison.py; add JERC to MET propagation (cms-btv-pog#39)
- patch for the daskoutput name (cms-btv-pog#40)
- update plotting scripts and add track variables (cms-btv-pog#41)
- fix setup.cfg for CI pipelines (cms-btv-pog#42)
- MC merging in plotting scripts and move to PuppiMET (cms-btv-pog#43)
- plotting style improvements; future warnings in runner.py (cms-btv-pog#45)
- re-added worker port condition for lxplus; bug for xlabel in comparison.py (cms-btv-pog#48)
- updated 13 TeV single top and ttbar cross sections (cms-btv-pog#49)
- correctionlib(jsonpog-integration) implementation & fixes on actions (cms-btv-pog#50)
Ming-Yan pushed a commit to cms-rwth/CoffeaRunner that referenced this pull request May 9, 2023
@andreypz @Ming-Yan
feat : add full run2 camapign & several fixes on SFs
646d460 
- runner: add test option(run iterative with 1 file, 1 chunk)
- runner: add begin/ stop for splited file case
- config: add full run2 config example from Hc analysis
- example: add example workflow (test_wf) with config/example.py wrapped latest corrections & systematics implementation
- scripts: update fetch.py script (avoid merge data into one dict) & put error for empty list
- data: add run3 files back
- utils: add JEC shifts, rochester muon correction ttbar pT reweighting, pileup SFs
- utils: make SFs reader capable for more than 1 object
- utils: update collated scripts
- helper: add xsection for H+c

=============old commits==================
 # This is a combination of 2 commits.

* fix: previous fixes

- plot: add additional cross-section rescaler for sample set,  add mergemap for comparison script
- plot : fixes from Andrey in #16
- utils: add cross section template
- utils: trying to fix the plotting scripts
- utils: setup debug mode for plotting script

* Development from BTV

- minor fixes for runner.py and comparison.py; add JERC to MET propagation (cms-btv-pog#39)
- patch for the daskoutput name (cms-btv-pog#40)
- update plotting scripts and add track variables (cms-btv-pog#41)
- fix setup.cfg for CI pipelines (cms-btv-pog#42)
- MC merging in plotting scripts and move to PuppiMET (cms-btv-pog#43)
- plotting style improvements; future warnings in runner.py (cms-btv-pog#45)
- re-added worker port condition for lxplus; bug for xlabel in comparison.py (cms-btv-pog#48)
- updated 13 TeV single top and ttbar cross sections (cms-btv-pog#49)
- correctionlib(jsonpog-integration) implementation & fixes on actions (cms-btv-pog#50)
andreypz added a commit to cms-rwth/CoffeaRunner that referenced this pull request May 16, 2023
@Ming-Yan @andreypz
feat : add full run2 campaign config & several fixes on SFs (#18)
8091bee 
* feat : add full run2 camapign & several fixes on SFs

- runner: add test option(run iterative with 1 file, 1 chunk)
- runner: add begin/ stop for splited file case
- config: add full run2 config example from Hc analysis
- example: add example workflow (test_wf) with config/example.py wrapped latest corrections & systematics implementation
- scripts: update fetch.py script (avoid merge data into one dict) & put error for empty list
- data: add run3 files back
- utils: add JEC shifts, rochester muon correction ttbar pT reweighting, pileup SFs
- utils: make SFs reader capable for more than 1 object
- utils: update collated scripts
- helper: add xsection for H+c

=============old commits==================
 # This is a combination of 2 commits.

* fix: previous fixes

- plot: add additional cross-section rescaler for sample set,  add mergemap for comparison script
- plot : fixes from Andrey in #16
- utils: add cross section template
- utils: trying to fix the plotting scripts
- utils: setup debug mode for plotting script

* Development from BTV

- minor fixes for runner.py and comparison.py; add JERC to MET propagation (cms-btv-pog#39)
- patch for the daskoutput name (cms-btv-pog#40)
- update plotting scripts and add track variables (cms-btv-pog#41)
- fix setup.cfg for CI pipelines (cms-btv-pog#42)
- MC merging in plotting scripts and move to PuppiMET (cms-btv-pog#43)
- plotting style improvements; future warnings in runner.py (cms-btv-pog#45)
- re-added worker port condition for lxplus; bug for xlabel in comparison.py (cms-btv-pog#48)
- updated 13 TeV single top and ttbar cross sections (cms-btv-pog#49)
- correctionlib(jsonpog-integration) implementation & fixes on actions (cms-btv-pog#50)

* feat: add array output to root files

---------

Co-authored-by: Andrey Pozdnyakov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrzejnovak andrzejnovak andrzejnovak left review comments

@AnnikaStein AnnikaStein AnnikaStein approved these changes

@demuller demuller demuller approved these changes

Assignees
No one assigned
Labels
CI core runner
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Ming-Yan @demuller @AnnikaStein @andrzejnovak