Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Implementation of two-sided tests with lower and upper bounds on the POI #45

Merged
merged 32 commits into from
Feb 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5b27c8b
update use
jackaraz Dec 20, 2024
97a851a
allow for negative poi in UL calculation
jackaraz Jan 29, 2025
c849d21
allow upto python 3.13
jackaraz Jan 29, 2025
f73998c
bump the version
jackaraz Jan 29, 2025
e2da2d0
move default_pdf to default
jackaraz Jan 29, 2025
f2225b9
move default_pdf to default
jackaraz Jan 29, 2025
ecc65ed
include latest version of python 3.12
jackaraz Jan 29, 2025
1230a42
change usage to normal dist instead of clipped normal
jackaraz Jan 29, 2025
ed4076d
update tests
jackaraz Jan 31, 2025
d633c7d
improve wrapper
jackaraz Jan 31, 2025
c379572
test poisson
jackaraz Jan 31, 2025
dfa9096
extend poiss tests
jackaraz Jan 31, 2025
ba63d6b
remove redundant call
jackaraz Jan 31, 2025
3effc7c
simplify
jackaraz Feb 1, 2025
8fa7218
update for numeric stability
jackaraz Feb 1, 2025
79675c1
add logcdf
jackaraz Feb 1, 2025
fc905c9
bugfix
jackaraz Feb 1, 2025
cbfc515
add analytic test
jackaraz Feb 3, 2025
d5d935f
add failsafe
jackaraz Feb 3, 2025
0e23a27
generalise POI UL function
jackaraz Feb 4, 2025
75dba0d
bump the version
jackaraz Feb 4, 2025
e67928a
first iteration on two sided limits
jackaraz Feb 7, 2025
3d08109
adjust CLs
jackaraz Feb 7, 2025
7e5843d
first iteration of the `chi2_test` function
jackaraz Feb 8, 2025
49e3039
improve documentation
jackaraz Feb 8, 2025
3ff81ab
add chi2-test test
jackaraz Feb 9, 2025
523c2ea
clean the code for the first stage of v0.2
jackaraz Feb 20, 2025
5826708
prepare CI/CD for deployment
jackaraz Feb 20, 2025
45e0dbc
remove beta
jackaraz Feb 20, 2025
2c4200f
update annotation for linux
jackaraz Feb 20, 2025
85d6b34
refine tests
jackaraz Feb 20, 2025
c3b5d8e
refine tolerance
jackaraz Feb 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .github/workflows/check+build+deploy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
name: Check/Build/Deploy

on:
release:
types: [published]
push:
branches: ["releases/**"]
workflow_dispatch:

jobs:
check:
name: OS ${{ matrix.os }}, Python ${{ matrix.python-version }}
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: [ubuntu-latest, macos-latest]

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install pytest pytest-cov
python -m pip install -e .
- name: Test with pytest
run: |
python -c 'import spey;spey.about()'
pytest --cov=spey tests/*py #--cov-fail-under 99

build:
name: Build wheel
runs-on: ubuntu-latest
needs: check
steps:
- name: Checkout git repository
uses: actions/checkout@v4

- name: Build Spey wheel
run: |
python -m pip install --upgrade pip
python -m pip install wheel
python setup.py bdist_wheel

- name: Upload wheels as artifacts
uses: actions/upload-artifact@v4
with:
name: wheel
path: ./dist/*.whl

deploy:
if: ${{ !github.event.release.prerelease && !contains(github.ref, '-rc') && !contains(github.ref, '-beta') && !contains(github.ref, '-alpha') }}
runs-on: ubuntu-latest
needs: build
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
pattern: wheel
merge-multiple: true
path: dist

- name: Debug
run: ls -l dist

- uses: pypa/[email protected]
with:
user: ${{ secrets.TWINE_USERNAME }}
password: ${{ secrets.TWINE_PSSWRD }}
repository-url: https://pypi.org/project/spey/
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12.6"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: [ubuntu-latest, macos-latest]

steps:
Expand Down
6 changes: 3 additions & 3 deletions .zenodo.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"description": "smooth inference for reinterpretation studies",
"license": "MIT",
"title": "SpeysideHEP/spey: v0.1.11",
"version": "v0.1.11",
"title": "SpeysideHEP/spey: v0.2.0",
"version": "v0.2.0",
"upload_type": "software",
"creators": [
{
Expand Down Expand Up @@ -31,7 +31,7 @@
},
{
"scheme": "url",
"identifier": "https://github.com/SpeysideHEP/spey/tree/v0.1.11",
"identifier": "https://github.com/SpeysideHEP/spey/tree/v0.2.0",
"relation": "isSupplementTo"
},
{
Expand Down
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,13 @@ Finally, the name "Spey" originally comes from the Spey River, a river in the mi

| Accessor | Description |
| --------------------------------------- | ---------------------------------------------------------------------------- |
| ``"default_pdf.uncorrelated_background"`` | Constructs uncorrelated multi-bin statistical model. |
| ``"default_pdf.correlated_background"`` | Constructs correlated multi-bin statistical model with Gaussian nuisances. |
| ``"default_pdf.third_moment_expansion"`` | Implements the skewness of the likelihood by using third moments. |
| ``"default_pdf.effective_sigma"`` | Implements the skewness of the likelihood by using asymmetric uncertainties. |
| ``"default_pdf.poisson"`` | Implements simple Poisson-based likelihood without uncertainties. |
| ``"default_pdf.normal"`` | Implements Normal distribution. |
| ``"default_pdf.multivariate_normal"`` | Implements Multivariate normal distribution. |
| ``"default.uncorrelated_background"`` | Constructs uncorrelated multi-bin statistical model. |
| ``"default.correlated_background"`` | Constructs correlated multi-bin statistical model with Gaussian nuisances. |
| ``"default.third_moment_expansion"`` | Implements the skewness of the likelihood by using third moments. |
| ``"default.effective_sigma"`` | Implements the skewness of the likelihood by using asymmetric uncertainties. |
| ``"default.poisson"`` | Implements simple Poisson-based likelihood without uncertainties. |
| ``"default.normal"`` | Implements Normal distribution. |
| ``"default.multivariate_normal"`` | Implements Multivariate normal distribution. |
| ``"pyhf.uncorrelated_background"`` | Uses uncorrelated background functionality of pyhf (see [``spey-phyf`` plugin](https://github.com/SpeysideHEP/spey-pyhf)). |
| ``"pyhf"`` | Uses generic likelihood structure of pyhf (see [``spey-phyf`` plugin](https://github.com/SpeysideHEP/spey-pyhf)) |

Expand All @@ -75,17 +75,17 @@ likelihood prescriptions which can be checked via `AvailableBackends` function
```python
import spey
print(spey.AvailableBackends())
# ['default_pdf.correlated_background',
# 'default_pdf.effective_sigma',
# 'default_pdf.third_moment_expansion',
# 'default_pdf.uncorrelated_background']
# ['default.correlated_background',
# 'default.effective_sigma',
# 'default.third_moment_expansion',
# 'default.uncorrelated_background']
```

Using ``'default_pdf.uncorrelated_background'`` one can simply create single or multi-bin
Using ``'default.uncorrelated_background'`` one can simply create single or multi-bin
statistical models:

```python
pdf_wrapper = spey.get_backend('default_pdf.uncorrelated_background')
pdf_wrapper = spey.get_backend('default.uncorrelated_background')

data = [1]
signal_yields = [0.5]
Expand Down Expand Up @@ -147,7 +147,7 @@ collaborations to estimate the difference from the Standard Model rather than th
One can play the same game using the same backend for multi-bin histograms as follows;

```python
pdf_wrapper = spey.get_backend('default_pdf.uncorrelated_background')
pdf_wrapper = spey.get_backend('default.uncorrelated_background')

data = [36, 33]
signal = [12.0, 15.0]
Expand Down
34 changes: 25 additions & 9 deletions docs/bib/cited.bib
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
@article{Ghosh:2024puc,
author = {Ghosh, Kirtiman and Huitu, Katri and Sahu, Rameswar},
title = {{Revisiting Universal Extra-Dimension Model with Gravity Mediated Decays}},
eprint = {2412.09344},
archiveprefix = {arXiv},
primaryclass = {hep-ph},
month = {12},
year = {2024},
journal = {}
}


@article{Goodsell:2024aig,
author = {Goodsell, Mark D.},
title = {{HackAnalysis 2: A powerful and hackable recasting tool}},
Expand All @@ -15,25 +27,29 @@ @article{Agin:2024yfs
eprint = {2404.12423},
archiveprefix = {arXiv},
primaryclass = {hep-ph},
month = {4},
year = {2024},
journal = {}
doi = {10.1140/epjc/s10052-024-13594-9},
journal = {Eur. Phys. J. C},
volume = {84},
number = {11},
pages = {1218},
year = {2024}
}


@article{Feike:2024zfz,
archiveprefix = {arXiv},
author = {Feike, Alexander and Fiaschi, Juri and Fuks, Benjamin and Klasen, Michael and Puck Neuwirth, Alexander},
title = {{Combination and reinterpretation of LHC SUSY searches}},
eprint = {2403.11715},
month = {3},
archiveprefix = {arXiv},
primaryclass = {hep-ph},
reportnumber = {MS-TP-23-49},
title = {{Combination and Reinterpretation of LHC SUSY Searches}},
year = {2024},
journal = {}
doi = {10.1007/JHEP07(2024)122},
journal = {JHEP},
volume = {07},
pages = {122},
year = {2024}
}


@article{Elmer:2023wtr,
author = {Elmer, Nina and Madigan, Maeve and Plehn, Tilman and Schmal, Nikita},
title = {{Staying on Top of SMEFT-Likelihood Analyses}},
Expand Down
2 changes: 1 addition & 1 deletion docs/comb.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Let us first import all the necessary packages and construct the data (please ad
>>> models = {}
>>> # loop overall data
>>> for data in example_data["data"]:
>>> pdf_wrapper = spey.get_backend("default_pdf.uncorrelated_background")
>>> pdf_wrapper = spey.get_backend("default.uncorrelated_background")

>>> stat_model = pdf_wrapper(
... signal_yields=data["signal_yields"],
Expand Down
4 changes: 2 additions & 2 deletions docs/exclusion.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ The `allow_negative_signal` keyword controls which test statistic is used and re

For complex statistical models, optimizing the likelihood can be challenging and depends on the choice of optimizer. Spey uses SciPy for optimization and fitting tasks. Any additional keyword arguments not explicitly covered in the {func}`~spey.StatisticalModel.exclusion_confidence_level` function description are passed directly to the optimizer, allowing users to customize its behavior through the interface.

Below we compare the exclusion limits computed with each approach. This comparisson uses normal distribution for the likelihood ([`default_pdf.normal`](#normal)) background yields are set to $n_b$, uncertainties are shown with $\sigma$ and observations are given with $n$.
Below we compare the exclusion limits computed with each approach. This comparisson uses normal distribution for the likelihood ([`default.normal`](#normal)) background yields are set to $n_b$, uncertainties are shown with $\sigma$ and observations are given with $n$.

```{figure} ./figs/comparisson_observed.png
---
Expand All @@ -90,7 +90,7 @@ exclusion limit calculator comparisson for observed p-values
```{code-cell} ipython3
import spey

stat_wrapper = spey.get_backend("default_pdf.normal")
stat_wrapper = spey.get_backend("default.normal")
statistical_model = stat_wrapper(
signal_yields=[3.0],
background_yields=[2.0],
Expand Down
Loading
Loading