Issue/4/evaluation rearrangement #61
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…L into issue/7/creation-baseline
…magnitude normalizing flow. Demo for the creation class.
Issue/7/creation baseline
… creation demo, and made a plot of some of the pdfs. Removed redundancies in if states in some of the scripts. Removed some old plots.
…le in redshift_flow_example.ipynb
…again to make sure everything is fixed.
Issue/7/creation baseline
…ng on adding tests to test_creator.py.
…sed with normalizing flows we are building.
added 21 commits
June 21, 2021 16:51
…rdingly to ensure consistency in the i/o from rail estination to rail evaluation.
…py, in examples directory where there is no name conflict.
… and compute PIT (to be continued)
…imized, just working).
…s added automatically by PyCharm.
…to inform that it is based on a discrete distribution, as required by scipy.stats.entropy function.
return output as standardized named tuple as all other metrics.
…the new structure.
… markdown cells accordingly.
|
@aimalz this branch is ready to merge with the evaluation baseline. I already moved the main.py to the examples directory and tested it there. |
|
I'm not sure if I'm the one supposed to press the merge button. Just in case, I'll wait for people to take a look at the code changes added by me this week. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We took a stab at rearranging the evaluation subpackage for the Golden Spike v1 release. The initial goal of this work was to isolate the functionality that we hope to ultimately call through qp, hopefully making that future transition a little easier. More of the changes, however, ended up being related to enforcing some of the API conventions established in the estimation subpackage, specifically those affecting contributors adding new metrics and those affecting how we as experimenters will use it for the baseline tests of the Golden Spike. We should converge on these before getting too far into unit testing.
We got as far as the PIT and metrics thereof (modulo a snag with unexpected behavior of qp) and are working on a demo notebook as well as a similar rearrangement for the CDE Loss -- changes will indeed have to be propagated to the plotting functionality as well, but that can be done back in the primary evaluation branch, along with the unit tests, once this patch is merged in.