Create Branch review table #6648

matentzn · 2023-09-15T08:35:33Z

@cmungall Here is where SPARQL really suxx. If you can provide me a SQL query to deal with this specific issue, I will start switching to OAK for table reports.

UPDATE:

Using oaklib:

reports/mondo-curation-branch-review.tsv:
	python src/scripts/branch_review.py create-review-table -o $@ -f obsoletion_terms.tsv -B branch_ids.tsv

OR

reports/mondo-curation-branch-review.tsv:
	python src/scripts/branch_review.py create-review-table -o reports.tsv -f obsoletion_terms.tsv -b MONDO:0005151

Pseudo-code

all_branches = [....list of all branches ...]
branch_dict = { branch_id: branch_id.get_descendants() }

def create_review_table(branch_id, obsoletion_candidate_ids):
    affected_mondo_ids = id.get_direct_children() for id in obsoletion_candidate_ids #these are the classes that we need to review.
    
    for affected_id in affected_mondo_ids:
        # FOR THIS ANALYSIS ONLY CONSIDER MONDO IDS, NOT BFO ETC AS PARENTS
        parents = affected_id.get_direct_parents()
        parents.remove(obsoletion_candidate_ids)
        parents_inside_branch = intersection(parents, branch_dict[branch_id])
        parents_outside_branch = parents.remove(parents_inside_branch)
        ancestors = affected_id.get_ancestors()
        ancestors_inside_branch = intersection(ancestors, branch_dict[branch_id])
        ancestors_outside_branch = parents.remove(ancestors_inside_branch)
        
        if no ancestors: affected_status = ORPHANED
        elif no ancestors_inside_branch = LEFT_THIS_BRANCH
        elif ???? = LEFT_OTHER_BRANCH <- I DONT KNOW HOW TO WRITE THIS BUT LETS DO THIS AFTER EVERYTHING ELSE. FOR NOW JUST: IF ANY(ancestor in all_branches) 
        elif parents_inside_branch = RETAINS_PARENT

        else : RETAINS_ANCESTOR
        
        data = [affected_id, labels(parents_inside_branch), labels(parents_outside_branch), labels(ancestors_inside_branch), labels(ancestors_outside_branch), affected_status] 

def labels(curies):
    return [ f"{c.label} ({c})" for c in curies ]

…iative/mondo into branch-review-table

matentzn

I made some, but not all comments, as it's hard to read the code now that the affected entity variable has been introduced.

Can you rename all the variables to say affected_entity instead of obsoletion candidate where it makes sense? And also, instead of "filtered" say how, like "non_obsolete_parents". I will Check again tomorrow

src/scripts/branch_review.py

matentzn · 2023-09-26T20:13:30Z

src/scripts/branch_review.py

+
+            if len(affected_mondo_ids) > 0:
+                for obsoletion_candidate in affected_mondo_ids:
+                    parent_of_obsolete_candidate = get_parent_from_relations(


Suggested change

parent_of_obsolete_candidate = get_parent_from_relations(

parents_of_affected = get_parent_from_relations(

matentzn · 2023-09-26T20:16:59Z

src/scripts/branch_review.py

+                        for x in parent_of_obsolete_candidate
+                        if x not in affected_mondo_ids
+                    ]
+                    filtered_parent_of_obsolete_candidate = [


Same variAble name as above?

matentzn · 2023-09-26T20:19:35Z

src/scripts/branch_review.py

+                    filtered_parent_of_obsolete_candidate = [
+                        x
+                        for x in parent_of_obsolete_candidate
+                        if x not in affected_mondo_ids


I don't think this filter is needed

This is needed for lines below. I think my understanding is faulty here and may need clarification.

matentzn · 2023-09-26T20:20:31Z

src/scripts/branch_review.py

+                    filtered_ancestor_of_obsolete_candidate = [
+                        x
+                        for x in ancestors_of_obsolete_candidate
+                        if x not in affected_mondo_ids and x.startswith("MONDO:")


Also not innobsoletion candidates

matentzn · 2023-09-26T20:23:52Z

src/scripts/branch_review.py

+                    parents_inside_branch = [
+                        i
+                        for i in parents_inside_branch
+                        if i not in filtered_parent_of_obsolete_candidate


Filter Not needed

matentzn · 2023-09-26T20:24:59Z

src/scripts/branch_review.py

+                    parents_outside_branch = [
+                        i
+                        for i in parents_outside_branch
+                        if i not in filtered_parent_of_obsolete_candidate


hrshdhgd · 2023-09-26T22:58:43Z

I committed just refactors to variable names and some list <=> set

sabrinatoro · 2023-09-29T18:28:17Z

What is needed:

report direct children of the obsoletion candidates
you can focus on one branch at a time. Branches can be found here. We are starting with MONDO:0005151 - endocrine system disorder
since obsoletion candidates can be at different level within a branch, we need to know whether the direct parents (and ancestors) are obsoletion candidates
- option 1: remove all parent/ancestor that are obsoletion candidates from the report
- option 2: rename all obsoletion candidates to include "TO BE OBSOLETED" at the beginning of their label. (this should be done for the whole ontology, so we know that a term might actually loose more than one parent)
Please find examples here, included column name suggestions.

sabrinatoro · 2023-09-30T16:11:18Z

Reviewing [this spreadsheet]:

the obsoletion candidate (colE ) and their children (colB) are not in the branch review (colA)
For example: obsoletion candidate MONDO:0013742 (colE) is not in the MONDO:0005151 (colA) branch.
BUT all the children of the obsoletion candidate (ColB) are reported (e.g. there are 2 children for obsoletion candidate MONDO:0013742)
disregarding the "in the branch" or "out of the branch" (since this is incorrect because of what I mentioned above), some direct parents are missing. For example:
MONDO:0008964, should have a total of 4 direct parents: MONDO:0000824 ('congenital diarrhea'); MONDO:0000249 ('secretory diarrhea'), MONDO:0015178 (OBSOLETION CANDIDATE); MONDO:0000824 ('congenital diarrhea')

Note that the first example I look at, the obsoletion candidate MONDO:0008347 was already obsoleted... I am assuming this is not going to pose any problem since we will run this after each obsoletion round. But I still want to mention it to make sure.

twhetzel · 2023-10-02T03:58:18Z

@sabrinatoro in the second bullet point above:

- disregarding the "in the branch" or "out of the branch" (since this is incorrect because of what I mentioned above), some direct parents are missing. For example:
MONDO:0008964, should have a total of 4 direct parents: MONDO:0000824 ('congenital diarrhea'); MONDO:0000249 ('secretory diarrhea'), MONDO:0015178 (OBSOLETION CANDIDATE); MONDO:0000824 ('congenital diarrhea')

the term MONDO:0000824 ('congenital diarrhea') is duplicated

src/ontology/mondo.Makefile

matentzn · 2023-10-04T15:27:04Z

src/scripts/branch_review.py

+
+            # Column J: Affected Status = status
+            status = ""
+            if len(other_parents_in_branch) > 0 and all(


Suggested change

if len(other_parents_in_branch) > 0 and all(

if len(other_parents_in_branch) > 0 and any(

This seems to be the last issue in the table!

matentzn

Looks very nice!

src/scripts/branch_review.py

matentzn · 2023-10-05T09:42:04Z

src/scripts/branch_review.py

+            if len(other_parents_in_branch) > 0 and any(
+                " - TO_BE_OBSOLETED" not in parent for parent in other_parents_in_branch
+            ):
+                status = STAYS_IN_THE_BRANCH


@sabrinatoro in my opinion, this status should be renamed to: "Retains parent in branch" for clarity

We probably also want a code that expresses if the class retains an ancestor in he branch, and check that right after this one, so maybe this is a better naming scheme?:

Retains parent in branch (currently STAYS_IN_THE_BRANCH)

Retains ancestor in branch (currently missing)

Leaves branch but retains parent in other branch (currently LEAVES_THE_BRANCH)

Orphaned (leave as is)

@hrshdhgd

cc @hrshdhgd

@hrshdhgd

@hrshdhgd

sabrinatoro · 2023-10-06T18:30:34Z

@hrshdhgd and I reviewed spreadsheets created from different files (included relaxed, inferred, relaxed/inferred). Here is what we found out:

"relaxed" does not include parents that are inferred.
"inferred" (alone or in combination with "relax") always removes direct parents that are also ancestors.

Solution (that we are testing) is to create 3 spreadsheets:

spreadsheet 1: created using the "relaxed" file. --> give ALL direct parent that are asserted
spreadsheet 2: created using the "inferred" file --> give all inferred parent
spreadsheet 3 = FINAL/combined spreadsheet : combination of spreadsheet 1 and 2 (with removing all redundancy)

note: the "affected status" (orphan, leaves branch, stays in branch) should be calculated based on the information in the "combined" spreadsheet.

twhetzel · 2023-10-31T18:27:06Z

@hrshdhgd @matentzn - this is the PR I intended to ask when it will be merged. I think there may also be another separate PR for this Branch Review table as well.

matentzn added 7 commits September 15, 2023 10:37

Add branch-review table

ff582a7

Create quick-check.sparqlbook

7a79d14

Create curate-branch-review.sparql

c357625

update branch review ytables

73784e6

Update curate-branch-review.sparql

19b5e1f

Merge branch 'branch-review-table' of https://github.com/monarch-init…

99c538d

…iative/mondo into branch-review-table

update file

bcade23

hrshdhgd self-assigned this Sep 15, 2023

hrshdhgd and others added 13 commits September 22, 2023 13:42

First stab at generating table using oaklib

6c33aab

code update

fa035c7

update makefile

8ed1b8d

Merge branch 'master' into branch-review-table

5fe0d1b

update code for clearer outputs

03ea0a3

formatted

6d7f813

Merge branch 'branch-review-table' of https://github.com/monarch-init…

d0122ce

…iative/mondo into branch-review-table

updated code

d23452e

updated code

b371add

other columns should not have obsoletion_candidates

6d9903a

other columns should not have obsoletion_candidates

17ee229

minor cleanup

cca2c62

formatted and corrected code

8b3f8a6

matentzn commented Sep 26, 2023

View reviewed changes

upadted code based on NIco's suggestions

de41959

Added some comments

9b4ecba

updated code

8f67763

modified code as per requirements and understanding

c5e64cb

matentzn commented Oct 4, 2023

View reviewed changes

hrshdhgd added 3 commits October 4, 2023 10:40

deleted as per Nico's request

7f1f6b2

tweaked logic

795acb5

formatted

67781dc

sabrinatoro mentioned this pull request Oct 4, 2023

Mass obsoletion strategy #6731

Closed

removing items from parents code commented

f71ed4d

matentzn commented Oct 5, 2023

View reviewed changes

matentzn and others added 4 commits October 5, 2023 19:05

Use an unreduced version of mondo

cd00173

cc @hrshdhgd

comment updated

97a4c5e

Merge branch 'master' into branch-review-table

2811cc4

make multiple obsoletion tables

b4b854a

@hrshdhgd

hrshdhgd and others added 14 commits October 6, 2023 18:30

relax+reason separately

d4834e0

Added extra rule

ef76d6d

Added comment

a429630

bug fixes affected status

5995dce

obsoletion needed to relax and reason

9e338c0

formatted

51e33d7

fixed the CURIE appearing in both in & out of branch

755b4eb

remove 1 id as per Sabrina's request

d3f9062

make file has for loop for split results

2d543af

indentation fixed

e421f49

#6762 rules addressed

684cac5

Fixes #6762

cc5586e

Added sparql query

ca3c7a9

Merge branch 'master' into branch-review-table

addfa02

hrshdhgd mentioned this pull request Oct 30, 2023

Branch review code for node obsoletion monarch-initiative/mondolib#7

Merged

hrshdhgd added 2 commits November 15, 2023 18:22

Merge branch 'master' into branch-review-table

31bdf13

added _2 to report filenames for a separate run

dfed161

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Branch review table #6648

Create Branch review table #6648

matentzn commented Sep 15, 2023 •

edited by hrshdhgd

Loading

matentzn left a comment

matentzn Sep 26, 2023

matentzn Sep 26, 2023

matentzn Sep 26, 2023

hrshdhgd Sep 26, 2023

matentzn Sep 26, 2023

matentzn Sep 26, 2023

matentzn Sep 26, 2023

hrshdhgd commented Sep 26, 2023

sabrinatoro commented Sep 29, 2023

sabrinatoro commented Sep 30, 2023

twhetzel commented Oct 2, 2023

matentzn Oct 4, 2023

matentzn left a comment

matentzn Oct 5, 2023

matentzn Oct 5, 2023

sabrinatoro commented Oct 6, 2023

twhetzel commented Oct 31, 2023

	parent_of_obsolete_candidate = get_parent_from_relations(
	parents_of_affected = get_parent_from_relations(

	if len(other_parents_in_branch) > 0 and all(
	if len(other_parents_in_branch) > 0 and any(

Create Branch review table #6648

Are you sure you want to change the base?

Create Branch review table #6648

Conversation

matentzn commented Sep 15, 2023 • edited by hrshdhgd Loading

UPDATE:

matentzn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hrshdhgd commented Sep 26, 2023

sabrinatoro commented Sep 29, 2023

sabrinatoro commented Sep 30, 2023

twhetzel commented Oct 2, 2023

Choose a reason for hiding this comment

matentzn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabrinatoro commented Oct 6, 2023

twhetzel commented Oct 31, 2023

matentzn commented Sep 15, 2023 •

edited by hrshdhgd

Loading