Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a Pandera/PySpark base class for entities in a knowledge graph ontology #2

Merged
merged 48 commits into from
Aug 22, 2022
Merged

Create a Pandera/PySpark base class for entities in a knowledge graph ontology #2

merged 48 commits into from
Aug 22, 2022

Conversation

rjurney
Copy link
Contributor

@rjurney rjurney commented Aug 1, 2022

See #1 for more information.

@rjurney rjurney added enhancement New feature or request etl Extract, Transform, Load (ETL) labels Aug 1, 2022
@rjurney rjurney requested a review from ThePigLA August 1, 2022 01:16
@rjurney rjurney marked this pull request as draft August 1, 2022 01:16
ThePigLA
ThePigLA approved these changes Aug 1, 2022
View reviewed changes
Copy link

@ThePigLA ThePigLA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm...is this ready for review?

@rjurney
Copy link
Contributor Author

rjurney commented Aug 1, 2022 •
edited
Loading

@ThePigLA no, that's why I marked it as a draft. I only just started. In the future I will wait until it is done to set reviewers, that always is confusing.

Russell Jurney added 23 commits August 2, 2022 00:30
Merge branch 'main' of github.com:Graphlet-AI/graphlet into rjurney/e…
e5ab29a 
…tl-init
Initial stub for EntityBase
89f01fb
Formatting for VSCode settings
c55f5ed
Initial classes for nodes and edges
ee55a3f
Moved test_spark.py to test_etl.py
ecfbbee
Make version test pass
8054f3e
Ignore D202 - no blank space after docstring
3d495f1
+ipython dev dependency
c4b578b
Ignore FNE008 - function names can't end with their first argument name
f60df10
Don't ignore FNE008
c30a439
+U100 - stupid thing ignores classnames in class functions
d84730e
Working entity base with more unit tests
ca0baee
Run pytest as a pre-commit hook
e4222e8
Validate the src and dst fields of an edge to be UUIDs
887c275
comment fix
314ac42
Make data/ exist
463b709
Allow function names to end with their first variable name
0fd5f5d
Disable unit tests during pre-commit
df7a056
+xmltodict as dev dependency
9a4f1ab
Removed xmltodict as dev dependency, moved to normal. Added ujson.
4ca49ad
Added types-ujson and types-xmltodict to make mypy happy. Added tqdm …
de81fe0 
…for DBLP JSON writing progress bar.
+types-tqdm
985801e
+requests +types-requests
37e72bd
Russell Jurney and others added 21 commits August 9, 2022 11:48
Ignore data/ directory
7629780
minus types-tqdm plus tqdm-stubs
93ee5e0
Changed entity_type from a Enu to an typing.Optional[str] with a vali…
c769181 
…dator that sets the default value to the class name of the sub-class of EntityBase.
Merge branch 'main' of github.com:Graphlet-AI/graphlet into rjurney/e…
238f5d6 
…tl-init
Path utilities
8aa0643
Ignore error about unneccessary return types as it is dumb
4d7dc08
Spark ETL test code
8d59b98
+LICENSE
61a07cc
@rjurney
No idea what changed, permissions
536d0de
@rjurney
Perms
4139a3f
@rjurney
noop
93c630f
@rjurney
Unit test for pydantic-spark
5c55ea5
@rjurney
+pandas
a166e29
@rjurney
+pyarrow
a030bae
@rjurney
Instructions to update to poetry 1.2
0426f93
@rjurney
Set no-binary = pydantic
b23ed60
@rjurney
Commented out unit test for pydantic spark stuff
85afede
@rjurney
Test data for movies stuff for ETL tools
21006c3
@rjurney
Encode the path to dmypy in conda
7ed6c5d
@rjurney
D403 - don't have to capitalize first word on first line in comment
0594e0f
@rjurney
Utilities for downloading DBLP data and converting each type to its o…
3be2b04 
…wn JSON Lines file. Unit tests.
@rjurney rjurney changed the title Create a Pydantic/PySpark base class for entities in a knowledge graph ontology Create a Pandera/PySpark base class for entities in a knowledge graph ontology Aug 22, 2022
@rjurney
Copy link
Contributor Author

rjurney commented Aug 22, 2022

Am moving from Pydantic to Pandera... see comments and edits to issue #1

https://pandera.readthedocs.io/en/latest/dataframe_schemas.html

@rjurney
Copy link
Contributor Author

rjurney commented Aug 22, 2022

@ThePigLA @tanmoyio my PR has gotten too large. I am merging it! The README is MUCH bigger now :)

@rjurney rjurney added the documentation Improvements or additions to documentation label Aug 22, 2022
@rjurney rjurney requested a review from tanmoyio August 22, 2022 23:08
@rjurney rjurney marked this pull request as ready for review August 22, 2022 23:08
@rjurney rjurney merged commit 1324d62 into main Aug 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ThePigLA ThePigLA ThePigLA approved these changes

@tanmoyio tanmoyio Awaiting requested review from tanmoyio

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation enhancement New feature or request etl Extract, Transform, Load (ETL)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rjurney @ThePigLA