Skip to content

Commit

Permalink
changed variables for to satisfy pre-commit hooks
Browse files Browse the repository at this point in the history
  • Loading branch information
KeilanEvans committed Oct 8, 2024
1 parent 0706b9b commit 19ca257
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions rdsa_utils/helpers/pyspark.py
Original file line number Diff line number Diff line change
Expand Up @@ -576,13 +576,13 @@ def cut_lineage(df: SparkDF) -> SparkDF:
try:
logger.info("Converting SparkDF to Java RDD.")

jRDD = df._jdf.toJavaRDD()
jSchema = df._jdf.schema()
jRDD.cache()
jrdd = df._jdf.toJavaRDD()
jschema = df._jdf.schema()
jrdd.cache()
spark = df.sparkSession
newJavaDF = spark._jsparkSession.createDataFrame(jRDD, jSchema)
newDF = SparkDF(newJavaDF, spark)
return newDF
new_java_df = spark._jsparkSession.createDataFrame(jrdd, jschema)
new_df = SparkDF(new_java_df, spark)
return new_df
except Exception as e:
logger.error(f"An error occurred during the lineage cutting process: {e}")
raise
Expand Down

0 comments on commit 19ca257

Please sign in to comment.