Strange DPY-5002 error with executemany batcherrors #261
-
|
Hello all, I'm currently having a weird problem in Airflow using OracleHook, i know this forum pertains to oracledb but i think this is where i need help. So i've been writing a large migrating script to unify multiple similar databases into a singular one, and in one of the migrations i've been hit with:
And this error only occurs in one of the 40+ databases i'm currently migrating, it's such a weird error and i can't find anything about this specific error, theres another topic where someone is currently having a similar mistake but with a DPY-5003 error, and i don't know if they are connected. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 16 replies
-
|
Both of those errors indicate that the server is sending something that the client doesn't know how to process properly. If you can send packet traces that would be most helpful (set the environment variable |
Beta Was this translation helpful? Give feedback.
-
|
How many rows are you inserting? And how many batch errors are you expecting to be returned? The location in the code suggests an issue with this -- I will see if I can replicate based on what I am seeing. |
Beta Was this translation helpful? Give feedback.
-
|
About 250k per round, the data is sliced and then pass trough that code. Error count varies, worst case about 10k, best scenario no error whatsoever. |
Beta Was this translation helpful? Give feedback.
-
|
And now that you've said something about quantities, i've fixed this before, by slicing datasets i believe. |
Beta Was this translation helpful? Give feedback.
-
|
Ok. I can replicate. The protocol only allows for 65535 errors. If you exceed this value you get this error in thin mode. In thick mode you get invalid results (the number of batch errors is simply truncated by looking at the lowest 16 bits of the value). As such, this is a bug in the database and will need to be fixed there. In the meantime, you can avoid the issue by always ensuring that you never send more than 65535 rows to the database at a time when using batch errors. |
Beta Was this translation helpful? Give feedback.
-
|
You have no idea how happy knowing that it's easily avoidableand to know what causes this, i'll lower my dataset size on processing and continue testing, thanks a lot. |
Beta Was this translation helpful? Give feedback.
-
|
Or lower your bad data rate :) |
Beta Was this translation helpful? Give feedback.
-
|
Believe me, i would if i could, but my hands are tied. |
Beta Was this translation helpful? Give feedback.
-
|
python-oracledb 2.0.0 had improvements in messages when a large number of errors occur. |
Beta Was this translation helpful? Give feedback.
Ok. I can replicate. The protocol only allows for 65535 errors. If you exceed this value you get this error in thin mode. In thick mode you get invalid results (the number of batch errors is simply truncated by looking at the lowest 16 bits of the value). As such, this is a bug in the database and will need to be fixed there. In the meantime, you can avoid the issue by always ensuring that you never send more than 65535 rows to the database at a time when using batch errors.