NOT NULL constraint failed: bayesdb_crosscat_diagnostics.logscore

[fsaad]

bayeslite> ANALYZE solvers FOR 15 MINUTES wait;
Traceback (most recent call last):
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/shell/core.py", line 133, in default
    cursor = bql.execute_phrase(self._bdb, phrase)
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/bql.py", line 459, in execute_phrase
    ckpt_seconds=phrase.ckpt_seconds)
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/metamodels/crosscat.py", line 1073, in analyze_models
    'iterations': theta['iterations'],
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/bayesdb.py", line 236, in sql_execute
    self.sql_tracer, self._do_sql_execute, string, bindings)
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/bayesdb.py", line 178, in _maybe_trace
    return meth(string, bindings)
  File "/home/fsaad/pcp/bayeslite/build/lib.linux-x86_64-2.7/bayeslite/bayesdb.py", line 239, in _do_sql_execute
    return bql.BayesDBCursor(self, self.sqlite3.execute(string, bindings))
IntegrityError: NOT NULL constraint failed: bayesdb_crosscat_diagnostics.logscore

Suspicion this happened because I have some columns of all None - but we should be able to do something about that case.

[riastradh-probcomp]

This is a bug in Crosscat: somehow analyze is returning None, or more likely NaN, for some logscore.

[fsaad]

Oh boy just got it again, even after nullifying and ignoring columns of all None.

[riastradh-probcomp]

I don't expect any particular Nones in the table have anything to do with this. It sounds like an invalid floating-point operation in the logscore calculation which yields NaN.

One way to try to debug it would be to trap the invalid-operation exception in Crosscat, if code inspection doesn't find it.

Can you show your table's M_c? (select metadata_json from bayesdb_crosscat_metadata where generator_id = ?)

It might also be helpful to apply this patch to bayeslite:
assertnotnan.txt

[gregory-marton]

@fsaad, can you take next steps, or provide more info to independently reproduce?

[gregory-marton]

One output from the suggested assertion:
https://docs.google.com/document/d/1fpnbR4AxuWE0NetllswHRUFMD0hk3PqNCdvCnY8u2y8/edit
(In this case I do have columns of all nulls, intentionally).

[gregory-marton]

probcomp/crosscat@f91197c
19b8071
as additional diagnostics, and improved guessing in 8b87fd1
to help avoid the problem in the first place. It covers all instances I've run across so far, namely all-nulls, and all-null-or-same-value. It's still super easy to get this error in a repeatable way by specifying the column type for such a degenerate column as numeric, rather than letting guess ignore it.

[riastradh-probcomp]

It sounds like there is a bug in Crosscat causing NaN to come out. If we mask the bug in bayeslite, that will make it harder to find the bug in Crosscat. Did you trip over the assert you added in Crosscat, or only the one in bayeslite?

[BelhalK]

Same bug happening on a remote host but not on my local computer
What do you advise?

[gregory-marton]

updating to the latest bayeslite / bdbcontrib, ahead of the releases
(because I haven't released 0.1.6 yet, and it's fixed ahead of 0.1.5)

On Wed, Feb 3, 2016 at 3:55 PM, Belhal KARIMI [email protected]
wrote:

Same bug happening on a remote host but not on my local computer
What do you advise?

[image: not null not null constraint failed]
https://cloud.githubusercontent.com/assets/15817944/12796777/9281f7ea-ca8e-11e5-8d85-527ce0752133.png

—
Reply to this email directly or view it on GitHub
#284 (comment).

Gregory Marton

[email protected]
617-858-0775
http://csail.mit.edu/~gremio/cal

[asilversempirical]

I just tripped this bug in bayeslite 0.1.8. It sounds like that's unexpected. Do you want me to dump any information?

[riastradh-probcomp]

Fixing this requires finding where Crosscat is getting NaN in the logscore calculation. Last time I tried to track down sources of this symptom I liberally sprinkled assert(!isnan(...)) throughout Crosscat, starting in State.cpp's State::transition_* functions.

Here is an example of a bug that could cause this symptom (a bug which bayeslite's logic to guess statistical types tries to avoid in some cases, so it may not be the bug you're encountering): probcomp/crosscat#85

[asilversempirical]

Got it. I was just confused by Grem's comment that "...it's fixed ahead of 0.1.5"

Is there a straightforward workaround in the meantime? I could manually ignore the column if it's easy to figure out what column it's choking on, but I don't get much in the ways of debug output. Is that included in the output of that patch?

[riastradh-probcomp]

Probably Grem was referring to the bayeslite statistical type guessing heuristic to avoid modelling constant columns.

Unfortunately, at the time the NaN comes flying out of Crosscat, it is not associated with any particular column. We don't know all the conditions of column values (if they are isolated to individual columns!) under which Crosscat will compute NaN for a logscore. We know at least one condition: a constant column. There are more conditions which have not been fully analyzed, such as #388, which has a lead you might follow about normal sufficient statistics and update_continuous_hypers.

[riastradh-probcomp]

Of course, if you can discern that you have a constant column and then skip modelling it, that would be the easiest way to work around this! But if you don't obviously have a constant column then I'm afraid it will take some more work.

[alxempirical]

@asilversempirical, can you get a backtrace if you recompile crosscat with FP exceptions turned on in State.cpp?

[asilversempirical]

Performance got so bad after enabling FP exceptions that it couldn't initialize the models. I never got to analyze to try to trigger the issue.

[alxempirical]

Thanks for trying it. I'll try it on some of the smaller test cases I ran into on the other thread when I get some time.

[fsaad]

d2376c1