Quick Start Example executes infinitely

[polinariabar]

Describe the bug
Trying to launch example from https://uber.github.io/orbit/tutorials/quick_start.html
The line dlt.fit(df=train_df) is executed infinitely (I've been waiting for hours and nothing happened)

To Reproduce
Steps to reproduce the behavior:
Code:

%matplotlib inline
import orbit
from orbit.utils.dataset import load_iclaims
from orbit.models.dlt import ETSFull
from orbit.diagnostics.plot import plot_predicted_data

df = load_iclaims()
date_col = 'week'
response_col = 'claims'
test_size = 52
train_df = df[:-test_size]
test_df = df[-test_size:]

dlt = ETSFull(
    response_col=response_col,
    date_col=date_col,
    seasonality=52,
    seed=8888,
)

dlt.fit(df=train_df)

Expected behavior
As in the example, I expected the code to compile in few minutes.

Environment (please complete the following information):

• OS: macOS Big Sur 11.4
• Python Version: 3.8.5
• Versions of Major Dependencies pandas==1.1.3, scikit-learn==0.23.1, cython==0.29.21 , orbit==1.0.15

[polinariabar]

Sometimes, when I run the cell with this line dlt.fit(df=train_df), I recieve INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_982090c5656030fa038b63e5c383dbff NOW. and the cell continues to be executed

[wangzhishi]

can you check if there are any .pkl files in the installed package folder orbit/stan_compiled/? if so, maybe try remove them and recompile/refit for one more time

[polinariabar]

The file was deleted, but it didn't help.

Also I noticed this output in jupiters console when I launch the cell:

ModuleNotFoundError: No module named 'stanfit4anon_model_982090c5656030fa038b63e5c383dbff_5534157429603675071'
Process SpawnPoolWorker-4:
Traceback (most recent call last):
  File "/Users/polinariabar/opt/anaconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/polinariabar/opt/anaconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/polinariabar/opt/anaconda3/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/polinariabar/opt/anaconda3/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'stanfit4anon_model_982090c5656030fa038b63e5c383dbff_5534157429603675071'

[polinariabar]

Also, if I delete .pkl file and run the cell, I recieve in jupyters console a lot of warnings, you can find them by link below
https://yadi.sk/d/y-sJeczS5ylgtw

[wangzhishi]

these logs are expected when you re-compile the pystan model. This will only be done once (when you run the model again, you won't see them anymore).

Question: now are you able to run the model?

[polinariabar]

No, the problem still remains

[polinariabar]

I launched program without using anaconda and here is the output:

no display found. Using non-interactive Agg backend
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
before
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_982090c5656030fa038b63e5c383dbff NOW.
Traceback (most recent call last):
  File "test.py", line 22, in <module>
    dlt.fit(df=train_df)
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/orbit/models/ets.py", line 417, in fit
    model_extract = estimator.fit(
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/orbit/estimators/stan_estimator.py", line 105, in fit
    compiled_stan_file = get_compiled_stan_model(model_name)
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/orbit/utils/stan.py", line 44, in get_compiled_stan_model
    compile_stan_model(stan_model_name)
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/orbit/utils/stan.py", line 31, in compile_stan_model
    sm = StanModel(model_code=model_code)
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/pystan/model.py", line 384, in __init__
    self.module = load_module(self.module_name, lib_dir)
  File "/Users/polinariabar/.local/lib/python3.8/site-packages/pystan/model.py", line 50, in load_module
    return __import__(module_name)
ImportError: dlopen(/var/folders/0p/2tx_6srn6klchhb6z3fy4s1h0000gn/T/pystan_utt34pin/stanfit4anon_model_982090c5656030fa038b63e5c383dbff_3154386277716755761.cpython-38-darwin.so, 2): Symbol not found: __ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev
  Referenced from: /var/folders/0p/2tx_6srn6klchhb6z3fy4s1h0000gn/T/pystan_utt34pin/stanfit4anon_model_982090c5656030fa038b63e5c383dbff_3154386277716755761.cpython-38-darwin.so
  Expected in: flat namespace
 in /var/folders/0p/2tx_6srn6klchhb6z3fy4s1h0000gn/T/pystan_utt34pin/stanfit4anon_model_982090c5656030fa038b63e5c383dbff_3154386277716755761.cpython-38-darwin.so

[psarka]

Same happens for me, fit spams with stan logs and then just hangs.
Ubuntu 20.04, python 3.8.10, orbit 1.10

[pnrsze]

same issue. DLT example was still running at ~ 10 mins, with INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_982090c5656030fa038b63e5c383dbff NOW.

[jfogelberg]

I have the same issue. If I convert the notebook to a .py file and run, it works without problems. After running the .py file once I can run the notebook after that. Not sure if the changes I make in the notebook after that will be applied correctly or not (since it seems like the issue is related to how the model is compiled when running the code in the notebook).

[edwinnglabs]

There is an additional compiling step when you first run such model. That's because in the backend stan is compiled in C++ code. After the first time, it stores a pickle object for future use. Hence, you should expect a much slower run time afterward. If you see the log is with something like INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_982090c5656030fa038b63e5c383dbff NOW., it means you are on the compiling step still.

Perhaps one way is to double check you can run pystan code smoothly. The installation of pystan2 can be found here

[jfogelberg]

For me the code runs fine if I run it in a .py file, I only get the issue with the model not compiling when I run it in the jupyter notebook (can leave it for several hours without completing).
I can run other packages that use pystan (e.g. facebook prophet) in jupyter notebooks without problems.

[wangzhishi]

The problem of model compiling hanging in jupyter notebook is due to the varied behaviors of multiprocessing across different operating systems and python versions. This problem is usually observed in Windows and python version >= 3.8. There are some similar issue raised for pystan:

https://discourse.mc-stan.org/t/pystan-throws-error-when-running-chains-in-parallel-n-jobs-1/17563 (@polinariabar your jupyter error log is similar to this one?)
stan-dev/pystan#80

Based on my research, there are two solutions to overcome this:

1. run this at the very beginning of your notebook

   import multiprocessing
   multiprocessing.set_start_method("fork", force=True)
   

   it is possible that the compiling still hangs after adding this, depending your OS/python version.
2. run a basic model fitting from a plain python script in the terminal. This serves to generate .pkl file for the underlying stan model. After this you can switch to notebook, and the compiling will not be done again since the model is already compiled.
   this solution should work in general, which is also suggested by @jfogelberg.

[wangzhishi]

As pointed by @brandoncbryant (#699 (comment)), execute the code in __main__ block might also dodge this issue.

[gpatil-beam]

Thanks @wangzhishi , multiprocessing setting worked.

[TeunvdWeij]

Nothing seems to work for me. If I put the same code in a .py file, it does work.

I have WSL2, Ubuntu 20.04, Python 3.8.10.

Multiprocessing doesn't do anything, neither does putting my code in the __main__ block. When I run the same model in a .py file, it does not create a .pkl file, only a .sqlite file. This also does not work.

To reproduce:

import multiprocessing
multiprocessing.set_start_method("fork", force=True)

import pystan

def main():
    model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
    model = pystan.StanModel(model_code=model_code)
    y = model.sampling().extract()['y']
    y.mean()  # with luck the result will be near 0

if __name__ == "__main__":
    main()