Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disconnecting issue again and again #5037

Open
Abhishek5077 opened this issue Jan 2, 2025 · 5 comments
Open

Disconnecting issue again and again #5037

Abhishek5077 opened this issue Jan 2, 2025 · 5 comments

Comments

@Abhishek5077
Copy link

Describe the current behavior
A clear and concise explanation of what is currently happening.

Describe the expected behavior
A clear and concise explanation of what you expected to happen.

What web browser you are using
(Chrome, Firefox, Safari, etc.)

Additional context
Link to a minimal, public, self-contained notebook that reproduces this issue.

  • Share the file using your GitHub account using File > Save a copy as a GitHub Gist.
  • or Share Drive notebooks using the Share button then 'Get Shareable Link'.
@Abhishek5077
Copy link
Author

The tool is disconnecting again and again... and even when I type negative prompts, negativity is showing in the picture

@docdnp
Copy link

docdnp commented Jan 3, 2025

Here some screenshot of the issue described.
image

Here the log when the runtime is disconnected:
colab.research.google.com-1735933611601.log

@EvanWiederspan
Copy link

Can you share the notebook you were using? Does this happen for all notebooks, or just the one?

@docdnp
Copy link

docdnp commented Jan 17, 2025

Can you share the notebook you were using? Does this happen for all notebooks, or just the one?

I can't share the notebook itself. But I can share the code of the cell that is running when this behavior occurs:

training_history = None

if os.path.exists(result_training_log):
  model = load_model_results(result_model_file, result_training_log)
else:
  original_stdout = sys.stdout
  with open(result_training_log, "w") as f:
      sys.stdout = Tee(original_stdout, f)
      training_history = model.fit(train_dataset,
                                  validation_data=test_dataset,
                                  epochs=epochs,
                                  steps_per_epoch=steps_per_epoch,
                                  callbacks=all_callbacks,
                                  verbose=1)
  sys.stdout = original_stdout

Actually it's a keras model which runs for several hours. I've encountered the problem each time this kind of cells are executed.

@cperry-goog
Copy link

I see a bunch of socketio errors in the log, assuming this is related to socketio we're working on a larger fix that will resolve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants