"RuntimeError: This event loop is already running" When attempting REPL example #178

JacobCallahan · 2020-08-25T19:59:10Z

Each time I attempt the REPl example, I get the traceback below. Sync context-manager example worked though.
playwright==0.8.0
REPL Example: https://github.com/microsoft/playwright-python#repl-support-without-context-managers

IPython session

 jake  (e) venvplaywright  ~  ipython
/home/jake/.local/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
Python 3.8.3 (default, May 29 2020, 00:00:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from playwright import sync_playwright                                                                                                                                                

In [2]: playwright = sync_playwright().start()                                                                                                                                                

Traceback (most recent call last):
  File "/usr/local/bin/ipython", line 11, in <module>
    sys.exit(start_ipython())
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/__init__.py", line 126, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/jake/.local/lib/python3.8/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/ipapp.py", line 356, in start
    self.shell.mainloop()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 558, in mainloop
    self.interact()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 541, in interact
    code = self.prompt_for_code()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 467, in prompt_for_code
    text = self.pt_app.prompt(
  File "/home/jake/.local/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 994, in prompt
    return self.app.run(set_exception_handler=set_exception_handler)
  File "/home/jake/.local/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 811, in run
    return loop.run_until_complete(
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 592, in run_until_complete
    self._check_running()
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 552, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

If you suspect this is an IPython 7.13.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

sys:1: RuntimeWarning: coroutine 'Application.run_async' was never awaited

The text was updated successfully, but these errors were encountered:

mxschmitt · 2020-08-25T20:20:56Z

for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:

import nest_asyncio
nest_asyncio.apply()

Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?

JacobCallahan · 2020-08-25T21:13:58Z

@mxschmitt thanks for the quick response!
nest_asyncio isn't a standard package or one installed as a requirement for playwright. As such, I'm guessing most users wouldn't have it installed. However, that did solve the problem:

In [1]: import nest_asyncio                                                                                                                                   

In [2]: nest_asyncio.apply()                                                                                                                                  

In [3]: from playwright import sync_playwright                                                                                                                

In [4]: playwright = sync_playwright().start()                                                                                                                

In [5]:

My use-case within IPython is largely API exploration and interactive process development before writing a script or integration with an application.

Additional note, your example works perfectly in the base python REPL.

bubthegreat · 2020-10-02T03:17:27Z

@mxschmitt - FYI, the nested async doesn't work for jupyter notebooks for at least the sync REPL example:

import nest_asyncio
nest_asyncio.apply()

from playwright import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-3-85d403adca24> in <module>
----> 1 playwright = sync_playwright().start()

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
     32 
     33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34     return SyncPlaywrightContextManager()
     35 
     36 

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in __init__(self)
     81 class SyncPlaywrightContextManager:
     82     def __init__(self) -> None:
---> 83         self._connection = run_driver()
     84         self._playwright: SyncPlaywright
     85 

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver()
     76     # if loop.is_running():
     77     #     raise Error("Can only run one Playwright at a time.")
---> 78     return loop.run_until_complete(run_driver_async())
     79 
     80 

c:\users\bubth\development\playwright\venv\lib\site-packages\nest_asyncio.py in run_until_complete(self, future)
     93                 raise RuntimeError(
     94                     'Event loop stopped before Future completed.')
---> 95             return f.result()
     96         finally:
     97             events._set_running_loop(old_running_loop)

C:\Program Files\Python37\lib\asyncio\futures.py in result(self)
    176         self.__log_traceback = False
    177         if self._exception is not None:
--> 178             raise self._exception
    179         return self._result
    180 

C:\Program Files\Python37\lib\asyncio\tasks.py in __step(***failed resolving arguments***)
    221                 # We use the `send` method directly, because coroutines
    222                 # don't have `__iter__` and `__next__` methods.
--> 223                 result = coro.send(None)
    224             else:
    225                 result = coro.throw(exc)

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver_async()
     62         stdout=asyncio.subprocess.PIPE,
     63         stderr=_get_stderr_fileno(),
---> 64         limit=32768,
     65     )
     66     assert proc.stdout

C:\Program Files\Python37\lib\asyncio\subprocess.py in create_subprocess_exec(program, stdin, stdout, stderr, loop, limit, *args, **kwds)
    215         program, *args,
    216         stdin=stdin, stdout=stdout,
--> 217         stderr=stderr, **kwds)
    218     return Process(transport, protocol, loop)

C:\Program Files\Python37\lib\asyncio\base_events.py in subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, *args, **kwargs)
   1531         transport = await self._make_subprocess_transport(
   1532             protocol, popen_args, False, stdin, stdout, stderr,
-> 1533             bufsize, **kwargs)
   1534         if self._debug and debug_log is not None:
   1535             logger.info('%s: %r', debug_log, transport)

C:\Program Files\Python37\lib\asyncio\base_events.py in _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
    461                                          extra=None, **kwargs):
    462         """Create subprocess transport."""
--> 463         raise NotImplementedError
    464 
    465     def _write_to_self(self):

NotImplementedError:

pavelfeldman · 2020-10-06T03:14:09Z

@bubthegreat: This looks like a SelectorEventLoop's limitation on Python 3.7 on Windows. Starting with 3.8, a more complete implementation of the loop is used on Windows and it should support running subprocesses. See the platform support section for more details.

We do install this more capable ProactorEventLoop manually on 3.7 see here, but it might be that nest_asyncio does not respect our selection. So it is a combination of Windows + Python 3.7 + nest_asyncio that triggers it.

bubthegreat · 2020-10-06T04:02:51Z

I assume upgrading to 3.9 would improve this, still saw issues, and downgraded to 3.8.3 - still having issues with sync_playwright:

import nest_asyncio
nest_asyncio.apply()
from playwright import sync_playwright                                                                                                                
playwright = sync_playwright().start()    
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-2-dfa883f2869c> in <module>
      1 from playwright import sync_playwright
----> 2 playwright = sync_playwright().start()

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
     32 
     33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34     return SyncPlaywrightContextManager()
     35 
     36 

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in __init__(self)
     86 class SyncPlaywrightContextManager:
     87     def __init__(self) -> None:
---> 88         self._connection = run_driver()
     89         self._playwright: SyncPlaywright
     90 

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in run_driver()
     80     loop = asyncio.get_event_loop()
     81     if loop.is_running():
---> 82         raise Error("Can only run one Playwright at a time.")
     83     return loop.run_until_complete(run_driver_async())
     84 

Error: Can only run one Playwright at a time.

ddelange · 2020-10-06T09:20:53Z

Currently, playwright disallows attaching to a running loop:

playwright-python/playwright/main.py

Lines 79 to 83 in d98d38b

    
           def run_driver() -> Connection: 
        
               loop = asyncio.get_event_loop() 
        
               if loop.is_running(): 
        
                   raise Error("Can only run one Playwright at a time.") 
        
               return loop.run_until_complete(run_driver_async())

This should be fine in plain IPython (does not launch an asyncio event loop in the main thread), but will fail in Jupyter that uses tornado>=5 (launches asyncio event loop on startup under the hood).

As calling run_until_complete is disallowed on running loops, this will fail in Jupyter.

Minimal reproducible example that will run in latest IPython, but will break on Jupyter environments that use tornado>=5 under the hood:

import asyncio
print(asyncio.run(asyncio.sleep(0)))

So for Jupyter, additionally a pip install 'tornado<5' 'notebook<5.7.9' should fix loop issues introduced in tornado>=5 (and if you need jupyterhub, use jupyterhub<0.9.0). refs jupyter/notebook#3397 (comment) ipython/ipykernel#548

pavelfeldman · 2020-10-06T14:35:32Z

My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter?

bubthegreat · 2020-10-06T14:47:43Z

If I get time tonight ill play around with it- I dont see any reason for it not to run in the existing loop - or are there architectural challenges with it sharing the loop?

…

On Tue, Oct 6, 2020, 8:35 AM Pavel Feldman ***@***.***> wrote: My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5CHIMV52GR76C7R7XFYKTSJMTMLANCNFSM4QLBAFHQ> .

ddelange · 2020-10-08T07:51:17Z

Executing a cell in Jupyter will actually require a lock on the main loop from ipykernel, so you could schedule execution of a Future in one cell (which will start execution once that cell has finished), and get the result in the next cell. Doing it all in one cell is certainly not trivial without nested loops.

tals · 2020-12-24T15:44:43Z

for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:
import nest_asyncio
nest_asyncio.apply()
Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?

Was just bitten by this.

It is very convenient to use Jupyter for explorative programming, especially when the thing you're working with is heavy (like a browser in a particular state).
For my use case, I wanted to spin up a few browser instances and use them to test out a complicated feature that involves webrtc etc.

Also ran into this cute tutorial demonstrating why this is nice (he's using Selenium): https://medium.com/shakuro/adopting-ipython-jupyter-for-selenium-testing-d02309dd00b8

The stated workaround sadly didn't work for me, neither on sync nor async versions.

Hope this helps!

tals · 2020-12-24T16:26:16Z

Found a workaround for getting Playwright working within Jupyter.

It leverages an awesome python library called RPyc to run Playwright outside of the kernel while keeping the API the same.

!pip install rpyc

import rpyc
import subprocess
subprocess.Popen('rpyc_classic.py -m forking'.split())  # start RPyc worker

conn = rpyc.classic.connect('localhost')
rplaywright = conn.modules['playwright']  

# rplaywright has the same API as playwright
pr = rplaywright.sync_playwright().start()
browser = pr.chromium.launch()
page = browser.newPage()
page.goto('http://whatsmyuseragent.org/')
page.screenshot(path='example.png')
browser.close()

# Grab screen
from PIL import Image
from io import BytesIO

browser = pr.chromium.launch()

page = browser.newPage()
page.goto('http://localhost:4000/v/tf3nhh')
img=page.screenshot(path='example.png')
Image.open(BytesIO(img))

wohenniubi · 2021-01-02T04:13:19Z

Found a workaround for getting Playwright working within Jupyter.

It leverages an awesome python library called RPyc to run Playwright outside of the kernel while keeping the API the same.

!pip install rpyc

import rpyc
import subprocess
subprocess.Popen('rpyc_classic.py -m forking'.split())  # start RPyc worker

conn = rpyc.classic.connect('localhost')
rplaywright = conn.modules['playwright']  

# rplaywright has the same API as playwright
pr = rplaywright.sync_playwright().start()
browser = pr.chromium.launch()
page = browser.newPage()
page.goto('http://whatsmyuseragent.org/')
page.screenshot(path='example.png')
browser.close()

# Grab screen
from PIL import Image
from io import BytesIO

browser = pr.chromium.launch()

page = browser.newPage()
page.goto('http://localhost:4000/v/tf3nhh')
img=page.screenshot(path='example.png')
Image.open(BytesIO(img))

Hi tals, I follow the wolkaround that you mentioned, however, seems it can't help in win10. The error info shows "OSError: [WinError 193] %1 is not a valid Win32 application"

The complete error is as follows. Thanks for any further info.

OSError Traceback (most recent call last)
in
9 import rpyc
10 import subprocess
---> 11 subprocess.Popen('rpyc_classic.py -m forking'.split()) # start RPyc worker
12
13 conn = rpyc.classic.connect('localhost')

c:\program files\python38\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
852 encoding=encoding, errors=errors)
853
--> 854 self._execute_child(args, executable, preexec_fn, close_fds,
855 pass_fds, cwd, env,
856 startupinfo, creationflags, shell,

c:\program files\python38\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
1305 # Start the process
1306 try:
-> 1307 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
1308 # no special security
1309 None, None,

OSError: [WinError 193] %1 is not a valid Win32 application

tals · 2021-01-02T11:52:08Z

@wohenniubi sorry don't have a windows machine to test this out, but try figuring out how to run the rpyc_classic.py program on command line first

wohenniubi · 2021-01-04T00:57:54Z

@tals Thanks for the response. After several trials without success, I might switch back to use selenium by now.
Anyway, I'll keep on watching this jupyter issue and this promising tool, playwright.

@wohenniubi sorry don't have a windows machine to test this out, but try figuring out how to run the rpyc_classic.py program on command line first

mxschmitt · 2021-04-24T09:43:51Z

Closed for now as part of the triage. Please reopen if it's still persistent.

tals · 2021-04-25T19:20:04Z

@mxschmitt not sure why its closed - this problem still exists. You can reproduce this on the latest versions of jupyter+playwright

ddelange · 2021-04-25T19:44:21Z

The bug is upstream: ipython/ipykernel#548

And there is a workaround: #178 (comment)

Maybe could warn the jupyter user with a verbose re-raising try-except?

tals · 2021-04-25T20:58:44Z

@ddelange the proposed workaround sadly doesn't work on jupyter: #178 (comment)

The only thing I could do to get it working was running it outside of the process with RPyc

bubthegreat · 2022-03-08T21:31:22Z

It's been almost two years and this still isn't fixed that I can find - playwright is explicitly stopping me from using nested loops:

import nest_asyncio
nest_asyncio.apply()

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
Input In [3], in <cell line: 2>()
      1 from playwright.sync_api import sync_playwright
----> 2 playwright = sync_playwright().start()

File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:76, in PlaywrightContextManager.start(self)
     75 def start(self) -> SyncPlaywright:
---> 76     return self.__enter__()

File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:42, in PlaywrightContextManager.__enter__(self)
     40             own_loop = loop
     41         if loop.is_running():
---> 42             raise Error(
     43                 """It looks like you are using Playwright Sync API inside the asyncio loop.
     44 Please use the Async API instead."""
     45             )
     47         def greenlet_main() -> None:
     48             loop.run_until_complete(self._connection.run_as_sync())

Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.

The workarounds don't work for those of us using windows, and they don't appear to work in WSL either - is there a reason we can't get more information on why this is still closed two years later with no traction and no functional workarounds for presumably a fairly normal use case of using a jupyter notebook for exploratory programming?

It seems hard to believe that we just assume everyone will use raw iPython for everything so I'm confused by the reluctance to actually fix this.

If I'm understanding the asyncio base_events.py error, it's because there's no implementation of the _make_subprocess_transport function - is that not something that would be on the playwright package to implement in it's async implementation?

hkfsc · 2022-06-29T09:21:19Z

@tals Rpyc will only works if we will not call any playwright sub-modules directly, tomerfiliba-org/rpyc#496.
In my case, it not works well, still finding a way to let playwright sync api work with jupyter now...

danielichis · 2022-09-21T02:24:46Z

still waiting i want to leave selenium but this stop me

ddelange · 2022-09-21T04:49:02Z

If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (await & async def syntax), instead of the sync wrappers. They don't support sync wrappers around async libs...

liudonghua123 · 2022-11-02T16:38:52Z

still does not work in jupyter (vscode interactive).

Sync API is not allowed!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
# Error: It looks like you are using Playwright Sync API inside the asyncio loop.
# Please use the Async API instead.
p = sync_playwright().start()

Async API is not worked!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser.close()
# NotImplementedError
asyncio.run(main())

Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。

ddelange · 2022-11-02T16:41:01Z

can you try the async snippet without nest_asyncio, and post the full error traceback here?

liudonghua123 · 2022-11-02T16:45:17Z

can you try the async snippet without nest_asyncio, and post the full error traceback here?

Here is my snapshot and error logs.

NotImplementedError                       Traceback (most recent call last)
Cell In [33], line 9
      7         await browser.close()
      8 # NotImplementedError
----> 9 asyncio.run(main())

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\nest_asyncio.py:35, in _patch_asyncio.<locals>.run(main, debug)
     [33](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=32) task = asyncio.ensure_future(main)
     [34](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=33) try:
---> [35](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=34)     return loop.run_until_complete(task)
     [36](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=35) finally:
     [37](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=36)     if not task.done():

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\nest_asyncio.py:89, in _patch_loop.<locals>.run_until_complete(self, future)
     [86](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=85) if not f.done():
     [87](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=86)     raise RuntimeError(
     [88](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=87)         'Event loop stopped before Future completed.')
---> [89](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=88) return f.result()

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\futures.py:201, in Future.result(self)
    [199](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=198) self.__log_traceback = False
    [200](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=199) if self._exception is not None:
--> [201](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=200)     raise self._exception.with_traceback(self._exception_tb)
    [202](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=201) return self._result
...
    [496](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=495)                                      extra=None, **kwargs):
    [497](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=496)     """Create subprocess transport."""
--> [498](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=497)     raise NotImplementedError

NotImplementedError:

ddelange · 2022-11-02T16:53:09Z

this error is coming from nest_asyncio ^, can you run it in a fresh kernel without nest_asyncio?

liudonghua123 · 2022-11-02T16:56:13Z

It seems sync API is not work in plain python script.

The following code just launch browser, open page then exit. I did not call browser.close().

from playwright.sync_api import sync_playwright

p = sync_playwright().start()
browser = p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
page = browser.new_page()
page.goto('https://cnki.net/')

Here is ipython error logs.

C:\Users\Liu.D.H>ipython
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from playwright.sync_api import sync_playwright
   ...:
   ...: p = sync_playwright().start()
   ...: browser = p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
   ...: page = browser.new_page()
   ...: page.goto('https://cnki.net/')
Out[1]: <Response url='https://cnki.net/' request=<Request url='https://cnki.net/' method='GET'>>

Traceback (most recent call last):
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\Scripts\ipython.exe\__main__.py", line 7, in <module>
    sys.exit(start_ipython())
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\__init__.py", line 124, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\traitlets\config\application.py", line 976, in launch_instance
    app.start()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\ipapp.py", line 318, in start
    self.shell.mainloop()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 685, in mainloop
    self.interact()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 670, in interact
    code = self.prompt_for_code()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 609, in prompt_for_code
    text = self.pt_app.prompt(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\prompt_toolkit\shortcuts\prompt.py", line 1034, in prompt
    return self.app.run(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\prompt_toolkit\application\application.py", line 937, in run
    return loop.run_until_complete(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 622, in run_until_complete
    self._check_running()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 582, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

If you suspect this is an IPython 8.5.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

sys:1: RuntimeWarning: coroutine 'Application.run_async' was never awaited

C:\Users\Liu.D.H>

liudonghua123 · 2022-11-02T16:59:29Z

this error is coming from nest_asyncio ^, can you run it in a fresh kernel without nest_asyncio?

I restarted the kernel, the errors changed.

I run the following code block.

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser.close()

asyncio.run(main())

RuntimeError                              Traceback (most recent call last)
d:\code\python\cnki_crawler_playwright\main.py in line 9
      [35](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=34)         await browser.close()
      [36](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=35) # NotImplementedError
----> [37](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=36) asyncio.run(main())

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py:33, in run(main, debug)
      [9](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=8) """Execute the coroutine and return the result.
     [10](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=9) 
     [11](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=10) This function runs the passed coroutine, taking care of
   (...)
     [30](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=29)     asyncio.run(main())
     [31](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=30) """
     [32](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=31) if events._get_running_loop() is not None:
---> [33](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=32)     raise RuntimeError(
     [34](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=33)         "asyncio.run() cannot be called from a running event loop")
     [36](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=35) if not coroutines.iscoroutine(main):
     [37](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=36)     raise ValueError("a coroutine was expected, got {!r}".format(main))

RuntimeError: asyncio.run() cannot be called from a running event loop

ddelange · 2022-11-02T17:02:50Z

instead of asyncio.run(main()), try await main()

liudonghua123 · 2022-11-02T17:05:16Z

instead of asyncio.run(main()), try await main()

I tried, but still not work.

NotImplementedError                       Traceback (most recent call last)
d:\code\python\cnki_crawler_playwright\main.py in line 10
      [34](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=33)         await page.goto("https://cnki.net/")
      [35](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=34)         await browser.close()
---> [38](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=37) await main()

d:\code\python\cnki_crawler_playwright\main.py in line 3, in main()
      [30](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=29) async def main():
----> [31](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=30)     async with async_playwright() as p:
      [32](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=31)         browser = await p.chromium.launch(channel="chrome", headless=False, slow_mo=100)
      [33](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=32)         page = await browser.new_page()

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\async_api\_context_manager.py:46, in PlaywrightContextManager.__aenter__(self)
     [44](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=43) if not playwright_future.done():
     [45](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=44)     playwright_future.cancel()
---> [46](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=45) playwright = AsyncPlaywright(next(iter(done)).result())
     [47](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=46) playwright.stop = self.__aexit__  # type: ignore
     [48](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=47) return playwright

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_transport.py:121, in PipeTransport.connect(self)
    [118](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=117)     if getattr(sys, "frozen", False):
    [119](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=118)         env.setdefault("PLAYWRIGHT_BROWSERS_PATH", "0")
--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120)     self._proc = await asyncio.create_subprocess_exec(
    [122](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=121)         str(self._driver_executable),
...
    [496](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=495)                                      extra=None, **kwargs):
    [497](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=496)     """Create subprocess transport."""
--> [498](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=497)     raise NotImplementedError

NotImplementedError:

liudonghua123 · 2022-11-02T17:10:49Z

I also tried asyncio.get_event_loop().run_until_complete(main()), however, I got RuntimeError: This event loop is already running.

logs

RuntimeError                              Traceback (most recent call last)
d:\code\python\cnki_crawler_playwright\main.py in line 10
      [34](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=33)         await page.goto("https://cnki.net/")
      [35](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=34)         await browser.close()
---> [38](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=37) asyncio.get_event_loop().run_until_complete(main())

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py:622, in BaseEventLoop.run_until_complete(self, future)
    [611](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=610) """Run until the Future is done.
    [612](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=611) 
    [613](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=612) If the argument is a coroutine, it is wrapped in a Task.
   (...)
    [619](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=618) Return the Future's result, or raise its exception.
    [620](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=619) """
    [621](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=620) self._check_closed()
--> [622](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=621) self._check_running()
    [624](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=623) new_task = not futures.isfuture(future)
    [625](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=624) future = tasks.ensure_future(future, loop=self)

File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py:582, in BaseEventLoop._check_running(self)
    [580](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=579) def _check_running(self):
    [581](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=580)     if self.is_running():
--> [582](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=581)         raise RuntimeError('This event loop is already running')
    [583](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=582)     if events._get_running_loop() is not None:
    [584](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=583)         raise RuntimeError(
    [585](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=584)             'Cannot run the event loop while another loop is running')

RuntimeError: This event loop is already running

ddelange · 2022-11-02T17:27:54Z

I have no problem running the example on a fresh jupyter server:

The fact that your traceback hangs here:

--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120) self._proc = await asyncio.create_subprocess_exec(

Makes me think you might not be running in a clean jupyter environment. Maybe vscode does some magic?

liudonghua123 · 2022-11-03T16:01:46Z

I have no problem running the example on a fresh jupyter server:

The fact that your traceback hangs here:

--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120) self._proc = await asyncio.create_subprocess_exec(

Makes me think you might not be running in a clean jupyter environment. Maybe vscode does some magic?

Hi, I tried to update jupyter via pip install -U jupyterlab notebook voila. Then tried the following code in jupyterlab. But still not works.

import asyncio
from playwright.async_api import async_playwright

async def run(playwright):
    chromium = playwright.chromium
    browser = await chromium.launch(channel="chrome",headless=False, slow_mo=100).launch()
    page = await browser.new_page()
    await page.goto("https://example.com")
    await browser.close()

async def main():
    async with async_playwright() as playwright:
        await run(playwright)

await main()

logs

C:\Users\Liu.D.H>pip install -U jupyterlab notebook voila
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: jupyterlab in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (3.4.7)
Collecting jupyterlab
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2b/d4/e0627f216bfb451e807cb8c2c8a0fc27e47cc76e483e539e8e213b95518e/jupyterlab-3.5.0-py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 9.2 MB/s eta 0:00:00
Requirement already satisfied: notebook in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (6.4.12)
Collecting notebook
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/db/40/2d321ba572dc9a94a090d92c9826291a1dcee1e05bc6c1d641ce419b701d/notebook-6.5.2-py3-none-any.whl (439 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 439.1/439.1 kB 9.1 MB/s eta 0:00:00
Requirement already satisfied: voila in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (0.3.6)
Collecting voila
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ce/63/55971b7c40ab1c4d87beec34775811b8aeea9a123b2d0abc41febc58526c/voila-0.4.0-py3-none-any.whl (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 11.0 MB/s eta 0:00:00
Requirement already satisfied: nbclassic in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (0.4.3)
Requirement already satisfied: packaging in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (21.3)
Requirement already satisfied: jupyterlab-server~=2.10 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (2.15.0)
Requirement already satisfied: jupyter-core in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (4.11.1)
Requirement already satisfied: jupyter-server<3,>=1.16.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (1.18.1)
Requirement already satisfied: tomli in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (2.0.1)
Requirement already satisfied: jinja2>=2.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (3.1.2)
Requirement already satisfied: ipython in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (8.5.0)
Requirement already satisfied: tornado>=6.1.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab) (6.2)
Requirement already satisfied: terminado>=0.8.3 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.15.0)
Requirement already satisfied: ipykernel in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (6.15.1)
Requirement already satisfied: argon2-cffi in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (21.3.0)
Requirement already satisfied: traitlets>=4.2.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (5.3.0)
Requirement already satisfied: nest-asyncio>=1.5 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (1.5.5)
Requirement already satisfied: nbconvert>=5 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (6.5.3)
Requirement already satisfied: ipython-genutils in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.2.0)
Requirement already satisfied: jupyter-client>=5.3.4 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (7.3.4)
Collecting nbclassic
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/85/2a240df7326b7311ebd926c12d7df5394aef2f9f76ffbb294079cc43960e/nbclassic-0.4.8-py3-none-any.whl (9.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.8/9.8 MB 11.2 MB/s eta 0:00:00
Requirement already satisfied: prometheus-client in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.14.1)
Requirement already satisfied: pyzmq>=17 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (23.2.1)
Requirement already satisfied: nbformat in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (5.4.0)
Requirement already satisfied: Send2Trash>=1.8.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from notebook) (1.8.0)
Requirement already satisfied: websockets>=9.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from voila) (10.3)
Requirement already satisfied: nbclient<0.8,>=0.4.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from voila) (0.5.13)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jinja2>=2.1->jupyterlab) (2.1.1)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-client>=5.3.4->notebook) (2.8.2)
Requirement already satisfied: entrypoints in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-client>=5.3.4->notebook) (0.4)
Requirement already satisfied: pywin32>=1.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-core->jupyterlab) (304)
Requirement already satisfied: anyio<4,>=3.1.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server<3,>=1.16.0->jupyterlab) (3.6.1)
Requirement already satisfied: websocket-client in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server<3,>=1.16.0->jupyterlab) (1.3.3)
Requirement already satisfied: pywinpty in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server<3,>=1.16.0->jupyterlab) (2.0.7)
Requirement already satisfied: babel in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab-server~=2.10->jupyterlab) (2.10.3)
Requirement already satisfied: json5 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab-server~=2.10->jupyterlab) (0.9.9)
Requirement already satisfied: requests in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab-server~=2.10->jupyterlab) (2.28.1)
Requirement already satisfied: jsonschema>=3.0.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jupyterlab-server~=2.10->jupyterlab) (4.10.2)
Requirement already satisfied: notebook-shim>=0.1.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbclassic->jupyterlab) (0.1.0)
Requirement already satisfied: pygments>=2.4.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (2.13.0)
Requirement already satisfied: beautifulsoup4 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (4.11.1)
Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (1.5.0)
Requirement already satisfied: mistune<2,>=0.8.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.8.4)
Requirement already satisfied: lxml in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (4.9.1)
Requirement already satisfied: defusedxml in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.7.1)
Requirement already satisfied: bleach in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (5.0.1)
Requirement already satisfied: jupyterlab-pygments in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.2.2)
Requirement already satisfied: tinycss2 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (1.1.1)
Requirement already satisfied: fastjsonschema in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from nbformat->notebook) (2.16.1)
Requirement already satisfied: argon2-cffi-bindings in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from argon2-cffi->notebook) (21.2.0)
Requirement already satisfied: matplotlib-inline>=0.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipykernel->notebook) (0.1.5)
Requirement already satisfied: psutil in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipykernel->notebook) (5.9.1)
Requirement already satisfied: debugpy>=1.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipykernel->notebook) (1.6.3)
Requirement already satisfied: backcall in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (0.2.0)
Requirement already satisfied: prompt-toolkit<3.1.0,>3.0.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (3.0.30)
Requirement already satisfied: decorator in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (5.1.1)
Requirement already satisfied: pickleshare in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (0.7.5)
Requirement already satisfied: jedi>=0.16 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (0.18.1)
Requirement already satisfied: colorama in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (0.4.5)
Requirement already satisfied: stack-data in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from ipython->jupyterlab) (0.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from packaging->jupyterlab) (3.0.9)
Requirement already satisfied: idna>=2.8 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from anyio<4,>=3.1.0->jupyter-server<3,>=1.16.0->jupyterlab) (3.3)
Requirement already satisfied: sniffio>=1.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from anyio<4,>=3.1.0->jupyter-server<3,>=1.16.0->jupyterlab) (1.2.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jedi>=0.16->ipython->jupyterlab) (0.8.3)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab) (22.1.0)
Requirement already satisfied: wcwidth in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from prompt-toolkit<3.1.0,>3.0.1->ipython->jupyterlab) (0.2.5)
Requirement already satisfied: six>=1.5 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from python-dateutil>=2.8.2->jupyter-client>=5.3.4->notebook) (1.16.0)
Requirement already satisfied: cffi>=1.0.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from argon2-cffi-bindings->argon2-cffi->notebook) (1.15.1)
Requirement already satisfied: pytz>=2015.7 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from babel->jupyterlab-server~=2.10->jupyterlab) (2022.2.1)
Requirement already satisfied: soupsieve>1.2 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from beautifulsoup4->nbconvert>=5->notebook) (2.3.2.post1)
Requirement already satisfied: webencodings in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from bleach->nbconvert>=5->notebook) (0.5.1)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from requests->jupyterlab-server~=2.10->jupyterlab) (2022.6.15)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from requests->jupyterlab-server~=2.10->jupyterlab) (1.26.11)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from requests->jupyterlab-server~=2.10->jupyterlab) (2.1.0)
Requirement already satisfied: executing in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from stack-data->ipython->jupyterlab) (0.10.0)
Requirement already satisfied: asttokens in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from stack-data->ipython->jupyterlab) (2.0.8)
Requirement already satisfied: pure-eval in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from stack-data->ipython->jupyterlab) (0.2.2)
Requirement already satisfied: pycparser in c:\users\liu.d.h\appdata\local\programs\python\python310\lib\site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook) (2.21)
Installing collected packages: voila, nbclassic, notebook, jupyterlab
  Attempting uninstall: voila
    Found existing installation: voila 0.3.6
    Uninstalling voila-0.3.6:
      Successfully uninstalled voila-0.3.6
  Attempting uninstall: nbclassic
    Found existing installation: nbclassic 0.4.3
    Uninstalling nbclassic-0.4.3:
      Successfully uninstalled nbclassic-0.4.3
  Attempting uninstall: notebook
    Found existing installation: notebook 6.4.12
    Uninstalling notebook-6.4.12:
      Successfully uninstalled notebook-6.4.12
  Attempting uninstall: jupyterlab
    Found existing installation: jupyterlab 3.4.7
    Uninstalling jupyterlab-3.4.7:
      Successfully uninstalled jupyterlab-3.4.7
Successfully installed jupyterlab-3.5.0 nbclassic-0.4.8 notebook-6.5.2 voila-0.4.0

C:\Users\Liu.D.H>jupyter --version
Selected Jupyter core packages...
IPython          : 8.5.0
ipykernel        : 6.15.1
ipywidgets       : not installed
jupyter_client   : 7.3.4
jupyter_core     : 4.11.1
jupyter_server   : 1.18.1
jupyterlab       : 3.5.0
nbclient         : 0.5.13
nbconvert        : 6.5.3
nbformat         : 5.4.0
notebook         : 6.5.2
qtconsole        : not installed
traitlets        : 5.3.0

C:\Users\Liu.D.H>

Task exception was never retrieved
future: <Task finished name='Task-5' coro=<Connection.run() done, defined at C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_connection.py:217> exception=NotImplementedError()>
Traceback (most recent call last):
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_connection.py", line 224, in run
    await self._transport.connect()
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
    raise exc
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_transport.py", line 121, in connect
    self._proc = await asyncio.create_subprocess_exec(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In [1], line 15
     12     async with async_playwright() as playwright:
     13         await run(playwright)
---> 15 await main()

Cell In [1], line 12, in main()
     11 async def main():
---> 12     async with async_playwright() as playwright:
     13         await run(playwright)

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\async_api\_context_manager.py:46, in PlaywrightContextManager.__aenter__(self)
     44 if not playwright_future.done():
     45     playwright_future.cancel()
---> 46 playwright = AsyncPlaywright(next(iter(done)).result())
     47 playwright.stop = self.__aexit__  # type: ignore
     48 return playwright

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_transport.py:121, in PipeTransport.connect(self)
    118     if getattr(sys, "frozen", False):
    119         env.setdefault("PLAYWRIGHT_BROWSERS_PATH", "0")
--> 121     self._proc = await asyncio.create_subprocess_exec(
    122         str(self._driver_executable),
    123         "run-driver",
    124         stdin=asyncio.subprocess.PIPE,
    125         stdout=asyncio.subprocess.PIPE,
    126         stderr=_get_stderr_fileno(),
    127         limit=32768,
    128         creationflags=creationflags,
    129         env=env,
    130     )
    131 except Exception as exc:
    132     self.on_error_future.set_exception(exc)

File ~\AppData\Local\Programs\Python\Python310\lib\asyncio\subprocess.py:218, in create_subprocess_exec(program, stdin, stdout, stderr, limit, *args, **kwds)
    215 loop = events.get_running_loop()
    216 protocol_factory = lambda: SubprocessStreamProtocol(limit=limit,
    217                                                     loop=loop)
--> 218 transport, protocol = await loop.subprocess_exec(
    219     protocol_factory,
    220     program, *args,
    221     stdin=stdin, stdout=stdout,
    222     stderr=stderr, **kwds)
    223 return Process(transport, protocol, loop)

File ~\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py:1667, in BaseEventLoop.subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, encoding, errors, text, *args, **kwargs)
   1665     debug_log = f'execute program {program!r}'
   1666     self._log_subprocess(debug_log, stdin, stdout, stderr)
-> 1667 transport = await self._make_subprocess_transport(
   1668     protocol, popen_args, False, stdin, stdout, stderr,
   1669     bufsize, **kwargs)
   1670 if self._debug and debug_log is not None:
   1671     logger.info('%s: %r', debug_log, transport)

File ~\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py:498, in BaseEventLoop._make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
    494 async def _make_subprocess_transport(self, protocol, args, shell,
    495                                      stdin, stdout, stderr, bufsize,
    496                                      extra=None, **kwargs):
    497     """Create subprocess transport."""
--> 498     raise NotImplementedError

NotImplementedError:

ddelange · 2022-11-03T16:14:45Z

do you get the same traceback when you run the snippet in ipython instead of jupyter? if yes, we eliminated the ipykernel loop issue, and my next guess would be the fact that you're on windows...

ddelange · 2022-11-03T18:18:17Z

potentially indeed your OS ref https://stackoverflow.com/q/70349876

@mxschmitt would this be caught by the Windows CI?

edit: more specifically https://stackoverflow.com/a/44639711/5511061

liudonghua123 · 2022-11-04T02:10:45Z

@ddelange Thanks. I found a workaround on windows now. NOT RECOMMEND!

From jupyter/notebook#5916, minrk/ipykernel@079f072, https://stackoverflow.com/questions/44633458/why-am-i-getting-notimplementederror-with-async-and-await-on-windows/74311290#74311290, https://github.com/ipython/ipykernel/blob/99a1becaa958b33d80fe337fdbc41305030fdb6d/ipykernel/kernelapp.py#L633-L637. I found a way to make it work hackly (not recommend) on windows.

Comment asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) in %LOCALAPPDATA%\Programs\Python\Python310\Lib\site-packages\ipykernel\kernelapp.py (change to your path).

Then the following code will work in jupyter notebook.

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome", headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser. Close()

await main()

liudonghua123 · 2022-11-04T03:02:23Z

still does not work in jupyter (vscode interactive).

Sync API is not allowed!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
# Error: It looks like you are using Playwright Sync API inside the asyncio loop.
# Please use the Async API instead.
p = sync_playwright().start()

Async API is not worked!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser.close()
# NotImplementedError
asyncio.run(main())

Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。

I have another question, why can't I use sync API in jupyter? I have to write a lot await in async API. The sync API is simple and straight, performance or jupyter does not allowed?

ddelange · 2022-11-04T07:20:07Z

If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (await & async def syntax), instead of the sync wrappers. They don't support sync wrappers around async libs...

jup it's not allowed

liudonghua123 · 2022-11-05T16:22:44Z

If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (await & async def syntax), instead of the sync wrappers. They don't support sync wrappers around async libs...

jup it's not allowed

Not allowed? However, I used a lot of packages like numpy, pandas, matplotlib, tensorflow, pytorch, and so on are sync style of API, most of them are async only.

Is there any ways to make sync API of playwright-python work on jupyter?

ddelange · 2022-11-05T19:48:05Z

I didn't double check, but I'm pretty sure those libraries don't use asyncio under the hood (they may still release the GIL however). For more information on the topic, see ipython/ipykernel#548 (comment).

mxschmitt mentioned this issue Oct 4, 2020

Incompatible: playwright and Jupyter #202

Closed

LeMoussel mentioned this issue Oct 5, 2020

Playwright with Scrapy? #213

Closed

wasdee mentioned this issue Oct 16, 2020

tolha connot run on jupyter notebook wasdee/tolha#1

Open

ddelange mentioned this issue Oct 21, 2020

ipykernel disallows jupyter user to create/attach to event loop ipython/ipykernel#548

Open

davidteather mentioned this issue Nov 12, 2020

[INSTALLATION] - Can only run one Playwright at a time davidteather/TikTok-Api#341

Closed

pavelfeldman added the P3-collecting-feedback label Dec 1, 2020

mborus mentioned this issue Jan 18, 2021

how to run playwright-python without interface #410

Closed

mxschmitt closed this as completed Apr 24, 2021

mxschmitt mentioned this issue May 27, 2021

[BUG] Async Playwright with Jupyter throws NotImplementedError #723

Closed

elkd mentioned this issue Mar 15, 2022

Playwright sync API isn't compatible with PySimpleGUI event loop elkd/deriv#1

Closed

davidteather mentioned this issue Mar 20, 2022

V5.1.0 - Memory Improvement, Jupyter Support (on Linux), Hierarchical Exceptions, davidteather/TikTok-Api#860

Merged

mxschmitt mentioned this issue Jan 5, 2023

---> 46 playwright = AsyncPlaywright(next(iter(done)).result()) [BUG] #1710

Closed

simonw mentioned this issue Jan 8, 2024

Upgrade for compatibility with new Datasette 1.0 JSON format simonw/datasette-search-all#19

Closed

"RuntimeError: This event loop is already running" When attempting REPL example #178

"RuntimeError: This event loop is already running" When attempting REPL example #178

Comments

JacobCallahan commented Aug 25, 2020

mxschmitt commented Aug 25, 2020

JacobCallahan commented Aug 25, 2020

bubthegreat commented Oct 2, 2020 • edited Loading

pavelfeldman commented Oct 6, 2020 • edited Loading

bubthegreat commented Oct 6, 2020

ddelange commented Oct 6, 2020 • edited Loading

pavelfeldman commented Oct 6, 2020

bubthegreat commented Oct 6, 2020 via email

ddelange commented Oct 8, 2020

tals commented Dec 24, 2020 • edited Loading

tals commented Dec 24, 2020 • edited Loading

wohenniubi commented Jan 2, 2021

tals commented Jan 2, 2021

wohenniubi commented Jan 4, 2021

mxschmitt commented Apr 24, 2021

tals commented Apr 25, 2021

ddelange commented Apr 25, 2021

tals commented Apr 25, 2021

bubthegreat commented Mar 8, 2022 • edited Loading

hkfsc commented Jun 29, 2022

danielichis commented Sep 21, 2022

ddelange commented Sep 21, 2022 • edited Loading

liudonghua123 commented Nov 2, 2022 • edited Loading

ddelange commented Nov 2, 2022

liudonghua123 commented Nov 2, 2022 • edited Loading

ddelange commented Nov 2, 2022

liudonghua123 commented Nov 2, 2022

liudonghua123 commented Nov 2, 2022

ddelange commented Nov 2, 2022

liudonghua123 commented Nov 2, 2022

liudonghua123 commented Nov 2, 2022

ddelange commented Nov 2, 2022

liudonghua123 commented Nov 3, 2022

ddelange commented Nov 3, 2022

ddelange commented Nov 3, 2022 • edited Loading

liudonghua123 commented Nov 4, 2022

liudonghua123 commented Nov 4, 2022

ddelange commented Nov 4, 2022

liudonghua123 commented Nov 5, 2022

ddelange commented Nov 5, 2022

bubthegreat commented Oct 2, 2020 •

edited

Loading

pavelfeldman commented Oct 6, 2020 •

edited

Loading

ddelange commented Oct 6, 2020 •

edited

Loading

tals commented Dec 24, 2020 •

edited

Loading

tals commented Dec 24, 2020 •

edited

Loading

bubthegreat commented Mar 8, 2022 •

edited

Loading

ddelange commented Sep 21, 2022 •

edited

Loading

liudonghua123 commented Nov 2, 2022 •

edited

Loading

liudonghua123 commented Nov 2, 2022 •

edited

Loading

ddelange commented Nov 3, 2022 •

edited

Loading