Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Headful browsing to Inspect's Web Tooling #1

Closed
wants to merge 10 commits into from
Closed

Conversation

cmathw
Copy link
Collaborator

@cmathw cmathw commented Nov 28, 2024

This PR contains:

  • New features
  • Changes to dev-tools e.g. CI config / github tooling
  • Docs
  • Bug fixes
  • Code refactor

What is the current behavior? (You can also link to an open issue here)

The web_browser tooling only makes use of headless browser, it is not possible to view this browser graphically in real-time.

What is the new behavior?

This PR implements headful browsing, allowing users to view an agent's interactions with the browser in real time via a VNC viewer.

Tests have been added for this new behaviour in test_playwright_crawler. These are essentially just parameterising playwright's headless flag to test both True and False, instead of just True originally.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

This PR should be accompanied by an update to the image (aisiuk/inspect-web-browser-tool) hosted at Dockerhub for web_browser tooling. This PR's implementation defaults to headless mode. For users making use of web_browser tooling who are not aware of this PR, there should be no downstream impacts.

Other information:

This PR's implementation makes use of a layered Dockerfile approach, the user selects a build target in their task's docker-compose file. For example, in the examples/browser/compose.yaml file, the user can choose headful browsing with:

services:
  default:
    build:
      context: ../../src/inspect_ai/tool/_tools/_web_browser/_resources
      dockerfile: Dockerfile
      target: headful
    ports:
      - "127.0.0.1:5900:5900"

If a target is not specified, or it is specified as headless the Dockerfile will build the headless image. In this instance, it is not necessary to specify the 127.0.0.1:5900:5900 port mapping either (this is only needed for VNC viewing).

This PR achieves this broadly by:

  • Replacing the playwright headless flag with an environment variable instead of hardcoding to True (i.e. the "HEADLESS" environment variable).
  • Each Docker build set this "HEADLESS" environment variable depending on the target chosen by the user. The Dockerfile uses a layered approach such that only the layer's necessary for each respective build are included in the image.
  • The same entrypoint.sh file is copied across into the web_browser container irrespective of target, it uses the "HEADLESS" environment variable to dictate the actions taken at container start.

Note: When running tests, a chromium browser will open/close for the headful tests, if necessary I'm open to adding decorators to these tests so they are only run when specified.

@cmathw cmathw changed the title [DRAFT] Add Headful browsing to Inspect's Web Tooling Add Headful browsing to Inspect's Web Tooling Nov 29, 2024
@cmathw cmathw changed the base branch from headful-integration to main January 6, 2025 03:22
@cmathw cmathw changed the base branch from main to headful-integration January 6, 2025 03:22
@cmathw cmathw changed the base branch from headful-integration to main January 6, 2025 03:23
@cmathw cmathw changed the base branch from main to headful-integration January 6, 2025 03:28
@cmathw cmathw changed the base branch from headful-integration to main January 6, 2025 04:02
@cmathw
Copy link
Collaborator Author

cmathw commented Jan 6, 2025

Closing this PR as it's being moved to the upstream repository here.

@cmathw cmathw closed this Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant