GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

lysnikolaou · 2024-11-21T11:25:26Z

Rationale for this change

There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.

What changes are included in this PR?

Add jobs to build free-threaded wheels on Windows
Move VCPKG-related stuff to a reusable base job

Are these changes tested?

No tests required.

Are there any user-facing changes?

No.

GitHub Issue: [Python][Packaging] Enable free-threaded wheels on Windows #44421

github-actions · 2024-11-21T11:25:53Z

⚠️ GitHub issue #44421 has been automatically assigned in GitHub to PR creator.

lysnikolaou · 2024-11-26T13:53:14Z

If someone could trigger the new jobs, that'd be really helpful!

raulcd · 2024-11-27T09:36:26Z

@github-actions crossbow submit -g wheel-windows-*

github-actions · 2024-11-27T09:37:40Z

Invalid group(s) {'wheel-windows-*'}. Must be one of {'wheel', 'cuda', 'example-cpp', 'linux', 'homebrew', 'fuzz', 'integration', 'cpp', 'example-python', 'nightly-release', 'verify-rc-source-linux', 'c-glib', 'example', 'python', 'nightly', 'conan', 'verify-rc-wheels', 'packaging', 'r', 'verify-rc-source', 'verify-rc', 'nightly-packaging', 'nightly-tests', 'vcpkg', 'java', 'linux-arm64', 'verify-rc-binaries', 'conda', 'test', 'verify-rc-source-macos', 'ruby', 'verify-rc-jars', 'linux-amd64'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/12047528593

raulcd · 2024-11-27T09:40:58Z

@github-actions crossbow submit wheel-windows-*

github-actions · 2024-11-27T09:43:17Z

Revision: 78d8f9e

Submitted crossbow builds: ursacomputing/crossbow @ actions-86ba33e477

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

raulcd

Thanks for the PR, I triggered the CI jobs

ci/docker/python-wheel-windows-test-vs2019.dockerfile

lysnikolaou · 2024-11-27T15:23:12Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

lysnikolaou · 2024-11-27T15:36:49Z

Also, the failure on 3.9 is that py cannot be found when running py -%PYTHON% setup.py bdist_wheel || exit /B 1, which seems odd.

raulcd · 2024-12-02T11:51:27Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

I haven't tested and I am unsure if this works but maybe we can try this (https://stackoverflow.com/a/21912169):
py -3.13t -c "import pyarrow" || echo ERROR && exit /B 1 instead of py -3.13t -c "import pyarrow" || exit /B 1

lysnikolaou · 2024-12-02T13:21:21Z

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

pitrou · 2024-12-04T13:57:39Z

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

Hmm, that one should be fixed in CPython AFAIU: python/cpython#111650

pitrou · 2024-12-04T13:59:43Z

Ah, scratch that, the rationale is actually here:
https://github.com/pypa/distutils/pull/310/files

pitrou · 2024-12-04T14:02:42Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

Well, stderr should already be shown by default, so it's weird that nothing gets displayed.

pitrou · 2024-12-18T17:31:56Z

@github-actions crossbow submit wheel-windows-*

pitrou · 2024-12-18T17:32:23Z

Given setuptools does not seem in a hurry to merge the distutils fix, I've tried to push a manual workaround in this PR.

github-actions · 2024-12-18T17:34:22Z

Revision: 83a76e3

Submitted crossbow builds: ursacomputing/crossbow @ actions-5bfc22ac30

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2024-12-18T19:59:10Z

Well, building the free-threaded wheel failed with:

LINK : fatal error LNK1104: cannot open file 'python313t.lib' [C:\arrow\python\build\temp.win-amd64-cpython-313t\arrow_python.vcxproj]

Edit: I've checked: that file exists in a Python 3.13 Windows install.

pitrou · 2024-12-18T20:26:02Z

@github-actions crossbow submit wheel-windows-cp313*

pitrou · 2025-01-06T08:26:32Z

@github-actions crossbow submit wheel-windows*

pitrou · 2025-01-06T08:27:29Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T08:28:45Z

Revision: f2e5f86

Submitted crossbow builds: ursacomputing/crossbow @ actions-8af848e3db

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

github-actions · 2025-01-06T08:29:43Z

Revision: 3b7e272

Submitted crossbow builds: ursacomputing/crossbow @ actions-ab1b9c5aa7

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-06T09:01:04Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T09:03:16Z

Revision: e51b395

Submitted crossbow builds: ursacomputing/crossbow @ actions-6c4f3fb019

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-06T09:27:08Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T09:29:22Z

Revision: 8416d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-72745204d9

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-06T12:06:51Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T12:09:12Z

Revision: 7ebb73e

Submitted crossbow builds: ursacomputing/crossbow @ actions-29a34e2f58

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-06T13:09:40Z

@kou @raulcd This is rebased now and ready for (re-)review. Do you want to take a look?

pitrou · 2025-01-06T16:10:59Z

It looks like pypa/setuptools#4662 is now available in setuptools 75.7.0. I will try to undo the custom workaround here.

pitrou · 2025-01-06T16:16:53Z

@github-actions crossbow submit wheel-windows*

pitrou · 2025-01-06T16:20:55Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T16:23:26Z

Revision: b3ce08c

Submitted crossbow builds: ursacomputing/crossbow @ actions-ec43532048

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

.pre-commit-config.yaml

ci/docker/python-free-threaded-wheel-windows-test-vs2019.dockerfile

ci/docker/python-free-threaded-wheel-windows-vs2019.dockerfile

ci/docker/python-wheel-windows-test-vs2019-base.dockerfile

docker-compose.yml

kou · 2025-01-07T00:18:19Z

ci/docker/python-wheel-windows-vs2019-base.dockerfile

+
+# based on mcr.microsoft.com/windows/servercore:ltsc2019
+# contains choco and vs2019 preinstalled
+FROM abrarov/msvc-2019:2.11.0


Can we use python-wheel-windows-test-vs2019-base.dockerfile as the base image?

Why would we? That would make things more complicated.

Because I felt that python-wheel-windows-{,test-}vs2019-base.dockerfile have many duplication. But I re-read them now, there are not so much duplication. Sorry.

pitrou · 2025-01-07T09:32:58Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-07T09:35:22Z

Revision: 764437b

Submitted crossbow builds: ursacomputing/crossbow @ actions-f31cf1d015

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-07T10:57:28Z

Updated CI still green, this is ready for review again @kou

kou

+1

kou · 2025-01-07T20:42:10Z

.env

+PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-06
+PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-06a


Can we avoid the a suffix?

Suggested change

PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-06

PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-06a

PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-08

PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-08

kou · 2025-01-07T20:44:34Z

.pre-commit-config.yaml

@@ -39,7 +39,7 @@ repos:
        files: >-
          (
          ?^ci/docker/conda-python-emscripten\.dockerfile$|
-          ?^ci/docker/python-wheel-windows-test-vs2019\.dockerfile$|
+          ?^ci/docker/python-.*-wheel-windows-test-vs2019.*\.dockerfile$|


Does this match python-wheel-windows-vs2019-base.dockerfile?

Probably not, but this is asking for more and more things that are outside of the scope of this PR...

Also, the diagnostics emitted by hadolint don't seem terribly useful, especially when we have to silence most of them.

kou · 2025-01-07T20:47:22Z

ci/docker/python-free-threaded-wheel-windows-test-vs2019.dockerfile

+# when you update this file.
+
+ARG base
+# hadolint ignore=DL3006


Could you add a comment that explains what is DL3006?
A link to https://github.com/hadolint/hadolint/wiki/DL3006 and the summary in the page ("Always tag the version of an image explicitly.") will be enough.

Uh, sure. But is hadolint actually useful? It seems terribly limited.

kou · 2025-01-07T20:50:14Z

ci/scripts/python_wheel_windows_build.bat

-echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+echo "=== (%PYTHON%) Clear output directories and leftovers ==="


Ah, I missed it.

lysnikolaou requested review from assignUser, jonkeane, kou and raulcd as code owners November 21, 2024 11:25

github-actions bot added the awaiting review Awaiting review label Nov 21, 2024

raulcd reviewed Nov 27, 2024

View reviewed changes

ci/docker/python-wheel-windows-test-vs2019.dockerfile Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Nov 27, 2024

pitrou force-pushed the windows-free-threaded-wheels branch from 95a0a53 to 83a76e3 Compare December 18, 2024 17:31

github-actions bot added the Component: Python label Dec 18, 2024

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the windows-free-threaded-wheels branch from 7ebb73e to b3ce08c Compare January 6, 2025 16:20

github-actions bot removed the Component: Python label Jan 6, 2025

kou reviewed Jan 7, 2025

View reviewed changes

Add configuration for building & testing free-threaded wheels on Windows

097c9c6

pitrou force-pushed the windows-free-threaded-wheels branch from 4c9872d to 097c9c6 Compare January 7, 2025 09:29

Lint more Dockerfiles

764437b

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Jan 7, 2025

kou approved these changes Jan 7, 2025

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Jan 7, 2025

		PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-06
		PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-06a

		echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
		echo "=== (%PYTHON%) Clear output directories and leftovers ==="

GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

Are you sure you want to change the base?

GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

Conversation

lysnikolaou commented Nov 21, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Nov 21, 2024

lysnikolaou commented Nov 26, 2024

raulcd commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

raulcd commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

raulcd left a comment

Choose a reason for hiding this comment

lysnikolaou commented Nov 27, 2024

lysnikolaou commented Nov 27, 2024

raulcd commented Dec 2, 2024

lysnikolaou commented Dec 2, 2024

pitrou commented Dec 4, 2024

pitrou commented Dec 4, 2024

pitrou commented Dec 4, 2024

pitrou commented Dec 18, 2024

pitrou commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

pitrou commented Dec 18, 2024 • edited Loading

pitrou commented Dec 18, 2024

pitrou commented Jan 6, 2025

pitrou commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

pitrou commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

pitrou commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

pitrou commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

pitrou commented Jan 6, 2025

pitrou commented Jan 6, 2025

pitrou commented Jan 6, 2025

This comment was marked as outdated.

pitrou commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

pitrou commented Jan 7, 2025

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysnikolaou commented Nov 21, 2024 •

edited by github-actions bot

Loading

pitrou commented Dec 18, 2024 •

edited

Loading

pitrou Jan 7, 2025 •

edited

Loading