Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[red-knot] Typeshed patching: use build.rs instead of workflow #15370

Merged
merged 2 commits into from
Jan 9, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .github/workflows/sync_typeshed.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,6 @@ jobs:
cp -r typeshed/stdlib ruff/crates/red_knot_vendored/vendor/typeshed/stdlib
rm -rf ruff/crates/red_knot_vendored/vendor/typeshed/stdlib/@tests
git -C typeshed rev-parse HEAD > ruff/crates/red_knot_vendored/vendor/typeshed/source_commit.txt
# Patch the typeshed stubs to include `knot_extensions`
ln -s ../../../knot_extensions/knot_extensions.pyi ruff/crates/red_knot_vendored/vendor/typeshed/stdlib/
echo "# Patch applied for red_knot:" >> ruff/crates/red_knot_vendored/vendor/typeshed/stdlib/VERSIONS
echo "knot_extensions: 3.0-" >> ruff/crates/red_knot_vendored/vendor/typeshed/stdlib/VERSIONS
- name: Commit the changes
id: commit
if: ${{ steps.sync.outcome == 'success' }}
Expand Down
17 changes: 15 additions & 2 deletions crates/red_knot_vendored/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
//! in `crates/red_knot_vendored/vendor/typeshed` change.

use std::fs::File;
use std::io::Write;
use std::path::Path;

use path_slash::PathExt;
Expand All @@ -14,6 +15,7 @@ use zip::write::{FileOptions, ZipWriter};
use zip::CompressionMethod;

const TYPESHED_SOURCE_DIR: &str = "vendor/typeshed";
const KNOT_EXTENSIONS_STUBS: &str = "knot_extensions/knot_extensions.pyi";
const TYPESHED_ZIP_LOCATION: &str = "/zipped_typeshed.zip";

/// Recursively zip the contents of an entire directory.
Expand Down Expand Up @@ -55,21 +57,32 @@ fn zip_dir(directory_path: &str, writer: File) -> ZipResult<File> {
// Some unzip tools unzip files with directory paths correctly, some do not!
if absolute_path.is_file() {
println!("adding file {absolute_path:?} as {normalized_relative_path:?} ...");
zip.start_file(normalized_relative_path, options)?;
zip.start_file(&*normalized_relative_path, options)?;
let mut f = File::open(absolute_path)?;
std::io::copy(&mut f, &mut zip).unwrap();

// Patch the VERSIONS file to make `knot_extensions` available
if normalized_relative_path == "stdlib/VERSIONS" {
writeln!(&mut zip, "knot_extensions: 3.0-")?;
}
Comment on lines +64 to +67
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I originally wrote this function, I intended to write it so that it could be used to zip up any directory into a zip archive -- that's why it's called zip_dir(), and that's why it accepts the typeshed directory path as a parameter rather than hardcoding it in the function. I think since I originally wrote it, it's already grown a few features that make it quite specific to its use-case here, such as the conditional logic a few lines above for deciding what the compression method should be. So I don't really mind much hardcoding things like "stdlib/VERSIONS" directly into the function body. But at this stage, it probably doesn't make much sense for the typeshed directory to be passed into the function as a parameter, and we should probably change the name of the function 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about that, I figured it would be okay to modify in this way since it wasn't being used anywhere else and not in a place where it could be imported from anywhere else. See #15372

} else if !normalized_relative_path.is_empty() {
// Only if not root! Avoids path spec / warning
// and mapname conversion failed error on unzip
println!("adding dir {absolute_path:?} as {normalized_relative_path:?} ...");
zip.add_directory(normalized_relative_path, options)?;
}
}

// Patch typeshed and add the stubs for the `knot_extensions` module
println!("adding file {KNOT_EXTENSIONS_STUBS} as stdlib/knot_extensions.pyi ...");
zip.start_file("stdlib/knot_extensions.pyi", options)?;
let mut f = File::open(KNOT_EXTENSIONS_STUBS)?;
std::io::copy(&mut f, &mut zip).unwrap();

zip.finish()
}

fn main() {
println!("cargo::rerun-if-changed={TYPESHED_SOURCE_DIR}");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this means that we will rebuild red_knot_vendored if any file in the package has changed (ref). I need this, because knot_extensions.pyi is not in the vendor/typeshed directory, and we need to pick up changes to it.

And I don't think there's a huge drawback since there are no files in red_knot_vendored that should not trigger a rebuild except for the README.

assert!(
Path::new(TYPESHED_SOURCE_DIR).is_dir(),
"Where is typeshed?"
Expand Down
2 changes: 0 additions & 2 deletions crates/red_knot_vendored/vendor/typeshed/stdlib/VERSIONS
Original file line number Diff line number Diff line change
Expand Up @@ -341,5 +341,3 @@ zipfile._path: 3.12-
zipimport: 3.0-
zlib: 3.0-
zoneinfo: 3.9-
# Patch applied for red_knot:
knot_extensions: 3.0-

This file was deleted.

Loading