Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error installing gem in MacOS M1 #34

Closed
AlessandroTolomio opened this issue Apr 18, 2024 · 9 comments
Closed

Error installing gem in MacOS M1 #34

AlessandroTolomio opened this issue Apr 18, 2024 · 9 comments

Comments

@AlessandroTolomio
Copy link 

Installing tiktoken_ruby 0.0.8 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.0.0 -r ./siteconf20240418-10855-35ug4t.rb extconf.rb
checking for gcc... yes
checking for g++... yes
checking for gcc-ar... yes
checking for cargo... no

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
make DESTDIR\= clean

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
make DESTDIR\=
info: downloading installer
info: profile set to 'minimal'
info: default host triple is aarch64-unknown-linux-gnu
info: skipping toolchain installation


Rust is installed now. Great!

To get started you need Cargo's bin directory 
(/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
in your PATH
environment variable. This has not been done automatically.

To configure your current shell, you need to source
the corresponding env file under 
/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo.

This is usually done by running one of the following (note the leading DOT):
. 
"/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
# For sh/bash/zsh/ash/dash/pdksh
source 
"/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/env.fish"
# For fish
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2024-04-09, rust version 1.77.2 (25ef9e3d8 2024-04-09)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'

  stable-aarch64-unknown-linux-gnu installed - rustc 1.77.2 (25ef9e3d8 2024-04-09)

info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'
info: checking for self-update
info: using existing install for 'stable-aarch64-unknown-linux-gnu'
info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'

  stable-aarch64-unknown-linux-gnu unchanged - rustc 1.77.2 (25ef9e3d8 2024-04-09)

info: note that the toolchain 'stable-aarch64-unknown-linux-gnu' is currently in use (environment override by RUSTUP_TOOLCHAIN)
generating target/release/libtiktoken_ruby.so (release)
/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release -- -C linker=gcc -L native=/usr/local/lib -C link-arg=-lm
    Updating crates.io index
    Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
    Updating git submodule `https://github.com/zurawiki/tiktoken`
 Downloading crates ...
  Downloaded aho-corasick v0.7.20
  Downloaded magnus-macros v0.6.0
  Downloaded shlex v1.1.0
  Downloaded autocfg v1.1.0
  Downloaded bit-set v0.5.3
  Downloaded either v1.10.0
  Downloaded cfg-if v1.0.0
  Downloaded glob v0.3.1
  Downloaded rb-sys-env v0.1.2
  Downloaded seq-macro v0.3.5
  Downloaded rustc-hash v1.1.0
  Downloaded scopeguard v1.1.0
  Downloaded once_cell v1.17.1
  Downloaded smallvec v1.10.0
  Downloaded quote v1.0.33
  Downloaded parking_lot_core v0.9.7
  Downloaded parking_lot v0.12.1
  Downloaded unicode-ident v1.0.8
  Downloaded serde v1.0.157
  Downloaded minimal-lexical v0.2.1
  Downloaded regex-automata v0.1.10
  Downloaded syn v2.0.31
  Downloaded bstr v1.4.0
  Downloaded libc v0.2.140
  Downloaded regex-syntax v0.6.28
  Downloaded regex v1.7.1
  Downloaded nom v7.1.3
  Downloaded magnus v0.6.1
  Downloaded itertools v0.12.1
  Downloaded bindgen v0.69.4
  Downloaded fancy-regex v0.11.0
  Downloaded base64 v0.21.0
  Downloaded proc-macro2 v1.0.66
  Downloaded memchr v2.5.0
  Downloaded lock_api v0.4.9
  Downloaded clang-sys v1.6.0
  Downloaded anyhow v1.0.70
  Downloaded shell-words v1.1.0
  Downloaded libloading v0.7.4
  Downloaded bitflags v2.4.0
  Downloaded rb-sys-build v0.9.87
  Downloaded cexpr v0.6.0
  Downloaded rb-sys v0.9.87
  Downloaded lazycell v1.3.0
  Downloaded lazy_static v1.4.0
  Downloaded bit-vec v0.6.3
   Compiling memchr v2.5.0
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.8
   Compiling glob v0.3.1
   Compiling libc v0.2.140
   Compiling cfg-if v1.0.0
   Compiling minimal-lexical v0.2.1
   Compiling libloading v0.7.4
   Compiling clang-sys v1.6.0
   Compiling aho-corasick v0.7.20
   Compiling nom v7.1.3
   Compiling quote v1.0.33
   Compiling syn v2.0.31
   Compiling regex-syntax v0.6.28
   Compiling either v1.10.0
   Compiling bindgen v0.69.4
   Compiling itertools v0.12.1
   Compiling cexpr v0.6.0
   Compiling lazycell v1.3.0
   Compiling rustc-hash v1.1.0
   Compiling shlex v1.1.0
   Compiling regex v1.7.1
   Compiling bitflags v2.4.0
   Compiling lazy_static v1.4.0
   Compiling shell-words v1.1.0
   Compiling autocfg v1.1.0
   Compiling lock_api v0.4.9
   Compiling parking_lot_core v0.9.7
   Compiling anyhow v1.0.70
   Compiling scopeguard v1.1.0
   Compiling bit-vec v0.6.3
   Compiling smallvec v1.10.0
   Compiling rb-sys-env v0.1.2
   Compiling magnus v0.6.1
   Compiling bit-set v0.5.3
   Compiling regex-automata v0.1.10
   Compiling once_cell v1.17.1
   Compiling magnus-macros v0.6.0
   Compiling bstr v1.4.0
   Compiling rb-sys-build v0.9.87
   Compiling fancy-regex v0.11.0
   Compiling parking_lot v0.12.1
   Compiling base64 v0.21.0
   Compiling rb-sys v0.9.87
   Compiling seq-macro v0.3.5
   Compiling tiktoken-rs v0.3.2 (https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
error: failed to run custom build command for `rb-sys v0.9.87`

Caused by:
  process didn't exit successfully: `/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/target/release/build/rb-sys-c65a4afa838c9b06/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
  cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
  cargo:rerun-if-env-changed=RBCONFIG_platform
  cargo:rerun-if-env-changed=RBCONFIG_arch
  cargo:rerun-if-env-changed=RUBY_ROOT
  cargo:rerun-if-env-changed=RUBY_VERSION
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-changed=build/version.rs
  cargo:rerun-if-changed=build/features.rs
  cargo:rerun-if-changed=build/stable_api_config.rs
  cargo:rerun-if-changed=build/main.rs
  cargo:rerun-if-env-changed=RUBY_STATIC
  cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_MAJOR
  cargo:rerun-if-env-changed=RBCONFIG_MINOR
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS

  --- stderr
INFO: using bindgen with clang args: ["-I/usr/local/include/ruby-3.0.0", "-I/usr/local/include/ruby-3.0.0/aarch64-linux", "-fms-extensions", "-O3", "-ggdb3", "-Wall", "-Wextra", "-Wdeprecated-declarations", "-Wduplicated-cond", "-Wimplicit-function-declaration", "-Wimplicit-int",
"-Wmisleading-indentation", "-Wpointer-arith", "-Wwrite-strings", "-Wimplicit-fallthrough=0", "-Wmissing-noreturn", "-Wno-cast-function-type", "-Wno-constant-logical-operand", "-Wno-long-long", "-Wno-missing-field-initializers", "-Wno-overlength-strings", "-Wno-packed-bitfield-compat",
"-Wno-parentheses-equality", "-Wno-self-assign", "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", "-Wunused-variable"]
  #include "ruby.h"

  #ifdef HAVE_RUBY_DEBUG_H
  #include "ruby/debug.h"
  #endif
  #ifdef HAVE_RUBY_DEFINES_H
  #include "ruby/defines.h"
  #endif
  #ifdef HAVE_RUBY_ENCODING_H
  #include "ruby/encoding.h"
  #endif
  #ifdef HAVE_RUBY_FIBER_SCHEDULER_H
  #include "ruby/fiber/scheduler.h"
  #endif
  #ifdef HAVE_RUBY_INTERN_H
  #include "ruby/intern.h"
  #endif
  #ifdef HAVE_RUBY_IO_H
  #include "ruby/io.h"
  #endif
  #ifdef HAVE_RUBY_MEMORY_VIEW_H
  #include "ruby/memory_view.h"
  #endif
  #ifdef HAVE_RUBY_MISSING_H
  #include "ruby/missing.h"
  #endif
  #ifdef HAVE_RUBY_ONIGMO_H
  #include "ruby/onigmo.h"
  #endif
  #ifdef HAVE_RUBY_ONIGURUMA_H
  #include "ruby/oniguruma.h"
  #endif
  #ifdef HAVE_RUBY_RACTOR_H
  #include "ruby/ractor.h"
  #endif
  #ifdef HAVE_RUBY_RANDOM_H
  #include "ruby/random.h"
  #endif
  #ifdef HAVE_RUBY_RE_H
  #include "ruby/re.h"
  #endif
  #ifdef HAVE_RUBY_REGEX_H
  #include "ruby/regex.h"
  #endif
  #ifdef HAVE_RUBY_RUBY_H
  #include "ruby/ruby.h"
  #endif
  #ifdef HAVE_RUBY_ST_H
  #include "ruby/st.h"
  #endif
  #ifdef HAVE_RUBY_THREAD_H
  #include "ruby/thread.h"
  #endif
  #ifdef HAVE_RUBY_THREAD_NATIVE_H
  #include "ruby/thread_native.h"
  #endif
  #ifdef HAVE_RUBY_UTIL_H
  #include "ruby/util.h"
  #endif
  #ifdef HAVE_RUBY_VERSION_H
  #include "ruby/version.h"
  #endif
  #ifdef HAVE_RUBY_VM_H
  #include "ruby/vm.h"
  #endif
  #ifdef HAVE_RUBY_WIN32_H
  #include "ruby/win32.h"
  #endif
  #ifdef HAVE_RUBY_IO_BUFFER_H
  #include "ruby/io/buffer.h"
  #endif
  #ifdef HAVE_RUBY_ATOMIC_H
  #include "ruby/atomic.h"
  #endif
  struct rb_sys__Opaque__RString { struct RString dummy; };
  struct rb_sys__Opaque__RArray { struct RArray dummy; };
  thread 'main' panicked at /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.69.4/lib.rs:622:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:552: target/release/libtiktoken_ruby.so] Error 101

make failed, exit code 2

Gem files will remain installed in /usr/local/bundle/gems/tiktoken_ruby-0.0.8 for inspection.
Results logged to /usr/local/bundle/extensions/aarch64-linux/3.0.0/tiktoken_ruby-0.0.8/gem_make.out

  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:93:in `run'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:44:in `block in make'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `each'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `make'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:63:in `block in build'
  /usr/local/lib/ruby/3.0.0/tempfile.rb:317:in `open'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:159:in `build_extension'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:193:in `block in build_extensions'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `each'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `build_extensions'
  /usr/local/lib/ruby/3.0.0/rubygems/installer.rb:837:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/rubygems_gem_installer.rb:76:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/source/rubygems.rb:205:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/parallel_installer.rb:132:in `do_install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/parallel_installer.rb:123:in `block in worker_pool'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:62:in `apply_func'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:54:in `loop'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:54:in `process_queue'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.8), and Bundler cannot continue.
@gjtorikian
Copy link
Collaborator

Hello, is this still an issue for you? I'm on an M1 as well and haven't had any installation problems.

@MarioRuiz
Copy link

Installing 0.0.8 and 0.0.9 on Ubuntu 20.04.6 LTS (Focal Fossa) Same issue

@kwent
Copy link

kwent commented Jan 2, 2025 •
edited
Loading

Same issue here. Resolved on ubuntu by doing

apt-get install clang libclang-dev

@gjtorikian
Copy link
Collaborator

I want to help resolve this, but there seem to be two problems here:

  1. The original report refers to an M1 mac, but;
  2. The last two reports were about Ubuntu

I have a fairly good idea of what's going on: the gem is trying to manually compile on your system(s), your system doesn't have the proper build tools to compile it (eg., libclang), and the installation fails. But: the gem is already precompiled for these systems:

No one should have needed to compile the gem at all.

So, first I would ask, for the Ubuntu users:

  • What is your exact OS, CPU chipset, and Ruby version being used?
  • For the macOS users, same question; otherwise I'm going to assume that the Mac issue is resolved, but the Linux one is not

@kevindew
Copy link

kevindew commented Jan 7, 2025

I think the problem here is that there is different Ruby version constraints on the pre-compiled versions of the gem.

For 0.0.10 without precompile the constraint is >= 3.1.0
Whereas for the precompiled versions such as aarch64-linux the Ruby constraint is >= 3.2, < 3.3.dev

which means it's only available for Ruby 3.2.x versions.

Trying on a docker VM with bundle platform of aarch64-linux I can install with Ruby 3.2.4:

Installing tiktoken_ruby 0.0.10 (aarch64-linux)

whereas with Ruby 3.3 and 3.4 it tries to install with native extensions.

I expect the issue is in this line:

tiktoken_ruby/script/release

Line 24 in 020781a

bundle exec rb-sys-dock -p "$target" --ruby-versions 3.2 --build

Where I think you may want to omit the ruby-version or comma separate all supported ones

@gjtorikian
Copy link
Collaborator

To be sure, that line is a problem, yes. But, that script was added only for the 0.0.10 release; the reports before my comment were about 0.0.8 and 0.0.9.

(There’s actually also separate problem with the build for 3.3.5+ and 3.4 (oxidize-rb/rb-sys#475) that I’m keeping track of. I incorrectly hardcoded this to 3.2 after being unable to build with 3.3 and 3.4.)

@kevindew
Copy link

kevindew commented Jan 7, 2025

Sure, sorry about not direct relation. Would you like a fresh issue created for the 3.2 builds?

@gjtorikian
Copy link
Collaborator

No, my curse is that I’m fully aware of every problem 😆

My plan was to wait for the build issues to resolve, push a 0.0.11, and then close this issue. I mentioned earlier that I had an M1 with no issues; subsequent comments were about Ubuntu. I’ll do a better job looking at all the artifacts built for the 0.0.11 gem before pushing the release, but ultimately the problem is the same: for some reason, for some architecture types, the native gem is not being compiled as part of the packaging.

No one should need Rust tooling installed to use this gem: that’s the core problem that I intend to look into and fix. It’s just been exacerbated by this 3.3.5+ issue.

@gjtorikian gjtorikian mentioned this issue Jan 8, 2025
Merged
@gjtorikian
Copy link
Collaborator

I'm fairly certain this has been fixed as of 0.0.11, for both macOS and Linux. Please try it out. We can reopen the issue if it didn't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@gjtorikian @kevindew @MarioRuiz @kwent @AlessandroTolomio