cmd/link: unexpected bindingNone in '_go.buildid' #42082
Comments
|
Thanks for the report. It would be helpful to include some details on your Darwin system, e.g. OS version, Xcode version, and the output of "clang --version". |
|
This is consistently failing on dev laptops with macOS 10.15.7 in Uber. The detail link command printed out from Bazel rules_go contains some information about the environment: |
|
Thanks. Not sure right off the bat what might be causing this, but if there is any way you can create a shareable reproducer, that would be very helpful. Or possibly share the intermediates being created by the Go linker, if that is an option (via "mkdir /tmp/xxx ; go build ... -ldflags=-tmpdir=/tmp/xxx). |
cagedmantis
changed the title
cagedmantis
added
the
NeedsInvestigation
|
Unfortunately, we're not able currently able to publish our existing repro case. We are working on creating a repo case from open source components! I'll look into sharing the intermediates generated by the linker. |
|
I've been digging around in the linker intermediates of a different project with the same error: the only place with a reference to I don't know if that's relevant, since I don't have much understanding of how the linker works, but the |
|
@thanm, if we can get approval on our side. Is there a process by which we can share the linker intermediates through a private channel? |
That would be fine with me, although as far as I know we don't have an official policy or mechanism for such side channel communications. If you'd like to set something up, please email me (userid at google.com is same as github name). |
|
Another thing that would help would be if you could run the failing link by hand and then inspect the intermediates with "gobjdump". What I'm looking for is more info about the offending symbol, "_go.buildid". This is a text symbol that is emitted by the Go linker when linking on non-ELF systems. It is placed at the very start of the text segment (offset 0), and will contain a payload built from the value of the -buildid option passed to the linker (normally a hash like -buildid=gxOfactYKvIT8obo4Tx9/BBqcYSuatFUb4E_Y6MER/q64SD6zppB19Re5mq7xb/gxOfactYKvIT8obo4Tx9). The "_go.buildid" symbol should not have any relocations itself, nor should there be any relocations that point to it as a target symbol. Here's an example of how to dump the required info. The failure message from the linker refers to bindingNone, which (based on my very limited understanding of the Darwin linker) has to do with relocations -- so what I am trying to determine is if somehow there are relocations on or against the symbol (which seems impossible, but I don't have any other theories at the moment). |
|
Sorry, that should read in the recipe I sent. |
|
What would a problematic or unexpected relocation look like? This was the output for the linker intermediates, which is the same as the example you've given above; I'm guessing that that's the expected output. |
|
Yes, that's the expected "normal" output. So at this point I'm not sure what the problem might be-- more info needed. I think we're back where we started then -- to make progress I think we need a smaller reproducer or some way to inspect the linker intermediates. |
|
We're still looking into sharing artifacts. Meanwhile although I haven't found a reproducible case yet, I did find some interesting bits while investigating a bit further using It seems like the "unexpected bindingNone" exception is actually being thrown while throwing another exception: Patching up the code to throw the original exception results in something like: We have different projects runniing into this issue: one using (for referencee gobuild-pass is a simple cgo project taken from cgo docs, see https://gist.github.com/robbertvanginkel/05445fb48537f3b3b1ad8c51630685c4) I don't think those showed up for Some extra disassembly details that might be useful.I was initially looking at the contents of the object files, which all seemed to be the same when using `hexdump` or `objdump --disassemble=` However, using a more advanced disassembler tool (hopper) they showed somewhat different: I suppose that's due to resolving the fixups. Hopefully this can help in narrowing it down. |
|
Thanks. Looking at the ObjectDump and "otool -r" output is helpful, it does indeed confirm that there is a (bogus) relocation on _go.buildid; this should never happen. This points to a bug in the Go linker, but without a debuggable/buildable test case, I'm not sure how to go about locating the bug. One other thing that would be helpful would be if you could try your testcase using a tip version of Go (e.g. download and build Go source from https://go.googlesource.com/go). There have been a number of linker changes between Go 1.15 and the current trunk (Go 1.16 candidate). It's possible that the 1.16/trunk linker might fail in more descriptive or helpful way. Thanks. |
|
Using a recent commit ( I was considering a bisect, but a quick look through the history shows that the linker work was mostly done on a branch that was merged into master as of 52fe92f. Compiling this project at that commit and its parent on master, it fails at 2bfa45c but passes at 52fe92f. Would bisecting the |
|
@robbertvanginkel yes, bisecting on the dev.link branch is a viable option in this case. Current dev.link is a bit out of sync with trunk at this point, but in this case it should give you what you need. My guess at this point (based on the symptoms) is that this was a problem with over-aggressive unreachable method pruning; there was a fair amount of activity related to this on the dev.link branch between 1.14 and 1.15. |
|
Hmm, 59a702a is a bit surprising as a candidate. That commit shouldn't actually have any change on what the linker emits, it's just adding in some additional parallelism in the way that Macho relocations are emitted. Maybe the 1.14-1.15 bisect might provide some more insight. |
|
Tried another bisect between e77c99c and 1667b35, which seems to suggest 7466cad is when this started. Bisect logs for reference
git bisect log for #42082 (comment). Note how good/bad have reversed meaning here as I was searching from broken -> fixed. |
|
Change https://golang.org/cl/270941 mentions this issue: |
|
Change https://golang.org/cl/270942 mentions this issue: |
|
I cherry-picked https://golang.org/cl/270942 onto the 1.15.5 tag and rebuild the failing project to verify and it seems this solves the issue for our go-build project. Going to verify the bazel project too, but so far so good! |
|
Using the same go build from cherry-picking https://golang.org/cl/270942 onto the 1.15.5 tag also works for the bazel project, so I'm fairly confident that must have been it. |
|
@robbertvanginkel thanks for confirming! |
Immediately after a forward Seek, the offset we're writing to is
beyond len(buf)+len(heap):
|<--- buf --->|<--- heap --->|
^
off
If we do a copyHeap at this point, the new heapPos should not be
0:
|<---------- buf ----------->|<-heap->|
^
off
Recompute it.
For #42082.
Change-Id: Icb3e4e1c7bf7d1fd3d76a2e0d7dfcb319c661534
Reviewed-on: https://go-review.googlesource.com/c/go/+/270941
Trust: Cherry Zhang <[email protected]>
Run-TryBot: Cherry Zhang <[email protected]>
TryBot-Result: Go Bot <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
cherrymui
added
the
CherryPickCandidate
|
@cherrymui Should this issue target Go1.16 milestone, and once fixed, a backport of it requested via the process described at https://golang.org/wiki/MinorReleases? Or is there a reason not to follow that process in this case? |
|
From #42082 (comment), I see this issue is already resolved on tip via changes in @linzhp Is it known whether this problem applies to Go 1.14 as well, or just 1.15? |
|
@dmitshur this only applies to 1.15, we have not experienced the same for 1.14. |
|
@dmitshur @cherrymui any idea if/when https://go-review.googlesource.com/c/go/+/270942/ might be approved for cherry-pick and when a 1.15 patch release could be available? Want to throw a voice out there that this solves a real regression for us that was introduced in 1.15 and we'd like to be able to use the latest release again :) |
|
I'd like the CL to be cherry-picked. But I don't know when. Probably soon, as we do minor releases at beginning of each month, if I understand correctly. @dmitshur may have better answer. |
|
It would help make the cherry-pick review process easier and more consistent if we follow the process documented at https://golang.org/wiki/MinorReleases. @cherrymui Is it okay to make this issue target the Go 1.16 milestone, mark it as fixed by CL 270941, then request a new backport issue for Go 1.15, and update the commit message of CL 270942 accordingly? Creating a new backport issue gives a chance to highlight the short rationale for why this backport is needed. |
|
Yeah, feel free to do whatever that makes your workflow easier. I didn't make this issue target Go 1.16 as it just doesn't fail at Go 1.16. (CL 270941 is still nice to have.) |
|
@gopherbot please consider this for backport to 1.15, it's a linker regression that prevents some programs from linking succesfully. |
|
Backport issue(s) opened: #42948 (for 1.15). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases. |
|
I read through https://github.com/golang/go/wiki/MinorReleases and tried to follow the process by having the bot create #42948. With that I think the CherryPickCandidate/NeedsInvestigation labels can be removed from this issue and it can be resolved as b194b51 is in master. |
|
Thank you for doing that @robbertvanginkel. As discussed above, I'll make this issue target Go 1.16 and close it since this issue is resolved on tip. |
dmitshur
added
NeedsFix
Immediately after a forward Seek, the offset we're writing to is
beyond len(buf)+len(heap):
|<--- buf --->|<--- heap --->|
^
off
If we do a copyHeap at this point, the new heapPos should not be
0:
|<---------- buf ----------->|<-heap->|
^
off
Recompute it.
Updates #42082
Fixes #42948
Change-Id: Icb3e4e1c7bf7d1fd3d76a2e0d7dfcb319c661534
Reviewed-on: https://go-review.googlesource.com/c/go/+/270942
Run-TryBot: Carlos Amedee <[email protected]>
TryBot-Result: Go Bot <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
Reviewed-by: Jeremy Faller <[email protected]>
Trust: Cherry Zhang <[email protected]>
After upgrading to Go 1.15, the linker fails on some packages with
unexpected bindingNone in '_go.buildid'. This only happens on Darwin but not Linux. We haven't been able to construct the small example to reproduce the issue.The issues seems to related to the size of the linked binary. For example, a test binary that fails to link in Go 1.15 is 330MB when it's linked in Go 1.14.
Similar errors:
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Building some packages with either
go buildorbazel buildWhat did you expect to see?
Build succeeds
What did you see instead?
Linker fails with the following error:
The text was updated successfully, but these errors were encountered: