Unrecognized Mach-O magic

[sashkab]

I'm maintaining homebrew tap for universal python and need to keep universal openssl as well. Several times I attempted and failed to to switch from using lipo to MachO::Tools.merge_machos, but this time trying to figure out what I am doing wrong.

With lipo --create everything works as expected:

$ lipo -create /tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-i386/libcrypto.1.0.0.dylib /tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-x86_64/libcrypto.1.0.0.dylib -output /tmp/xyz.dylib
$ lipo -info /tmp/xyz.dylib
Architectures in the fat file: /tmp/xyz.dylib are: i386 x86_64
$ file /tmp/xyz.dylib
/tmp/xyz.dylib: Mach-O universal binary with 2 architectures: [i386:Mach-O dynamically linked shared library i386] [x86_64]
/tmp/xyz.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
/tmp/xyz.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64

and

$ file /tmp/zzz.a
/tmp/zzz.a: Mach-O universal binary with 2 architectures: [i386:current ar archive random library] [x86_64]
/tmp/zzz.a (for architecture i386):	current ar archive random library
/tmp/zzz.a (for architecture x86_64):	current ar archive random library
$ lipo -info /tmp/zzz.a
Architectures in the fat file: /tmp/zzz.a are: i386 x86_64

When I launch brew irb:

irb(main):004:0> libs = ["/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-i386/libcrypto.1.0.0.dylib","/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-x86_64/libcrypto.1.0.0.dylib"]
=> ["/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-i386/libcrypto.1.0.0.dylib", "/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-x86_64/libcrypto.1.0.0.dylib"]
irb(main):005:0> MachO::Tools.merge_machos("/tmp/xxx.dylib", *libs)
=> 3643072
$ lipo -info /tmp/xxx.dylib
fatal error: /Library/Developer/CommandLineTools/usr/bin/lipo: offset 1652032 of fat file /tmp/xxx.dylib (cputype (16777223) cpusubtype (3)) not aligned on its alignment (2^8)
$ file /tmp/xxx.dylib
/tmp/xxx.dylib: Mach-O universal binary with 2 architectures: [i386:Mach-O dynamically linked shared library i386] [x86_64]
/tmp/xxx.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
/tmp/xxx.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64

and

irb(main):012:0> static=["/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-i386/libcrypto.a", "/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-x86_64/libcrypto.a"]
=> ["/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-i386/libcrypto.a", "/tmp/uopenssl-20180702-16887-vjpn5r/openssl-1.0.2o/build-x86_64/libcrypto.a"]
irb(main):013:0> MachO::Tools.merge_machos("/tmp/xxx.a", *static)
MachO::MagicError: Unrecognized Mach-O magic: 0x213c6172
	from /usr/local/Homebrew/Library/Homebrew/vendor/macho/macho.rb:35:in `open'
	from /usr/local/Homebrew/Library/Homebrew/vendor/macho/macho/tools.rb:95:in `block in merge_machos'
	from /usr/local/Homebrew/Library/Homebrew/vendor/macho/macho/tools.rb:94:in `map'
	from /usr/local/Homebrew/Library/Homebrew/vendor/macho/macho/tools.rb:94:in `merge_machos'
	from (irb):13
	from /usr/local/Homebrew/Library/Homebrew/brew.rb:100:in `<main>'

What am I doing wrong, or did I ran into some kind of bug?

[woodruffw]

Looks like a bug to me. Could you send me the binaries used in both of the cases above?

[sashkab]

Files are attached.

ssl.zip

[sashkab]

Forgot to mention I'm using macOS 10.13.5, if that matters.

[woodruffw]

Thanks! I'll take a look at these tomorrow morning.

Being on 10.13.5 shouldn't change anything -- ruby-macho doesn't interact with the system at all besides through Ruby, and these errors don't look like the kind caused by changes to the Mach-O format itself.

[woodruffw]

Found the problem/one of the problems: fat Mach-Os expect to be aligned based on the fat_arch.align field, as a power of 2, but I'm not respecting that. Even worse, when I'm serializing fat Mach-Os, I'm filling fat_arch.align in with macho.alignment, which is a totally wrong (since that's usually 4 or 8, not 12 as would align with a memory page).

I'll fix these issues now.

[woodruffw]

#100 contains the WIP branch.

[woodruffw]

BTW, MachO::MagicError you're getting with the static libraries (libcrypto.a) is expected -- ruby-macho doesn't parse ar files, only flat/direct Mach-Os.

If you want to parse the contents of an ar, you can use ar x ${file} to extract the object files first and then feed those into ruby-macho 🙂