panic in _Cfunc_go_openssl_EVP_PKEY_derive_wrapper called from ExtractHKDF

[jmgvfr04]

Hello,

We are seeing an occasional panic in our go binary in the openssl stack.
This happens to be on SUSE Linux Enterprise Server. We have not seen it on Red Hat Linux, although we use the same binary for both so that seems to be just a coincidence.

Any pointers as to what this might be or how to avoid it would be much appreciated.

Thanks!
Joseph Milton Gilpin
Cohesity (f.k.a. Veritas)

Here is the panic stack (ms1.23.4-1 is our own convention which just indicates that we are using the MS FIPS compliant go version 1.23.4-1):

SIGSEGV: segmentation violation
PC=0x7f0e61a90cb1 m=17 sigcode=1 addr=0x40
signal arrived during cgo execution

goroutine 2336826 gp=0xc000532fc0 m=17 mp=0xc00007ca08 [syscall]:
runtime.cgocall(0xc8a990, 0xc001de57d8)
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/runtime/cgocall.go:167 +0x4b fp=0xc001de57b0 sp=0xc001de5778 pc=0x46dfcb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_PKEY_derive_wrapper(0x7f0e4802f5d0, 0xc001feac00, 0x30)
_cgo_gotypes.go:1656 +0x54 fp=0xc001de57d8 sp=0xc001de57b0 pc=0x4f4654
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF.func6(0xc00007ca08?, {0xc001feac00?, 0x0?, 0x0?}, {0xe4d590?, 0xc001de5878?})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x67 fp=0xc001de5828 sp=0xc001de57d8 pc=0x50b267
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0x56f35b?, {0xc000e4d590, 0x30, 0x30}, {0x0, 0x0, 0x0})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x19e fp=0xc001de5888 sp=0xc001de5828 pc=0x50b0de
crypto/internal/backend.ExtractHKDF(...)
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x1801580?, {0x0?, 0xf70a80?, 0xc000580008?}, {0x0?, 0x0?, 0x0?})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/key_schedule.go:95 +0x146 fp=0xc001de5908 sp=0xc001de5888 pc=0x64d566
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc001de5c50)
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:514 +0x2ce fp=0xc001de5a38 sp=0xc001de5908 pc=0x63154e
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc001de5c50)
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:132 +0x725 fp=0xc001de5ae0 sp=0xc001de5a38 pc=0x62f345
crypto/tls.(*Conn).clientHandshake(0xc00124b888, {0x10bbab0, 0xc001aeccd0})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/handshake_client.go:375 +0x845 fp=0xc001de5d30 sp=0xc001de5ae0 pc=0x6290a5
crypto/tls.(*Conn).clientHandshake-fm({0x10bbab0?, 0xc001aeccd0?})
:1 +0x33 fp=0xc001de5d58 sp=0xc001de5d30 pc=0x653af3
crypto/tls.(*Conn).handshakeContext(0xc00124b888, {0x10bbab0, 0xc000a53680})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/conn.go:1568 +0x3a6 fp=0xc001de5f70 sp=0xc001de5d58 pc=0x623fc6
crypto/tls.(*Conn).HandshakeContext(...)
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/crypto/tls/conn.go:1508
net/http.(*persistConn).addTLS.func2()
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/net/http/transport.go:1651 +0x6e fp=0xc001de5fe0 sp=0xc001de5f70 pc=0x71526e
runtime.goexit({})
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc001de5fe8 sp=0xc001de5fe0 pc=0x47cc81
created by net/http.(*persistConn).addTLS in goroutine 2336370
/net/code/extsrc/go/ms1.23.4-1/linuxR_x86/src/net/http/transport.go:1647 +0x309

We also saw a very similar (the same?) panic on an earlier version, here is that stack:

SIGSEGV: segmentation violation
PC=0x100000000 m=3 sigcode=1 addr=0x100000000
signal arrived during cgo execution

goroutine 56467648 gp=0xc00360d880 m=3 mp=0xc000077008 [syscall]:
runtime.cgocall(0xc1d5c0, 0xc00179d858)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/cgocall.go:157 +0x4b fp=0xc00179d830 sp=0xc00179d7f8 pc=0x41b7cb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_PKEY_derive(0x7fd48400c920, 0xc000136120, 0xc0005cc008)
_cgo_gotypes.go:1539 +0x4b fp=0xc00179d858 sp=0xc00179d830 pc=0x5a986b
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF.func6(0xc0007180f0?, {0xc000136120?, 0x30?, 0x30?}, 0xc0005cc008)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x67 fp=0xc00179d898 sp=0xc00179d858 pc=0x5bf8e7
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0xc000a4a090?, {0xc000a4a000, 0x30, 0x30}, {0xc000a4a0c0, 0x30, 0x30})
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x1a9 fp=0xc00179d8f0 sp=0xc00179d898 pc=0x5bf749
crypto/internal/backend.ExtractHKDF(...)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x19e71c0?, {0xc000a4a000?, 0x30?, 0x30?}, {0xc000a4a0c0?, 0x7?, 0x0?})
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/key_schedule.go:93 +0x145 fp=0xc00179d970 sp=0xc00179d8f0 pc=0x68dba5
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc00179dbd0)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:387 +0x138 fp=0xc00179dac0 sp=0xc00179d970 pc=0x6741f8
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc00179dbd0)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:86 +0x274 fp=0xc00179db00 sp=0xc00179dac0 pc=0x6727f4
crypto/tls.(*Conn).clientHandshake(0xc0012bc708, {0x12e9c30, 0xc000180230})
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client.go:265 +0x594 fp=0xc00179dd30 sp=0xc00179db00 pc=0x66d034
crypto/tls.(*Conn).clientHandshake-fm({0x12e9c30?, 0xc000180230?})
:1 +0x33 fp=0xc00179dd58 sp=0xc00179dd30 pc=0x693633
crypto/tls.(*Conn).handshakeContext(0xc0012bc708, {0x12e9bf8, 0xc001d00b10})
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1553 +0x3cb fp=0xc00179df70 sp=0xc00179dd58 pc=0x66aa6b
crypto/tls.(*Conn).HandshakeContext(...)
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1493
net/http.(*persistConn).addTLS.func2()
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1573 +0x6e fp=0xc00179dfe0 sp=0xc00179df70 pc=0x6f32ce
runtime.goexit({})
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00179dfe8 sp=0xc00179dfe0 pc=0x486be1
created by net/http.(*persistConn).addTLS in goroutine 56469757
/net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1569 +0x309

[dagood]

This looks like the same issue as microsoft/go#1319. Any luck finding a reproducer? Has any of the other info changed? (Happens occasionally, OpenSSL 1.1, SLES 15.5?)

[jmgvfr04]

good point, yes it is pretty much the same, except the MS go version is newer, have not been able to find a reproducer

[jmgvfr04]

I do see this note in the release notes for SLES 15 SP6

5.7.3 OpenSSL 3.1.4 is now default #
In SLES 15 SP6, OpenSSL has been updated to version 3.1.4, replacing OpenSSL 1.1.1.

Because the development packages of different versions are mutually exclusive and automatic conflict resolution is not performed during updates, libopenssl1_1-devel should be manually selected for de-installation.

I will ask my QE to upgrade the test system from SP5 to SP6 to see if the problem can be reproduced there or if it goes away.

[jmgvfr04]

We have not seen this panic after upgrading to SLES 15 SP6 (see above for openssl upgrade in SP6) so I think we can consider this issue resolved with openssl 3.1.4 on SLES