fatal error: fault [signal SIGBUS: bus error code=0x2 addr=0x7f3a457f3008 pc=0x9e2b65] #16302
Replies: 9 comments
-
@Tejaswini5327 Is it possible to provide the db file consumed by failed pod (pod-2 in your case)? Only one out of the three etcd pods keeps restarting? Other two PODs are running well? Please also provide the pod-2's log. |
Beta Was this translation helpful? Give feedback.
-
Hi @ahrtr , We can't provide the db file of [pod-2.] Yes only one out of 3 pods is continuously restarting, pod-0, pod-1 are running, pod-2 restarting. Attaching the pod-2 logs here: |
Beta Was this translation helpful? Give feedback.
-
Most likely the db file consumed by pod-2 is somehow corrupted. Please double check it using the bbolt,
Please also feel free to ping me on slack in K8s slack space to discuss this issue offline. |
Beta Was this translation helpful? Give feedback.
-
@ahrtr, |
Beta Was this translation helpful? Give feedback.
-
Please execute the command I suggested to double confirm the corruption firstly. What operation you did before the failure ? Do you have a reproduce steps? And then follow Replace a failed machine |
Beta Was this translation helpful? Give feedback.
-
Again, please ping me on slack (search "Benjamin Wang" in K8s slack workspace) if you want, and I am happy to take a closer look at the db file together with you. |
Beta Was this translation helpful? Give feedback.
-
@ahrtr with fresh installation of deployment, error happened with pod-2. No other operation is performed. Very Sorry we cant share the db file anywhere due to security constartints. |
Beta Was this translation helpful? Give feedback.
-
Please try to run the
|
Beta Was this translation helpful? Give feedback.
-
Thankkyou @ahrtr , will try to check the suggested |
Beta Was this translation helpful? Give feedback.
-
Bug report criteria
What happened?
We have a 3 pod service, in that pod-2 is restarting with exit code 143.
The following error is seen in pod-2 logs:
unexpected fault address 0x7f3a457f3008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f3a457f3008 pc=0x9e2b65]
goroutine 1 [running]:
runtime.throw(0x121d104, 0x5)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/runtime/panic.go:1117 +0x72 fp=0xc0003164a8 sp=0xc000316478 pc=0x4385b2
runtime.sigpanic()
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/runtime/signal_unix.go:731 +0x2c8 fp=0xc0003164e0 sp=0xc0003164a8 pc=0x44fe28
go.etcd.io/bbolt.(*Cursor).search(0xc000316680, 0x1a310c0, 0x4, 0x4, 0x3)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.etcd.io/[email protected]/cursor.go:249 +0x65 fp=0xc000316578 sp=0xc0003164e0 pc=0x9e2b65
go.etcd.io/bbolt.(*Cursor).seek(0xc000316680, 0x1a310c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.etcd.io/[email protected]/cursor.go:159 +0x7d fp=0xc0003165e8 sp=0xc000316578 pc=0x9e23fd
go.etcd.io/bbolt.(*Bucket).CreateBucket(0xc0004a8718, 0x1a310c0, 0x4, 0x4, 0xc1216da2b191a84b, 0x1000000010000, 0xc0002aff10)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.etcd.io/[email protected]/bucket.go:167 +0xf6 fp=0xc000316750 sp=0xc0003165e8 pc=0x9de8f6
go.etcd.io/bbolt.(*Tx).CreateBucket(...)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.etcd.io/[email protected]/tx.go:108
go.etcd.io/etcd/server/v3/mvcc/backend.(*batchTx).UnsafeCreateBucket(0xc0002fe240, 0x13bb100, 0x1a93280)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/batch_tx.go:116 +0x67 fp=0xc000316938 sp=0xc000316750 pc=0xb92bc7
go.etcd.io/etcd/server/v3/etcdserver/cindex.CreateMetaBucket(0x13c9d08, 0xc0002fe240)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/cindex/cindex.go:176 +0x82 fp=0xc000316978 sp=0xc000316938 pc=0xc4eea2
go.etcd.io/etcd/server/v3/etcdserver.NewServer(0xc0000679ea, 0x26, 0x0, 0x0, 0x0, 0x0, 0xc0004e81b0, 0x1, 0x1, 0xc000298510, ...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/server.go:385 +0x565 fp=0xc000317fb0 sp=0xc000316978 pc=0xd33fa5
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc000241500, 0xc00035a000, 0x0, 0x0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/embed/etcd.go:243 +0xef8 fp=0xc00031a6c0 sp=0xc000317fb0 pc=0xf3c958
go.etcd.io/etcd/server/v3/etcdmain.startEtcd(0xc000241500, 0x121e47b, 0x6, 0xc0002e7301, 0x2)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdmain/etcd.go:228 +0x32 fp=0xc00031a740 sp=0xc00031a6c0 pc=0xf9ceb2
go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2(0xc00021e1d0, 0x1, 0x1)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdmain/etcd.go:123 +0x24da fp=0xc00031df00 sp=0xc00031a740 pc=0xf9be3a
go.etcd.io/etcd/server/v3/etcdmain.Main(0xc00021e1d0, 0x1, 0x1)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdmain/main.go:40 +0x13f fp=0xc00031df60 sp=0xc00031df00 pc=0xfa703f
main.main()
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/main.go:32 +0x45 fp=0xc00031df88 sp=0xc00031df60 pc=0xfaac05
runtime.main()
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/runtime/proc.go:225 +0x256 fp=0xc00031dfe0 sp=0xc00031df88 pc=0x43adf6
runtime.goexit()
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00031dfe8 sp=0xc00031dfe0 pc=0x4722a1
goroutine 219 [IO wait]:
internal/poll.runtime_pollWait(0x7f3a65cb9698, 0x72, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/runtime/netpoll.go:227 +0x55
internal/poll.(*pollDesc).wait(0xc0002e7498, 0x72, 0x0, 0x0, 0x121f214)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0002e7480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc0002e7480, 0x8, 0xc000a86e08, 0x40dbbb)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000694420, 0x203000, 0x7f3a658b3b00, 0x10)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000694420, 0xc000552190, 0x10, 0x10, 0x10e1dc0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/net/tcpsock.go:261 +0x65
go.etcd.io/etcd/client/pkg/v3/transport.(*keepaliveListener).Accept(0xc0006964d0, 0x10, 0x1120960, 0xc000a86ea0, 0xc000a86ef8)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/keepalive_listener.go:52 +0x37
go.etcd.io/etcd/client/pkg/v3/transport.(*rwTimeoutListener).Accept(0xc00033bb00, 0xc000172c00, 0x0, 0xc000a86ef8, 0x40e318)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/timeout_listener.go:36 +0x37
crypto/certificate.(*listener).Accept(0xc000694090, 0xc0005521b0, 0xc000552170, 0xc00000a0f0, 0xc00000a100)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:67 +0x37
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop(0xc00055a000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:126 +0x1d3
created by go.etcd.io/etcd/client/pkg/v3/transport.newcertificateListener
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:84 +0x287
goroutine 108 [runnable]:
math/big.nat.divBasic(0xc000aa6000, 0x9, 0xd, 0xc000c2f4a0, 0x12, 0x16, 0xc000a43420, 0x9, 0xe)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:745 +0x66c
math/big.nat.divLarge(0x0, 0x0, 0x0, 0xc000c2f4a0, 0x11, 0x16, 0xc000c2f4a0, 0x11, 0x16, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:725 +0x29e
math/big.nat.div(0x0, 0x0, 0x0, 0xc000c2f4a0, 0x11, 0x16, 0xc000c2f4a0, 0x11, 0x16, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:672 +0x40d
math/big.(*Int).QuoRem(0xc000aa0f68, 0xc000aa10d8, 0xc00033bf00, 0xc000aa10d8, 0xc000aa0f68, 0xc000cd4920)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:239 +0xbf
math/big.(*Int).Mod(0xc000aa10d8, 0xc000aa10d8, 0xc00033bf00, 0xc000aa10d8)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:270 +0x113
crypto/elliptic.(*CurveParams).doubleJacobian(0xc00055a640, 0xc000cd4940, 0xc000cd4920, 0xc000cd4960, 0xc000cd4940, 0xc000cd4920, 0xc000cd4960)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:209 +0x9f
crypto/elliptic.(*CurveParams).ScalarMult(0xc00055a640, 0xc00033bfc0, 0xc00037a000, 0xc0004a01e0, 0x42, 0x42, 0x1, 0xc0004a01e0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:271 +0x15d
crypto/elliptic.(*CurveParams).ScalarBaseMult(0xc00055a640, 0xc0004a01e0, 0x42, 0x42, 0x42, 0x42)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:283 +0x5b
crypto/x509.parseECPrivateKey(0x0, 0xc000e84000, 0xdf, 0xe4, 0x0, 0x139e280, 0xc0004ee080)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:117 +0x3be
crypto/x509.ParseECPrivateKey(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:35
crypto/certificate.parsePrivateKey(0xc000e84000, 0xdf, 0xe4, 0xc0004d8000, 0x0, 0x0, 0x93)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:382 +0x179
crypto/certificate.X509KeyPair(0xc0004102d5, 0x0, 0x1, 0xc00029016d, 0x0, 0x93, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:330 +0x5d5
go.etcd.io/etcd/client/pkg/v3/certificateutil.NewCert(0xc000580420, 0x1c, 0xc000580440, 0x1b, 0x127bfe8, 0x7f3a65e81c00, 0x7f3a6604ffff, 0xc000a82708)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/certificateutil/certificateutil.go:68 +0x1ad
go.etcd.io/etcd/client/pkg/v3/transport.certificateInfo.baseConfig.func4(0xc00063a000, 0x40e318, 0xb8, 0x11b58c0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener.go:422 +0x6e
crypto/certificate.(*Config).getCertificate(0xc000742c00, 0xc00063a000, 0xc00063a000, 0x20, 0x20)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/common.go:1017 +0x3cb
crypto/certificate.(*serverHandshakeState).processClientHello(0xc000aa1ca0, 0x0, 0xc000613498)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:223 +0x2c8
crypto/certificate.(*serverHandshakeState).handshake(0xc000aa1ca0, 0xc000634000, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:64 +0x37
crypto/certificate.(*Conn).serverHandshake(0xc00023ee00, 0xc0006136a0, 0x7bfcd7)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:58 +0x14b
crypto/certificate.(*Conn).Handshake(0xc00023ee00, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/conn.go:1391 +0xc9
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop.func2(0xc000552040, 0xc00000a0f0, 0xc00000a100, 0xc000302a50, 0xc00055a000, 0x13ba6f0, 0xc0001c8000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:146 +0xa5
created by go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:137 +0x193
goroutine 109 [runnable]:
math/big.nat.shl(0x0, 0x0, 0x0, 0xc000334630, 0x11, 0x15, 0x3, 0xc000334580, 0x11, 0x16)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:993 +0x4bd
math/big.(*Int).Lsh(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:1009
crypto/elliptic.(*CurveParams).doubleJacobian(0xc00055a640, 0xc0007ac240, 0xc0007ac260, 0xc0007ac280, 0xc0007ac200, 0xc0007ac1e0, 0xc0007ac220)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:225 +0x396
crypto/elliptic.(*CurveParams).ScalarMult(0xc00055a640, 0xc00033bfc0, 0xc00037a000, 0xc0005b8320, 0x42, 0x42, 0x1, 0xc0005b8320)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:271 +0x15d
crypto/elliptic.(*CurveParams).ScalarBaseMult(0xc00055a640, 0xc0005b8320, 0x42, 0x42, 0x42, 0x42)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:283 +0x5b
crypto/x509.parseECPrivateKey(0x0, 0xc00051c000, 0xdf, 0xe4, 0x0, 0x139e280, 0xc0004d2a50)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:117 +0x3be
crypto/x509.ParseECPrivateKey(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:35
crypto/certificate.parsePrivateKey(0xc00051c000, 0xdf, 0xe4, 0xc0002d3b80, 0x0, 0x0, 0x93)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:382 +0x179
crypto/certificate.X509KeyPair(0xc0006da5d5, 0x0, 0x1, 0xc000054f6d, 0x0, 0x93, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:330 +0x5d5
go.etcd.io/etcd/client/pkg/v3/certificateutil.NewCert(0xc000580420, 0x1c, 0xc000580440, 0x1b, 0x127bfe8, 0x0, 0x0, 0x0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/certificateutil/certificateutil.go:68 +0x1ad
go.etcd.io/etcd/client/pkg/v3/transport.certificateInfo.baseConfig.func4(0xc00083a0c0, 0x40e318, 0xb8, 0x11b58c0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener.go:422 +0x6e
crypto/certificate.(*Config).getCertificate(0xc000742c00, 0xc00083a0c0, 0xc00083a0c0, 0x20, 0x20)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/common.go:1017 +0x3cb
crypto/certificate.(*serverHandshakeState).processClientHello(0xc000525ca0, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:223 +0x2c8
crypto/certificate.(*serverHandshakeState).handshake(0xc000525ca0, 0xc0002ea600, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:64 +0x37
crypto/certificate.(*Conn).serverHandshake(0xc00023f180, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:58 +0x14b
crypto/certificate.(*Conn).Handshake(0xc00023f180, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/conn.go:1391 +0xc9
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop.func2(0xc000552060, 0xc00000a0f0, 0xc00000a100, 0xc000302a50, 0xc00055a000, 0x13ba6f0, 0xc0001c8000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:146 +0xa5
created by go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:137 +0x193
goroutine 110 [runnable]:
math/big.basicMul(0xc000bf1ad0, 0x12, 0x16, 0xc000bf1760, 0x9, 0x16, 0xc000bf1760, 0x9, 0x16)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:195 +0x17f
math/big.nat.sqr(0xc000bf1ad0, 0x12, 0x16, 0xc000bf1760, 0x9, 0x16, 0xc000bf18c0, 0x11, 0x15)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:578 +0x7e5
math/big.(*Int).Mul(0xc000aa50b8, 0xc000aa50b8, 0xc000aa50b8, 0xc00052ef40)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:164 +0x76
crypto/elliptic.(*CurveParams).doubleJacobian(0xc00055a640, 0xc00052eee0, 0xc00052ef00, 0xc00052ef20, 0xc00052eea0, 0xc00052ee80, 0xc00052eec0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:252 +0x63d
crypto/elliptic.(*CurveParams).ScalarMult(0xc00055a640, 0xc00033bfc0, 0xc00037a000, 0xc00022e640, 0x42, 0x42, 0x1, 0xc00022e640)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:271 +0x15d
crypto/elliptic.(*CurveParams).ScalarBaseMult(0xc00055a640, 0xc00022e640, 0x42, 0x42, 0x42, 0x42)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:283 +0x5b
crypto/x509.parseECPrivateKey(0x0, 0xc000a9c000, 0xdf, 0xe4, 0x0, 0x139e280, 0xc000696550)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:117 +0x3be
crypto/x509.ParseECPrivateKey(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:35
crypto/certificate.parsePrivateKey(0xc000a9c000, 0xdf, 0xe4, 0xc000cc8000, 0x0, 0x0, 0x93)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:382 +0x179
crypto/certificate.X509KeyPair(0xc000a9a5d5, 0x0, 0x1, 0xc0004ba36d, 0x0, 0x93, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:330 +0x5d5
go.etcd.io/etcd/client/pkg/v3/certificateutil.NewCert(0xc000580420, 0x1c, 0xc000580440, 0x1b, 0x127bfe8, 0x5, 0xc0004c56d8, 0x44cbbf)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/certificateutil/certificateutil.go:68 +0x1ad
go.etcd.io/etcd/client/pkg/v3/transport.certificateInfo.baseConfig.func4(0xc000562180, 0x40e318, 0xb8, 0x13844c4)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener.go:422 +0x6e
crypto/certificate.(*Config).getCertificate(0xc000742c00, 0xc000562180, 0xc000562180, 0x20, 0x20)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/common.go:1017 +0x3cb
crypto/certificate.(*serverHandshakeState).processClientHello(0xc000aa5ca0, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:223 +0x2c8
crypto/certificate.(*serverHandshakeState).handshake(0xc000aa5ca0, 0xc0006ec000, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:64 +0x37
crypto/certificate.(*Conn).serverHandshake(0xc00023f880, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:58 +0x14b
crypto/certificate.(*Conn).Handshake(0xc00023f880, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/conn.go:1391 +0xc9
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop.func2(0xc0005520d0, 0xc00000a0f0, 0xc00000a100, 0xc000302a50, 0xc00055a000, 0x13ba6f0, 0xc0001c8000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:146 +0xa5
created by go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:137 +0x193
goroutine 111 [runnable]:
sync.(*Pool).Put(0x1aae860, 0x1209780, 0xc001088120)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/sync/pool.go:90 +0xbe
math/big.putNat(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:691
math/big.nat.divBasic(0xc000f044d0, 0x9, 0xd, 0xc00032c4d0, 0x12, 0x16, 0xc00054d8f0, 0x9, 0xe)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:808 +0x5bf
math/big.nat.divLarge(0x0, 0x0, 0x0, 0xc00032c4d0, 0x11, 0x16, 0xc00032c4d0, 0x11, 0x16, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:725 +0x29e
math/big.nat.div(0x0, 0x0, 0x0, 0xc00032c4d0, 0x11, 0x16, 0xc00032c4d0, 0x11, 0x16, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:672 +0x40d
math/big.(*Int).QuoRem(0xc000a30f68, 0xc000a8a700, 0xc00033bf00, 0xc000a8a700, 0xc000a30f68, 0xc000a8a6e0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:239 +0xbf
math/big.(*Int).Mod(0xc000a8a700, 0xc000a8a700, 0xc00033bf00, 0xc000a8a700)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:270 +0x113
crypto/elliptic.(*CurveParams).doubleJacobian(0xc00055a640, 0xc000a8a680, 0xc000a8a660, 0xc000a8a6a0, 0xc000a8a680, 0xc000a8a660, 0xc000a8a6a0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:243 +0x550
crypto/elliptic.(*CurveParams).ScalarMult(0xc00055a640, 0xc00033bfc0, 0xc00037a000, 0xc000a361e0, 0x42, 0x42, 0x1, 0xc000a361e0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:271 +0x15d
crypto/elliptic.(*CurveParams).ScalarBaseMult(0xc00055a640, 0xc000a361e0, 0x42, 0x42, 0x42, 0x42)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:283 +0x5b
crypto/x509.parseECPrivateKey(0x0, 0xc000a26000, 0xdf, 0xe4, 0x0, 0x139e280, 0xc000a18150)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:117 +0x3be
crypto/x509.ParseECPrivateKey(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:35
crypto/certificate.parsePrivateKey(0xc000a26000, 0xdf, 0xe4, 0xc000a3e000, 0x0, 0x0, 0x93)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:382 +0x179
crypto/certificate.X509KeyPair(0xc000e1c2d5, 0x0, 0x1, 0xc000e1e16d, 0x0, 0x93, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:330 +0x5d5
go.etcd.io/etcd/client/pkg/v3/certificateutil.NewCert(0xc000580420, 0x1c, 0xc000580440, 0x1b, 0x127bfe8, 0x7f3a65ec0900, 0x7f3a6604ffff, 0xc000d7f708)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/certificateutil/certificateutil.go:68 +0x1ad
go.etcd.io/etcd/client/pkg/v3/transport.certificateInfo.baseConfig.func4(0xc000e14000, 0x40e318, 0xb8, 0x13844c4)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener.go:422 +0x6e
crypto/certificate.(*Config).getCertificate(0xc000742c00, 0xc000e14000, 0xc000e14000, 0x20, 0x20)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/common.go:1017 +0x3cb
crypto/certificate.(*serverHandshakeState).processClientHello(0xc000a31ca0, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:223 +0x2c8
crypto/certificate.(*serverHandshakeState).handshake(0xc000a31ca0, 0xc000e08000, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:64 +0x37
crypto/certificate.(*Conn).serverHandshake(0xc0005aaa80, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:58 +0x14b
crypto/certificate.(*Conn).Handshake(0xc0005aaa80, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/conn.go:1391 +0xc9
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop.func2(0xc000552110, 0xc00000a0f0, 0xc00000a100, 0xc000302a50, 0xc00055a000, 0x13ba6f0, 0xc0001c8000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:146 +0xa5
created by go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:137 +0x193
goroutine 112 [runnable]:
sync.(*Pool).Put(0x1aae860, 0x1209780, 0xc001088108)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/sync/pool.go:90 +0xbe
math/big.putNat(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:691
math/big.nat.divBasic(0xc000876c30, 0x19, 0x1d, 0xc000d81b80, 0x22, 0x26, 0xc000cb1490, 0x9, 0xe)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:808 +0x5bf
math/big.nat.divLarge(0x0, 0x0, 0x0, 0xc000d81b80, 0x21, 0x26, 0xc000d81b80, 0x21, 0x26, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:725 +0x29e
math/big.nat.div(0x0, 0x0, 0x0, 0xc000d81b80, 0x21, 0x26, 0xc000d81b80, 0x21, 0x26, 0xc0002144e0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/nat.go:672 +0x40d
math/big.(*Int).QuoRem(0xc000520f68, 0xc00063e820, 0xc00033bf00, 0xc00063e820, 0xc000520f68, 0xc000521078)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:239 +0xbf
math/big.(*Int).Mod(0xc00063e820, 0xc00063e820, 0xc00033bf00, 0xc00063e820)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/math/big/int.go:270 +0x113
crypto/elliptic.(*CurveParams).doubleJacobian(0xc00055a640, 0xc00063e7c0, 0xc00063e7a0, 0xc00063e7e0, 0xc00063e7c0, 0xc00063e7a0, 0xc00063e7e0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:231 +0x455
crypto/elliptic.(*CurveParams).ScalarMult(0xc00055a640, 0xc00033bfc0, 0xc00037a000, 0xc0010841e0, 0x42, 0x42, 0x1, 0xc0010841e0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:271 +0x15d
crypto/elliptic.(*CurveParams).ScalarBaseMult(0xc00055a640, 0xc0010841e0, 0x42, 0x42, 0x42, 0x42)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/elliptic/elliptic.go:283 +0x5b
crypto/x509.parseECPrivateKey(0x0, 0xc000f00000, 0xdf, 0xe4, 0x0, 0x139e280, 0xc00109e080)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:117 +0x3be
crypto/x509.ParseECPrivateKey(...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/x509/sec1.go:35
crypto/certificate.parsePrivateKey(0xc000f00000, 0xdf, 0xe4, 0xc001090000, 0x0, 0x0, 0x93)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:382 +0x179
crypto/certificate.X509KeyPair(0xc000a9a2d5, 0x0, 0x1, 0xc0004ba16d, 0x0, 0x93, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/certificate.go:330 +0x5d5
go.etcd.io/etcd/client/pkg/v3/certificateutil.NewCert(0xc000580420, 0x1c, 0xc000580440, 0x1b, 0x127bfe8, 0x7f3a65ea4d00, 0x7f3a6604ffff, 0xc000a87708)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/certificateutil/certificateutil.go:68 +0x1ad
go.etcd.io/etcd/client/pkg/v3/transport.certificateInfo.baseConfig.func4(0xc0005620c0, 0x40e318, 0xb8, 0x11b58c0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener.go:422 +0x6e
crypto/certificate.(*Config).getCertificate(0xc000742c00, 0xc0005620c0, 0xc0005620c0, 0x20, 0x20)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/common.go:1017 +0x3cb
crypto/certificate.(*serverHandshakeState).processClientHello(0xc000521ca0, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:223 +0x2c8
crypto/certificate.(*serverHandshakeState).handshake(0xc000521ca0, 0xc0005fe200, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:64 +0x37
crypto/certificate.(*Conn).serverHandshake(0xc0005aae00, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/handshake_server.go:58 +0x14b
crypto/certificate.(*Conn).Handshake(0xc0005aae00, 0x0, 0x0)
/usr/local/google/home/siarkowicz/.gvm/gos/go1.16.15/src/crypto/certificate/conn.go:1391 +0xc9
go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop.func2(0xc000552170, 0xc00000a0f0, 0xc00000a100, 0xc000302a50, 0xc00055a000, 0x13ba6f0, 0xc0001c8000)
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:146 +0xa5
created by go.etcd.io/etcd/client/pkg/v3/transport.(*certificateListener).acceptLoop
/tmp/etcd-release-3.5.5/etcd/release/etcd/client/pkg/transport/listener_certificate.go:137 +0x193
goroutine 245 [select]:
go.etcd.io/etcd/server/v3/mvcc/backend.(*backend).run(0xc000562240)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:421 +0x11e
created by go.etcd.io/etcd/server/v3/mvcc/backend.newBackend
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:229 +0x5e5
What did you expect to happen?
restart of pod-2 should not happen.
How can we reproduce it (as minimally and precisely as possible)?
After installing, only pod-2 started restarting and po-1, pod-1 are running.
Anything else we need to know?
No response
Etcd version (please run commands below)
bash-4.4$ etcd --version
etcd Version: 3.5.5
Git SHA: 19002cf
Go Version: go1.16.15
Go OS/Arch: linux/amd64
bash-4.4$ etcdctl version
etcdctl version: 3.5.5
API version: 3.5
Etcd configuration (command line flags or environment variables)
paste your configuration here
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
Beta Was this translation helpful? Give feedback.
All reactions