Does etcd-server read from disk when sending snapshots to followers

[Zhiying12]

Given a cluster of three nodes, let's say we start two of them at the beginning. After loading several GiB keys, we start the third one. Based on the network monitoring log, I can see that leader sends its db to the new peer as a snapshot. However, I also monitored disk I/O, and I couldn't see the leader node read anything from the disk.

Does the leader maintain the whole db in memory?

[fuweid]

Hi, did you check the page cache in the leader?

The leader reads the boltdb from disk and send it to follower. The boltdb is mmaped into memory.
If the file has page cache in kernel, I think there is not disk IO.

REF:

etcd/server/etcdserver/snapshot_merge.go

Line 38 in 60051be

dbsnap := s.be.Snapshot()

[Zhiying12]

Appreciate it! A follow-up question. Does etcd do regular snapshot for v3 store (boltdb)? I checked the code for snapshot, and it seems it only does snapshot for v2 store. How does a follower recover the state from the snapshot sent by the leader?

[ahrtr]

Does etcd do regular snapshot for v3 store (boltdb)?

No. The leader only sends a snapshot to a slow follower when it's far lag behind the leader's data.

How does a follower recover the state from the snapshot sent by the leader?

Pls read

etcd/server/etcdserver/server.go

Line 941 in 2763392

func (s *EtcdServer) applySnapshot(ep *etcdProgress, toApply *toApply) {

[Zhiying12]

Thanks a lot! IMHO, the follower will replace its backend db with the leader's snapshot below.

etcd/server/storage/backend.go

Lines 59 to 66 in 2763392

	func OpenSnapshotBackend(cfg config.ServerConfig, ss *snap.Snapshotter, snapshot raftpb.Snapshot, hooks *BackendHooks) (backend.Backend, error) {
	snapPath, err := ss.DBFilePath(snapshot.Metadata.Index)
	if err != nil {
	return nil, fmt.Errorf("failed to find database snapshot file (%v)", err)
	}
	if err := os.Rename(snapPath, cfg.BackendPath()); err != nil {
	return nil, fmt.Errorf("failed to rename database snapshot file (%v)", err)
	}

After updating the db, the follower restores the corresponding consistIndex from the snapshot. So far, the follower catches up all the progress up to the snapshot. After that, it just needs to apply entries after the consistIndex. Is my understanding correct?

etcd/server/etcdserver/server.go

Lines 989 to 990 in 2763392

	s.consistIndex.SetBackend(newbe)
	verifySnapshotIndex(toApply.snapshot, s.consistIndex.ConsistentIndex())

Also, in snapshot() below, the etcd server just takes snapshot of V2 store and trims outdated log entries. It would creates several v2 snapshot files with suffix ".snap" under the snap folder. As v2 store is deprecated, does it just keep track of metadata, or are these files still useful other than compatibility purpose?

etcd/server/etcdserver/server.go

Line 2066 in 2763392

func (s *EtcdServer) snapshot(snapi uint64, confState raftpb.ConfState) {

Thanks everyone for your time and patience again!

[fuweid]

After that, it just needs to apply entries after the consistIndex. Is my understanding correct?

I didn't follow your question. After restore the snapshot, the leader can sync the log with follower. And then the follower applies the log.

As v2 store is deprecated, does it just keep track of metadata, or are these files still useful other than compatibility purpose?

The deprecated work is still WIP. Hope the following doc can help you.

• [WIP]Migrate etcd storage file to markdown. #15754
• RFC: StoreV2 deprecation plan #12913

[Zhiying12]

Thank you so much for your answer and reference!