[Bug] Should the SOCI snapshotter start after containerd in the example systemd unit file?

[ollypom]

Description

The systemd unit file in this repo starts the SOCI snapshotter before containerd.

soci-snapshotter/soci-snapshotter.service

Line 33 in bc17f08

Before=containerd.service

However if you use the containerd content store.

$  cat /etc/soci-snapshotter-grpc/config.toml 
...
[content_store]
type="containerd"
namespace="default"

And restart the server, the snapshotter fails to start

{
    "error": "cannot create content store: could not connect to containerd socket for content store access: failed to dial \"/run/containerd/containerd.sock\": context deadline exceeded: connection error: desc = \"transport: error while dialing: dial unix:///run/containerd/containerd.sock: timeout\"",
    "level": "fatal",
    "msg": "failed to configure filesystem",
    "time": "2023-11-10T11:39:48.945546470Z"
}

Steps to reproduce the bug

No response

Describe the results you expected

The soci snapshottter should start

Host information

1. OS: AL2
2. Snapshotter Version: 0.4.0
3. Containerd Version: 1.6.9

Any additional context or information about the bug

No response

[tuananh]

historically, stargz snapshotter doesn't have startup dependency on containerd service but since soci uses content_store from containerd by default(!?), we should change the service to be After ?

soci-snapshotter/soci/store/store.go

Lines 77 to 81 in e6dfa24

	func NewStoreConfig(opts ...Option) config.ContentStoreConfig {
	storeConfig := config.ContentStoreConfig{
	Type: config.DefaultContentStoreType,
	Namespace: namespaces.Default,
	}

[Kern--]

The containerd content store is not default and is still experimental.

The benefit to starting SOCI before containerd is that clients can rely on readiness of the containerd daemon to mean that the host is ready to pull images. If we switch the order, there is a risk that containerd is marked ready and an image pull is initiated before SOCI is ready.

I think we want to keep this order, but change SOCI to lazily instantiate the content store once containerd actually tries to use SOCI for an image pull. Would that work for you?