Silo Metadata and Placement Filtering

[rkargMsft]

This PR has two main features:

1. Placement Filter support: Adds ability to annotate grains with filters that can examine and manipulate the set of candidate silos that are passed to the Placement implementation
2. Silo Metadata: Ability to add (string, string) values to a silo that all silos in the cluster have access to. This aims to address [Proposal] Support defining Silo metadata #8189

Additionally, there are specific implementations of Placement Filtering using the Silo Metadata to specify either required keys that must match, or an ordered set of preferred keys to use for filtering down placement candidates.

• Add tests
• Create documentation
  • Silo Metadata
  • Placement Filtering
  • Matching Metadata Placement Filtering

Microsoft Reviewers: Open in CodeFlow

[rkargMsft]

Question: Why is metadata handled separate from the existing membership table as suggested in #8189?
Answer: There is no existing general storage on MembershipEntry and adding a new field would require every Clustering provider (in core, Contrib, and any non-public implementations) to update implementations to store and retrieve a new field for metadata. There isn't a good way to advertise optional features on a clustering provider to validate at startup so we'd be left with either runtime exceptions or with unexpected behavior (silo metadata would get configured but not actually get stored and be available for other silos to read).
Because of that, a separate mechanism is used to transport metadata (GrainServices). This could technically be made a bit more efficient by piggy backing on the existing membership gossip exchange, but given that it's a one-time pull per silo, it's quite likely that the current performance will work for all practical use cases. If there is a scale where even this once-per-silo pull is unacceptable, the current implementation can adjust its internals to us gossip without needing to change the interface exposed to consumers.

[rkargMsft]

Question: Why is this implemented in core instead of as a separate package?
Answer: There are several internal types that would be required to be public in order to make this work as a separate package so that approach was abandoned.
Additionally, having access to internal classes like MembershipTableManager makes the process of keeping the local cache of Silo Metadata more efficient since there isn't just blind polling of silo membership as would be required with a separate package.

[miguelhasse]

@rkargMsft this is something I really need. When can we expect this PR to move from draft? Thanks so much.

[rkargMsft]

When I’m back next week I’ll be working on the documentation to get this out of draft. I expect that others will be more available for feedback on the PR in the new year, too.

[rkargMsft]

Ordering of filters (through a parameter on the attribute) came up as being necessary as the documentation was being written. The order the attributes are applied in source is not necessarily the same order that they show up when querying metadata so explicit ordering is necessary.

Getting the ordering support implemented atm.

[miguelhasse]

@rkargMsft I would suggest moving the following into assembly Orleans.Core.Abstractions:

• PlacementFilterAttribute
• PlacementFilterStrategy

And eventually move specific implementations for PreferredMatchSiloMetadataPlacementFilter and RequiredMatchSiloMetadataPlacementFilter into a separate assembly (eg: Orleans.Placement.Filtering).

[ReubenBond]

@rkargMsft I would suggest moving the following into assembly Orleans.Core.Abstractions:

• PlacementFilterAttribute
• PlacementFilterStrategy

I agree with this comment

And eventually move specific implementations for PreferredMatchSiloMetadataPlacementFilter and RequiredMatchSiloMetadataPlacementFilter into a separate assembly (eg: Orleans.Placement.Filtering).

We can consider this, but I'm not in favor of fine-grained assembly/package model since it tends to degrade the user experience

[rkargMsft]

Documentation PR up. Ready for review.

[rkargMsft]

Looks like all the outstanding feedback has been addressed. Is there any further review on this before considering it for merging?

[dmorganMsft]

Any chance we can get this reviewed and released soon? This is the last piece my project is waiting on to be able to start using Orleans 🥳

[miguelhasse]

Any chance we can get this reviewed and released soon? This is the last piece my project is waiting on to be able to start using Orleans 🥳

@rkargMsft I'm also waiting for this to be released in order to be able to propose Orleans adoption based on a PoC that requires silo metadata information. Is there an expected release date?

[alimozdemir]

Hello there, great work thank you for that. Once it gets merged, what is the approximate release date? Is it going to be part of .NET 8/9 as well?

[Copilot]

Copilot reviewed 16 out of 31 changed files in this pull request and generated 2 comments.

Files not reviewed (15)

• src/Orleans.Runtime/MembershipService/SiloMetadata/ISiloMetadataGrainService.cs: Evaluated as low risk
• src/Orleans.Runtime/Placement/Filtering/PlacementFilterDirectorResolver.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadataHostingExtensions.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadataGrainService.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadataClient.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadata.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadaCache.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/ISiloMetadataClient.cs: Evaluated as low risk
• src/Orleans.Runtime/Hosting/DefaultSiloServices.cs: Evaluated as low risk
• src/Orleans.Core.Abstractions/Manifest/GrainProperties.cs: Evaluated as low risk
• src/Orleans.Runtime/MembershipService/SiloMetadata/ISiloMetadataCache.cs: Evaluated as low risk
• src/Orleans.Core/Placement/IPlacementFilterDirector.cs: Evaluated as low risk
• src/Orleans.Core.Abstractions/Placement/PlacementFilterStrategy.cs: Evaluated as low risk
• src/Orleans.Runtime/Placement/Filtering/RequiredMatchSiloMetadataPlacementFilterAttribute.cs: Evaluated as low risk
• src/Orleans.Runtime/Placement/Filtering/PreferredMatchSiloMetadataPlacementFilterAttribute.cs: Evaluated as low risk

Comments suppressed due to low confidence (1)

src/Orleans.Runtime/Placement/Filtering/PreferredMatchSiloMetadataPlacementFilterStrategy.cs:24

• The error message 'Invalid ordered-metadata-keys property value.' is unclear. It should be more descriptive, such as 'The ordered-metadata-keys property value is invalid. Ensure it is a comma-separated string of metadata keys.'

throw new ArgumentException("Invalid ordered-metadata-keys property value.");

[ReubenBond]

@rkargMsft thank you for your patience. This looks good to me. At some point in the future, we should consider designs for caching the result of compatibility checks + user-defined filters so that we do not have to run this process every time a grain is placed.

I rebased on main and pushed some suggestions in a commit. Please take a look and if they look good to you then I'll give it one final pass and merge.

[rkargMsft]

Tell me more about the per-placement work? I thought that the Resolver only ran once per grain type and should be relatively trivial cost if no filters are defined.

[ReubenBond]

@rkargMsft PlacementService.GetCompatibleSilos is not currently cached, so it runs on every placement call.

[ReubenBond]

PlacementTarget is currently being passed to IPlacementFilterDirector.Filter and it has RequestContextData and GrainId properties which means that the filter might depend on the either of those, making the whole thing un-cacheable. Maybe we need to create a new type or pass the values (GrainType, GrainInterfaceType, GrainVersion) directly to the filter. Probably the former.

EDIT: so that we don't paint ourselves into a corner, I'll create a new type and update the interface to use it:

public readonly record struct PlacementFilterContext(GrainType GrainType, GrainInterfaceType InterfaceType, ushort InterfaceVersion);

[ReubenBond]

Ok, added. I converted the tests to use class fixtures to make them run faster

[rkargMsft]

This previous code was copied from DistributedGrainDirectory. Should there be a follow up issue to update usages of the prior pattern with this one?

[ReubenBond]

Yes, we should clean up the other instances.

[rkargMsft]

Created #9360

[rkargMsft]

@rkargMsft PlacementService.GetCompatibleSilos is not currently cached, so it runs on every placement call.

Added #9361

[dmorganMsft]

Thank you @ReubenBond and @rkargMsft 🥳 !