[Breaking changes] Beta update

docs/DataModel.md

@@ -1,6 +1,14 @@
 # Data Model
 
-This document briefly describes the data model for features/experiments used in Excos.
+## General feature model
+
+A **feature** has a name and a set of **variants**, each of which has a unique id, a set of filters associated with it, a priority (in case multiple variants are resolved for one feature) and a set of settings.
+
+Filters can be applied over evaluation context. Allocation of experiments can also be implemented as a special filter.
+
+## Excos Configuration based provider
+
+This document briefly describes the data model for features/experiments used in Excos Configuration based provider. _The Excos.Option beta changes made the abstraction much looser to allow for more provider led customizations._
 
 A **unit** is either a user ID or a session ID, or some other semi-persistent identifier for which we will compute experiment allocation.
 

docs/Extensibility.md

@@ -4,6 +4,6 @@ As I was writing the base code for the library I kept in mind it needs to be ext
 
 You can integrate with different platforms and SDKs providing experimentation via the `IFeatureProvider` interface, while providing binding to options objects with `IConfigureOptions`.
 
-You can create custom variant overrides (depending on the platform you run the code on) via the `IFeatureVariantOverride` interface.
+~~You can create custom variant overrides (depending on the platform you run the code on) via the `IFeatureVariantOverride` interface.~~ _Deprecated in beta update_.
 
 You can create custom filters for features and variants with the `IFilteringCondition` and plug them into the configuration based provider with the `IFeatureFilterParser`.

docs/GrowthBookGuide.md

@@ -108,8 +108,10 @@ Optionally, before integrating with Growthbook, you can test the Excos feature d
 
 ```csharp
 builder.Services.BuildFeature("TestRollout")
-    .Configure(feature => feature.AllocationUnit = nameof(StoreOptionsContext.SessionId))
-    .Rollout<CatalogDisplayOptions>(75 /*percent*/, (options, _) => options.ItemsPerPage = 20)
+    .Rollout<CatalogDisplayOptions>(
+        75 /*percent*/,
+        (options, _) => options.ItemsPerPage = 20,
+        allocationUnit: nameof(StoreOptionsContext.SessionId))
     .Save();
 ```
 
@@ -209,10 +211,6 @@ public class ExperimentationAssignment
     [Key]
     public Guid SessionId { get; set; }
 
-    [Required]
-    [Column(TypeName = "nvarchar(128)")]
-    public string ExperimentName { get; set; }
-
     [Required]
     [Column(TypeName = "nvarchar(128)")]
     public string VariantId { get; set; }
@@ -240,24 +238,34 @@ With that in place we can implement our `ExperimentationService` class which bas
 public class ExperimentationService : IExperimentationService
 {
     private readonly ExperimentationContext _dbContext;
-    public ExperimentationService(ExperimentationContext dbContext) => _dbContext = dbContext;
+    private readonly /*Excos.Options.*/IFeatureEvaluation _featureEvaluation;
+
+    public ExperimentationService(ExperimentationContext dbContext, IFeatureEvaluation featureEvaluation)
+    {
+        _dbContext = dbContext;
+        _featureEvaluation = featureEvaluation;
+    }
 
-    public async Task SaveExperimentAssignmentAsync(Guid sessionId, string experimentName, string variantId)
+    public async Task SaveExperimentAssignmentAsync(StoreOptionsContext context)
     {
+        // check if we already have an assignment for this session
         var assignment = await _dbContext.Assignments
-            .Where(a => a.SessionId == sessionId)
+            .Where(a => a.SessionId == context.SessionId)
             .FirstOrDefaultAsync();
 
         if (assignment == null)
         {
-            assignment = new ExperimentationAssignment
+            await foreach (var variant in _featureEvaluation.EvaluateFeaturesAsync(context, default(CancellationToken)))
             {
-                SessionId = sessionId,
-                ExperimentName = experimentName,
-                VariantId = variantId
-            };
+                assignment = new ExperimentationAssignment
+                {
+                    SessionId = context.SessionId,
+                    VariantId = variant.Id
+                };
+
+                _dbContext.Assignments.Add(assignment);
+            }
 
-            _dbContext.Assignments.Add(assignment);
             await _dbContext.SaveChangesAsync();
         }
     }
@@ -291,9 +299,6 @@ With that in place, let's add the experimentation service to Index.cshtml.cs and
 public class CatalogDisplayOptions
 {
     public int ItemsPerPage { get; set; } = Constants.ITEMS_PER_PAGE;
-
-    // Add this field - it will be populated by Excos
-    public FeatureMetadata? FeatureMetadata { get; set; }
 }
 
 // Index.cshtml.cs
@@ -302,14 +307,8 @@ public async Task OnGet(CatalogIndexViewModel catalogModel, int? pageId)
     var context = HttpContext.ExtractStoreOptionsContext();
     var options = await _contextualOptions.GetAsync(context, default);
 
-    // Add this call to experimentation service based on metadata
-    if (options.FeatureMetadata is not null)
-    {
-        foreach (var feature in options.FeatureMetadata.Features)
-        {
-            await _experimentationService.SaveExperimentAssignmentAsync(context.SessionId, feature.FeatureName, feature.VariantId);
-        }
-    }
+    // Add this call to experimentation service
+    _experimentationService.SaveExperimentAssignmentAsync(context);
 
     CatalogModel = await _catalogViewModelService.GetCatalogItems(pageId ?? 0, options.ItemsPerPage, catalogModel.BrandFilterApplied, catalogModel.TypesFilterApplied);
 }
@@ -348,18 +347,13 @@ Let's start by connecting GrowthBook to our SQL Server instance. Go to "Metrics
 
 ![Add data source](./images/growthbook-demo-create-data-source.png)
 
-With that we will redefine the Identifier Type as `sessionid` and create the experiment assignment query called `ExperimentAssignmentBySession`. In the SQL below I'm doing a bit of a manipulation with the ExperimentName and VariantId. This is due to slight misalignment between Excos data schema and GrowthBooks. A GrowthBook experiment is saved as a VariantId `{tracking-label}:{treatmentId}`, while ExperimentName would contain the Feature name. Meanwhile, a rollout or static value override defined in GrowthBook would be only identified by Feature name and variant id. This query will allow you to use both GrowthBook experiments and native Excos experiments.
+With that we will redefine the Identifier Type as `sessionid` and create the experiment assignment query called `ExperimentAssignmentBySession`. In the SQL below I'm doing a bit of a manipulation to extract experiment label and variant id for GrowthBook. This is due to the fact that Excos expects variant Ids to be uniquely identifying both the feature and variant. A GrowthBook experiment is saved as a VariantId `{tracking-label}:{treatmentId}`. This query will allow you to use both GrowthBook experiments and other Excos experiments using the configuration provider.
 
 ```sql
 SELECT
   SessionId as sessionid,
   Timestamp as timestamp,
-  COALESCE(
-    NULLIF(
-      SUBSTRING(VariantId, 1,
-        (GREATEST(1, CHARINDEX(':', VariantId)) - 1)),
-      ''),
-    ExperimentName) as experiment_id,
+  SUBSTRING(VariantId, 1, (GREATEST(1, CHARINDEX(':', VariantId)) - 1)) as experiment_id,
   SUBSTRING(VariantId, GREATEST(1, CHARINDEX(':', VariantId) + 1), 128) as variation_id
 FROM [exp].[Assignment]
 ```
@@ -422,8 +416,10 @@ I've created a simple A/A test (meaning no difference between control and treatm
 
 ```csharp
 builder.Services.BuildFeature("OfflineExperiment")
-    .Configure(feature => feature.AllocationUnit = nameof(StoreOptionsContext.SessionId))
-    .ABExperiment<CatalogDisplayOptions>((_, _) => { }, (_, _) => { }) // no change A/A experiment
+    .ABExperiment<CatalogDisplayOptions>(
+        (_, _) => { },
+        (_, _) => { }, // no change A/A experiment
+        allocationUnit: nameof(StoreOptionsContext.SessionId))
     .Save();
 ```
 

docs/Usage.md

@@ -5,7 +5,7 @@
 Following the Options.Contextual docs:
 
 1\. Define your context class that will be passed to `IContextualOptions.GetAsync()`.
-By default `UserId` property is used for allocation calculations. You can override it globally by configuring `ExcosOptions` or override it for a single feature by configuring the `AllocationUnit` property.
+By default `UserId` property is used for allocation calculations. You can override it for a single feature by configuring the `AllocationUnit` property.
 
 ```csharp
 [OptionsContext]
@@ -103,43 +103,6 @@ internal class WeatherForecastService
 }
 ```
 
-## Variant overrides
-You can also allow overriding the variant based on some context - example:
-
-```csharp
-class TestUserOverride : IFeatureVariantOverride
-{
-    public Task<VariantOverride?> TryOverrideAsync<TContext>(Feature experiment, TContext optionsContext, CancellationToken cancellationToken)
-        where TContext : IOptionsContext
-    {
-        var receiver = new Receiver();
-        optionsContext.PopulateReceiver(receiver);
-        if (experiment.Name == "MyExp" && receiver.UserId.IsTestUser())
-        {
-            return new VariantOverride
-            {
-                Id = "MyVariant",
-                OverrideProviderName = nameof(TestUserOverride),
-            };
-        }
-
-        return null;
-    }
-
-    private class Receiver : IOptionsContextReceiver
-    {
-        public Guid UserId;
-        public void Receive<T>(string key, T value)
-        {
-            if (key == nameof(UserId))
-            {
-                UserId = (Guid)value;
-            }
-        }
-    }
-}
-```
-
 ## Usages
 
 * Experiments

src/Excos.Benchmarks/ExcosVsFeatureManagement.cs

@@ -3,11 +3,9 @@
 
 using System.Text;
 using BenchmarkDotNet.Attributes;
-using Excos.Options.Abstractions;
-using Excos.Options.Abstractions.Data;
+using Excos.Options;
 using Excos.Options.Contextual;
 using Excos.Options.Providers;
-using Excos.Options.Utils;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Options.Contextual;
@@ -31,21 +29,12 @@ private IServiceProvider BuildExcosProvider()
         var services = new ServiceCollection();
         services.ConfigureExcos<TestOptions>("Test");
         services.AddExcosOptionsFeatureProvider();
-        services.AddOptions<FeatureCollection>()
-        .Configure(features => features.Add(new Feature
-        {
-            Name = "TestFeature",
-            ProviderName = "Tests",
-            Variants =
+        services.BuildFeature("TestFeature", "Tests")
+            .Rollout<TestOptions>(100, (options, name) =>
             {
-                new Variant
-                {
-                    Allocation = Allocation.Percentage(100),
-                    Configuration = new BasicConfigureOptions(),
-                    Id = "Basic"
-                }
-            }
-        }));
+                options.Setting = "Test";
+            })
+            .Save();
 
         return services.BuildServiceProvider(new ServiceProviderOptions { ValidateScopes = false, ValidateOnBuild = false });
     }
@@ -96,20 +85,18 @@ public object BuildAndResolveFM()
     }
 
     [Benchmark]
-    public async Task<string> GetExcosSettingsPooled()
+    public async Task<string> GetExcosSettingsContextual()
     {
-        PrivateObjectPool.EnablePooling = true;
         var contextualOptions = _excosProvider.GetRequiredService<IContextualOptions<TestOptions, TestContext>>();
         var options = await contextualOptions.GetAsync(new TestContext(), default);
         return options.Setting;
     }
 
     [Benchmark]
-    public async Task<string> GetExcosSettingsNew()
+    public async Task<string> GetExcosSettingsFeatureEvaluation()
     {
-        PrivateObjectPool.EnablePooling = false;
-        var contextualOptions = _excosProvider.GetRequiredService<IContextualOptions<TestOptions, TestContext>>();
-        var options = await contextualOptions.GetAsync(new TestContext(), default);
+        var eval = _excosProvider.GetRequiredService<IFeatureEvaluation>();
+        var options = await eval.EvaluateFeaturesAsync<TestOptions, TestContext>(string.Empty, new TestContext(), default);
         return options.Setting;
     }
 
@@ -128,17 +115,6 @@ private class TestOptions
     {
         public string Setting { get; set; } = string.Empty;
     }
-
-    private class BasicConfigureOptions : IConfigureOptions
-    {
-        public void Configure<TOptions>(TOptions input, string section) where TOptions : class
-        {
-            if (input is TestOptions test)
-            {
-                test.Setting = "Test";
-            }
-        }
-    }
 }
 
 [OptionsContext]

src/Excos.Benchmarks/Readme.md

@@ -1,23 +1,29 @@
 # Benchmark
 
 I wanted to see how does Excos compare performance wise to [Microsoft.FeatureManagement]().
-It is about 2x slower at the moment. However, I need to point out that it operates differently. With Excos and Contextual Options you're actually applying configuration over an Options class which can contain many different settings, while with FM you're checking a single feature flag to be True or False.
+It seems that Options.Contextual version is only a little bit slower than FM, while the pure Excos feature evaluation may be faster.
 
-I was able to optimize the amount of allocations with object pools. The biggest memory and perf hog was using LINQ.
-Given how most of the time you will be making fewer calls to Excos than to FM it might be enough at this point.
+However, I want to highlight here that Excos evaluation speed will depend on how the configuration binding and filtering is executed. This benchmark covers only a very small piece of the puzzle and real world performance characteristics may differ.
 
 ```
-BenchmarkDotNet v0.13.11, Windows 11 (10.0.22621.2715/22H2/2022Update/SunValley2)
+BenchmarkDotNet v0.13.11, Windows 11 (10.0.22631.4602/23H2/2023Update/SunValley3)
 12th Gen Intel Core i5-1235U, 1 CPU, 12 logical and 10 physical cores
-.NET SDK 8.0.100
-  [Host]     : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
-  DefaultJob : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
+.NET SDK 8.0.401
+  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
+  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
 ```
 
-| Method                 | Mean        | Error     | StdDev      | Gen0   | Gen1   | Allocated |
-|----------------------- |------------:|----------:|------------:|-------:|-------:|----------:|
-| BuildAndResolveExcos   | 21,844.2 ns | 427.07 ns | 1,063.56 ns | 4.3945 | 0.9766 |   27954 B |
-| BuildAndResolveFM      |  7,297.5 ns | 160.23 ns |   464.84 ns | 2.8381 | 0.7019 |   17921 B |
-| GetExcosSettingsPooled |    996.2 ns |  19.13 ns |    16.96 ns | 0.0591 |      - |     376 B |
-| GetExcosSettingsNew    |    872.9 ns |  17.20 ns |    16.89 ns | 0.1345 |      - |     848 B |
-| GetFMSetting           |    387.3 ns |   7.05 ns |     8.92 ns | 0.1578 |      - |     992 B |
+| Method                            | Mean        | Error     | StdDev    | Gen0   | Gen1   | Allocated |
+|---------------------------------- |------------:|----------:|----------:|-------:|-------:|----------:|
+| BuildAndResolveExcos              | 14,003.7 ns | 129.04 ns | 114.39 ns | 3.6011 | 0.8545 |   22784 B |
+| BuildAndResolveFM                 |  5,979.1 ns |  92.25 ns |  86.29 ns | 2.8381 | 0.7019 |   17841 B |
+| GetExcosSettingsContextual        |    439.9 ns |   8.43 ns |  10.35 ns | 0.1249 |      - |     784 B |
+| GetExcosSettingsFeatureEvaluation |    287.3 ns |   3.59 ns |   3.36 ns | 0.0930 |      - |     584 B |
+| GetFMSetting                      |    402.6 ns |   5.01 ns |   4.19 ns | 0.1578 |      - |     992 B |
+
+## Previous notes (before beta)
+
+It is about 2x slower at the moment. However, I need to point out that it operates differently. With Excos and Contextual Options you're actually applying configuration over an Options class which can contain many different settings, while with FM you're checking a single feature flag to be True or False.
+
+I was able to optimize the amount of allocations with object pools. The biggest memory and perf hog was using LINQ.
+Given how most of the time you will be making fewer calls to Excos than to FM it might be enough at this point.