Replies: 4 comments 4 replies
-
Thanks for creating the AEP. :D Here's my understanding of the Catalog idea :
Rough implementation : Currently, Antidote defines a global Uncertainties :
Let me know if I missed/misunderstood something @flisboac! |
Beta Was this translation helpful? Give feedback.
-
I must apologize in advance for not commenting earlier, but yes, that's pretty much what I had in mind. I only want to add one thing, which may very well be a nitpick, or not be justifiable as a functionality. Please do tell me what you think (because I'm on the fence about it, in a way). What about declaring a provider programmatically, ie.: # Just an example
from antidote import create_catalog, injectable, Catalog
class SomeService:
...
catalog = create_catalog()
catalog.provide(injectable.of(SomeService))
# Or even:
def build_catalog(config_file: str, *, enable_some_service: bool = False) -> Catalog:
catalog = create_catalog()
# ...
if enable_some_service:
catalog.provide(injectable.of(SomeService))
return catalog I suggest this because:
Specifically for (2), I only suggest it because in the past I found it very useful to re-provide dependencies with some small twists, like specifying a different configuration location, or other assorted parameters for catalog/module creation. This was especially handy when you had no option of changing some upstream library (e.g. the class has no injectable configuration, but you must inject that very same class, whilst trying not to break compatibility). |
Beta Was this translation helpful? Give feedback.
-
No, it's fine, not everything needs to be in the PR. The V2 has a lot of things in it. ;)
I'm not really in favor of it. I see two issues with it:
To me, it seems like the dependency you had to change should have been defined with the equivalent of |
Beta Was this translation helpful? Give feedback.
-
Well, now I realize that you cannot override dependencies even in V2. Every mechanism for dependency registration checks for duplicates in the other providers and child catalogs. Outside of Without overrides, specifying |
Beta Was this translation helpful? Give feedback.
-
Migrating the Catalogs feature request from #33 to a new Github discussion. All the relevant chat history is here; if I miss anything, or if a rewrite is needed with a "formal" proposal (i.e. more focused), just tell me!
P.S.: AEP stands for Antidote Enhancement Proposal 😎
Reading the documentation again, and some of the source-code, I now have a better understanding of the overall mechanism used by Antidote.
Not only the factory provider, but also the indirect provider, requires you to specify a factory function of sorts. I believe it must work this way because it's not possible to guarantee that the implementation classes are loaded (as in imported from modules and made available to the injector/app) unless the app import them at some point. Those factories guarantee that, and shifts the selection of implementation candidates to the user. It's a really smart decision, and I'm quite impressed.
Such a design forces your injection site (e.g. your app/lib) to indirectly depend on a predefined set of implementations. In a way, you still have the benefits of dependency injection, but the lack of service discovery makes implementing auto-provided library services a bit harder. I'll try to illustrate what I mean.
About multi injection
Back to the logger domain. As an example, suppose I have a "core" library declaring an interface
LoggerSink
. Then I may want to offer to users different implementations of a log sink, that will be injected by means of some indirect factory. An implementation would look like this:Code snippet: On library "my_logging_core" (with basic logging framework definitions)
Code snippet: On library "my_logging_remote" (with sinks for some external logging service/api)
Code snippet: On library "my_logging_os" (with sinks specialized on services in the local OS)
Code snippet: On library "my_logging_fd" (with sinks specialized on file descriptors)
Code snippet: On library "my_logging" (the entrypoint for the logging framework)
Code snippet: On a final app (i.e. actually using the "logging framework")
(I haven't tested this code, but I suppose it should work. Please correct me if I'm wrong!)
That is all completely fine for smaller apps, domains or teams. But as those increase in size or scope, or are further broken down, maintaining such a tight grip of dependencies at the library level (in my example, through
all_log_sinks
, or similar mechanisms) can become increasingly harder. As a library or framework author, the more freedom you can give the user, the better. I agree we should not suggest bad practices, or overwhelm the user with options, but in this case specifically, a better service discovery may allow those authors to loosen coupling even more, and allow more decentralization whilst maintaining a sane baseline (in the form of interfaces, and similar elements).Concerning this scenario, one could say that the app developer already have the freedom he needs, but that freedom comes at the cost of unnecessary complexity. For example, he could reuse the
multi_inject
mechanism to select exactly which sinks he wants to use, and then construct theLogger
himself:Code snippet: On a final app with customized log sink factories
This use case could be handled better, and by Antidote itself. What if we had a concept in Antidote for a collection of provider elements (Service, Component, Factories, Indirects, etc), much in the same vein as Modules in NestJS or Modules in AngularJS? The idea is for a library author to provide said collection, and it's up to the user to select which of those collections he wants to include in the auto-injection mechanism.
I would call such a collection a catalog (as in service catalog, etc), so that it does not conflict with Python's concept of a module. Also, I think catalog is a better description of the intention of that functionality.
NOTE: The same argument could be applied to single-element (but likewise indirect) injections too.
... And yet another proposal!
A catalog could be declared as a simple Python module with some well-known exported, non-dunded properties. As of now, from their NestJS and Angular concept counterparts, the only property we should care about is
exports
andimports
.exports
is mandatory, and would specify the injectables that will be considered when no injection is provided by the other providers. Injectable type could be deduced by how the element is decorated (e.g.@implements
, inheritingService
; possibly with some helper to determine which kind of element it is.imports
would be used for when a library needs to include the injectables of another from inside this catalog mechanism. It would incur some degree of indirection, but nothing that could be e.g. easily followed in any IDE by simple CTRL+click, etc. Also,imports
is entirely optional, and would just add the imported catalogs' exports (transitively) into the set of possible injections. (I guess this would be slower than the current mechanism, but not by much, as long as the chain is not too long, and the candidate set is not too big; perhaps some of the work can be optimized by pre-calculating the entire candidate set on Cython?)This would allow Antidote to keep the design philosophy of "explicit is better that implicit," because it would be clear for the user (from both library and app perspective) from where the injection candidates are coming from.
Expanding on the previous example (supposing we refactor it appropriately):
Code snippet: On module "my_logging_remote.catalog"
Code snippet: On module "my_logging_os.catalog"
Code snippet: On module "my_logging_fd.catalog"
Code snippet: On module "my_logging.catalog"
Code snippet: On the final app (default)
It stays the very same way.
Code snippet: On the final app (with custom log sink providing)
Note that this is all mostly about indirect providing.
By this design, perhaps a new
CatalogProvider
could be implemented, to cover the case when a provider is not immediately offered by the other providers. This would keep compatibility with the current injection mechanism, and offer an alternative for when a greater degree of control of indirection is needed. Libraries can have as many catalogs as needed, or offer some catalog factory to customize provisioning (much in the same way the very commonModule.forRoot
pattern in Angular works).Note that this is on top of the qualifiers for which this issue is for. Catalogs would be there to provide entire sets of injectables, and in some way filter them, at the container level (i.e. coarser control). Qualifiers would be used to slice and/or filter injection at the injection point level (i.e. more granular control). Also, contrary to modules in NestJS or Angular, there would be no isolation between catalogs, in the sense of internal vs external injectables (i.e. no "privates"), as I guess it would be quite hard to implement something like that and it would also not bring as much benefit.
One disadvantage of this approach is that I don't see an obvious way to alter
world
to make it also work with catalogs when injecting:clone()
, but using parent setup to look up injections, hence propagating the catalog set downstream)?world.get()
also accept catalogs, and each injection would be distinct from each other in terms of lookup?ContainerRef
s, and return a customized "world")Those questions prevented me from trying to implement a new Provider. I needed to check the feasibility of such an idea, hence why I'm walltexting (I'm sorry) :x
Follows the rest of the discussion history regarding this functionality:
From @flisboac
(...) I'd also add the possibility for a dependency to be resolved through the
ModuleRef
, with a flag to enable searching the dependency outside the catalog (ie. app and world-level) -- which is inspired by the same functionality in NestJS (in NestJS parlance, that isstrict: false
), and that I did use in more than one occasion.The advantage of this approach is that:
world
when searching for a dependency dynamically.In my initial proposal, the idea was for
world
to have "globals" (global-level dependencies), and nothing changes in regards to how it's implemented today. Whereas catalogs would implement "scoped" dependencies, much in the same way how modules in Python work (ie. you need to import dependencies to use them, and imports are local to the importing catalog; with optional caching, perhaps).In regards to the
@inject
decorator, the catalog selection would only apply to the entrypoint (ie. yourmain
function). I don't think injection points in constructor/function parameters need this at all.This would lessen the amount of dependencies the DI mechanism need to register and traverse, and it makes the whole mechanism a bit more scalable. Tests could be smaller and focused on specific catalogs, for example. Or the app may offer multiple entrypoints, each of them with their own catalog selection.
Otherwise, the only thing you would need, at the app-level, is to import the catalog, and then antidote would automagically include it, which is not very explicit and would raise some warnings (e.g. flake8 complaining the import is not used). Also, all entrypoints would see all catalogs as well, which defeats the purpose (as far as I understand it).
But I admit this could be an immense change to the codebase, in which case opting for a static config approach would be the only way. I also don't know how it'd play out with the Providers mechanism (do we extend the Provider API to allow for the catalog implementation? How to integrate existing providers with this supposed new means of resolution?). But again, this is just an idea; if it's not feasible, I will understand, and it's not a problem at all, antidote is already plenty helpful as it is!
From @Finistere
From @flisboac
I was considering the
@inject
use case, for when you configure a catalog per app entrypoint (ie. per "main" function), as you would want to have as many as needed. Updating my samples from before:Perfect! That's how I initially imagined them to work. More specifically, my worries were about duplicating efforts when splitting between world and catalogs' provider implementation. I understood initially that there would be distinct implementations for both, but looking at Antidote's code, I understand better what you mean.
What about this, for example, for the indirect provider (src/antidote/_providers/indirect.py)?
In any case, each catalog has their own provider instances, as you said. The user can add more providers, if needed (or not; I'm not entirely sure if it's a good idea, though).
From @Finistere
From @flisboac
The context manager syntax makes a lot of sense, and I'm pleased by the API you proposed. The
@inject
support would be a shortcut (an extra, or syntax sugar) to the user. It's not necessary at all, it's just to avoid the need for writing two functions for each entrypoint, or for when programmatic dependency resolution is not going to be used. It would allow for each entrypoint to be injected with the dependencies they need right away, at annotation level. Also, parameters that are not injected could still be passed along, something that may be useful for standalone functions (like the entrypoints).The same machinery used to implement the context manager idiom could be used to implement the decorator-based one (unless there's some problem with it, which I'm not seeing).
Regarding my example, the implementations of
aws_lambda_main
andaws_ecs_main
are not the same. Sorry for not making that obvious at first. They have different needs, do different tasks, or inject completely different dependencies. Therefore, their code is what is going to differ, and is what matters, not their injections. Them having the same dependencies is only coincidental. I just didn't bother to put different implementations for them at the time. :(Folows an updated example:
As a library author, I'll provide catalogs that can be used by both end-users (lambda and ECS). Catalog selection is primarily the responsibility of the end user (or, of a library provider, transitively).
Another advantage of
@inject
-decorated functions that they can be called as-is, for simpler use-cases. Now, how this is going to play out when they're called in the context of some other catalog is something I'm not sure myself how to deal with, and may be the reason for some opposition to this idea. When I first wrote the proposal, I was thinking about catalogs overriding whatever was in their context, much in the same way it is done today withworld
, but without aclone
. For example:But your suggested use is perhaps clearer in intent, and better from an Antidote-implementation point of view. Rewriting the previous example, I would have the following:
(...)
As a side note...
My specific needs, as of now, are for relatively short executables, parameterized by a combination of CLI arguments, environment variables and configuration values coming from all sorts of external places/services. Each project (a repository; one or more for each team and product) will have an assortment of job implementations of varied types, written as Python functions, executable modules or normal Python scripts. There will be a lot of projects, and a lot of jobs for each project (ie. it's numerous in both aspects). Each job type parses, fetches and merges parameters (CLI, env-vars, etc) in different ways, but some common properties among them are somewhat guaranteed. Each job will execute in a specific environment, or will interface with a specific systems, so a considerable number of services (service classes, etc) will be offered in the form of an internal framework, to standardize those services and some aspects of job implementations.
Catalogs would allow me to extend a baseline with the necessary implementations for each job type (not each job!), without enforcing the entire framework upon all of them (ie. not putting everything in world). Each job would pick the catalogs they need, so that the amount of scanned dependencies could be reduced, and dependencies could be more easily traceable. It's still up to the job implementation to orchestrate how its injected services are used, or implement details not covered entirely by a service (ie. things specific to that job).
I can see most of the smaller or simpler jobs using the short (
@inject
) form, because some use cases can be abstracted away quite significantly. This is what we were focusing on (ie. simplifying and reducing the amount of code we need to write). But the bigger ones may be complex enough to warrant more code. In both cases, the use of a DI library would do wonders.Well, that's the inverse of what I was thinking of, in terms of dependency resolution order.
My catalog proposal was based on my experiences with NestJS modules. I also used it as a base model. Please take a look at this diagram, from their documentation:
My initial idea was for dependencies to be looked up catalog-first. If
strict: True
, only the catalog and global-level (in our case,world
) is looked up. If dependency is not available locally, andstrict: False
, it would be looked up at application-level (which in our case, would be the application entrypoint's catalog) or at global level(in our case,
world), but only if
strict: false`. Overrides could then be implemented in terms of replacing the entrypoint's catalog with another, wich in turn imports the initial catalog. World could also be a last-level catalog, added by the resolution algorithm automatically.With providers last, dependencies would always come from
world
first, if catalogs would importworld
in some fashion. Perhaps that's why you favor world dependencies to be explicit, but I don't think it's a good idea to force specific catalogs, not even world, because it would become unwieldly rather fast. Provider-last also means that the context-manager will be the only way for dependency overrides to happen -- which I can deal with, but it's a bit limiting.In general, apart from factories, I don't think injection points (e.g. a type-annotated parameter or property) should bother with where the dependency comes from, at all. The only point at which it is relevant to specify a catalog, in my opinion, is on your application's entrypoint, or on catalog dependencies. The initial catalog selection is obviously important, but in any other point, neither library nor application authors should bother with where an injected parameter comes from.
Up to now, for factories, that degree of specificity was necessary so that importing the injected (implementation) class could be guaranteed. Now, the catalog is doing pretty much the same (guaranteeing that all the classes it offers are loaded), but in a well defined "DI package" that's explicitly selected by the user. That's why I don't think we need to be this specific.
In my understanding, a world dependency is there just so that it is available to all catalogs, regardless of whether they import each other or not. Such a feature should be used judiciously, though, e.g. a framework based on Antidote can provide some DI-managed registry service, but library authors using said framework should focus on providing their dependencies through catalogs.
Even when disconsidering world, if dependencies come from imported catalogs first, a bit of the benefits of having a catalog "scope" is lost, as local dependencies will be ignored in favor of external ones. For long chain of imports, the more front-facing catalogs lose a bit of relevance as dependency providers themselves, and would be mostly relegated to be simple catalog importers.
Also, by establishing catalogs and its inter-dependencies, we may have multiple injection candidates. The idea was to use catalogs not only to group those dependencies, but to also provide specialized ones. This is especially true for indirect providing (ie. when what's being requested is some ABC/interface). If imported catalogs have the preference, I will need to tell explicitly from which catalog each dependency comes from, either with context managers or some other mechanism, when said specializations are to be preferred. It's a level of specificity that I didn't want to enforce users with. It also makes it harder for library authors to provide said specializations, for the same reason. Qualifiers can alleviate this problem, but because of the resolution order, dependencies nearer the entrypoint catalog will have the least preference by default, which can be unintuitive.
From @flisboac
Regarding this again, I spent some time thinking about this resolution order specifically, and on second thought, it may make more sense for provider-last to be the default.
It guarantees a more deterministic resolution, because it will always go for the first provided dependency in the catalog hierarchy, much in the same way as class/module loaders are implemented in most languages. Once the dependency is found, that very same dependency is used again on later resolutions, something that won't be guaranteed if providers are checked first. Now, depending on the size of the catalog hierarchy, I'm not sure if it'll be easy for users to find where the dependency comes from, but from a "dependency loader" point of view, this makes more sense.
So, now I think your approach is the right way to go. But how will overriding work in this case (e.g. during tests)?
From @Finistere
From @Finistere
From @flisboac
Just to be sure we're on the same page regarding the terminology... The only mechanism by which you could be able to establish some kind of catalog hierarchy is via importing, right? In this case, the dependent catalog (child) imports its depending catalog (parent). So, in this case, the dependent catalog (child) would provide first, and only then would it provide from parents.
(You could also invert that logic, but well... Which will it be?)
That's reassuring. I was a bit unsure as to how this could be implemented.
Well, I'd add it only to
@inject
, because that could denote an application entrypoint, for the reasons I exposed in my previous comment.@factory
and@service
are just injectables (dependencies), and they will be part of a catalog, not require one. Only catalogs import catalogs. Consider a catalog like you would a Python module, and it'll make sense. Defining the catalog in an@inject
is only valid because the decorated element is outside the dependency injection mechanism (i.e. is not an injectable; it only receives injections).Just as an example, from NestJS's documentation, this is how you create a module:
Note the decoration there. We could even follow the same idea, and type-validate the new module via some
Protocol
type (instead of e.g. forcingCatsModule
to inherit aCatalog
class).In NestJS, providers can be classes, as long as it is decorated with
@Injectable
.Now, your application must run in terms of a "root module". Any module can be a root module. For example:
Note the
imports
there. In this case,AppModule
depends on and importsCatsModule
. That's what I'm suggesting we follow as well. (You can even export imported modules, which is super useful to create modules that aggregates functionalities from multiple modules!)To execute the app, you need to create an "entrypoint": it's either a script, or a function, etc, which instantiates a
NestApplication
from an initial module:The
app.listen
is only relevant if you have controllers in your app, which are REST endpoints NestJS automatically exposes via an internally managedexpress
web server. In this case,CatsController
would be served locally, atlocalhost:3000
.For Antidote, the approach should instead be more similar to standalone apps, because Antidote is not a web framework. That means some component of your app should be delegated as the entrypoint; it would then fetch and execute those managed services. For example:
bootstrap
is the entrypoint here.SomeTask
is an injectable class from some module. It's either provided or imported by the "root" (app) module. (It may be hard to find from whereSomeTask
is being provided, but I think that's for the better, because in most cases this won't be relevant. You should depend on the interface, not the implementation (e.g. in Python terms,SomeTask
could be just an ABC). I think just following the trail of module imports (or, in Antidote's case,world.debug
) is a good compromise for allowing IoC and improving discoverability.)What I suggested for
@inject
was some automation of this "entrypoint" logic. The entrypoint would not be injectable, but it could receive injections. Instead of doingget
s, you could just decorate some function's parameters with the types you want to inject, and@inject
would do the rest, provided you parameterize it with the catalog you want as a source/root.Beta Was this translation helpful? Give feedback.
All reactions