Web of trust related DVMs #38
Conversation
|
I think it would be useful to discuss some specific uses cases and consider which would be the more useful method to deliver WoT data: the proposed NIP-85 Trusted Assertions (see discussion here) or WoT related DVMs (this NIP). Here's one specific use case: Suppose a client wants to stratify reactions to a post, meaning: put reactions in order based on one of the reaction authors’ personalized WoT scores, which could be: follower count, “verified” follower count, GrapeRank score (contextual or generic), PageRank score, some other centrality score, number of hops away, etc. etc. Some of the above WoT scores, like PageRank, GrapeRank and other centrality scores, will require precalculation. No way around it. So I'd argue those should be exported using NIP-85, the advantage being ready availability, the disadvantage being they might be a little out of date. Other scores, like follower count, can be calculated on the fly and perhaps should be delivered via DVM, the advantage being the data would be more up to date. |
| [ | ||
| "param", | ||
| "limit", | ||
| 5 |
There was a problem hiding this comment.
these should be strings ;)
| 5 | |
| "5" |
There was a problem hiding this comment.
Hey @pablof7z, thanks for the review. Real question: why use strings? It doesn't matter and we can change it no problem, but I'm curious to know why.
Also, since you are a prolific client developer, what do you think about these DVMs more on a higher level? What would make you consider integrating them into Highligher or Olas?
There was a problem hiding this comment.
@pippellia-btc Pablo is right, these are nostr tags which should be strings
| [ | ||
| "param", | ||
| "proofs", | ||
| false |
Just want to say this is exactly the use case for
Not necessarily. We compute global Pagerank on the fly in about ~100ms, personalized Pagerank in ~800ms, and there are ways to further improve it. Also, there is a potential big risk in using cached values. What happens if Jack Dorsey key is hacked? Using cached values means that potentially a recommendation system might continue to recommend his key after the hack. That can cause people to get scammed. Therefore I would argue that using cached values should only be done for non-important metrics. Anyway, @wds4 what do you think about the structure of the DVMs? |
|
@pippellia-btc - before talking further about whether values should be cached and scores precalculated, let me see if I understand the nuts and bolts of how your system works, and you correct me if I'm wrong. You've said that you calculate scores for ~ 300k pubkeys, and you don't have a local cache. So when someone makes an API request, the first thing you do is query a handful of the most reliable relays (primal, etc) for kind 3 notes. (I'm assuming Vertex does not run its own relay.) Let's say you get 150k kind 3 notes back. Next, you extract every follow from those notes. I'd estimate that's ~ 5 million follows and it gets you up to the 300k figure (bc it includes pubkeys that are followed but don't have their own kind 3 notes). And that's what you use as the raw data for your calculations. Am I correct so far? |
I probably won't be able to make a meaningful evaluation until I try to build it into the grapevine-brainstorm API, which will force me to dig in to the nuts and bolts of this proposal. I've never worked with DVMs and I'm not quite ready to do that yet. Fiatjaf noted that he's never seen a working DVM, which doesn't exactly motivate me to invest the time to do that (yet). This is why I want to hear client devs weigh in on how they'd like to ingest WoT data: DVMs, Trusted Assertions, or something else. If there's a demand from client devs for WoT related DVMs, that'll motivate me to dig further. |
No no, I meant that the scores are not cached. Of course we have a continuously running crawler that stores the kind3s in a database (Redis). After each new kind3 event, we internally update some data structures (random walks). Because we do all this in advance, when a new request comes in, we are able to respond quickly with the pagerank values calculated on the most recent graph. |
OK, gotcha. So let's suppose the Jack Dorsey key got hacked 60 minutes ago and has started following a bunch of bots. It's not spamming content though, so most people don't notice. Presumably, your reddit database will always have the most recent kind 3 event from the hacked account. You calculate personalized pagerank in 800 milliseconds, but you're still using poisoned data. I presume you're counting on lots of people unfollowing him during those 60 minutes. But realistically, most plebs are not on nostr constantly and will never even take note that he was hacked. The best solution as I see it here is to make use of a reporting system, like mutes or NIP-56 reports, and give those a much greater weight than a follow. Realistically, if Dorsey gets hacked, he'll figure it out at some point and let Odell and a few other people know in the real world. We'll end up with some small number N of people who unfollow and flag his account as hacked, and they will do it as soon as he alerts them. But still, there will still be some larger number M >> N users who are still following the hacked account and probably will indefinitely. If those flags are incorporated into the score and weighted heavily enough, then it doesn't matter (or at least it matters a lot less) that he's still being followed by hundreds or thousands of users. This is precisely how GrapeRank works, and the relative weights of the various inputs are adjustable so we can determine empirically what constitutes "weighted heavily enough." It can be recalculated as frequently as we want, either on demand or on a schedule. Once a month, once every few minutes, anything in between. And if the server is down, no worries because the most recent scores are accessible as 30382 notes. Have you worked mutes or reports or some other flag system into your pagerank implementation? |
|
Just to sum up a few implementation details:
Presumably we can both speed up our PageRank calculations. I'm sure I could get GrapeRank calculating faster but that's not my focus right now. So many other priorities, including contextual GrapeRank scores which will require the base GrapeRank scores first and will only add to the total calculation time. And what happens when nostr usage increases by 10x, 100x, 1000x? Will it be feasible then for either of us to do all PageRank calculations on the fly? My presumption is that at some point there are going to be SOME calculations that cannot be done on the fly. No all, but some. So what's our next step? I really think we need to hear from client devs. If one or more client devs say they would like to access a WoT DVM then I'll look into building one and go from there. To speed things up, @pippellia-btc have you considered doing your own PR to incorporate a WoT DVM into a client? Or wait - is zapstore already using your WoT DVMs? |
Not really a lot. If high pagerank npubs (e.g. Odell, Gigi, ...) start unfollowing him, that will drop his rank significantly.
I agree that mutes and reports are a powerful tool to increase the speed of hack detection. We haven't implemented them yet, but it's coming
I think so, since our algorithms are O(1) in the size of the graph, and we can scale quite a bit vertically. Anyway, thinking about these scenarios is premature optimization imo.
Yes, zapstore is using it since v0.23. I think this discussion is OT, so maybe we can continue the conversation about our different implementation in another place, wdyt @wds4? |
Absolutely. Here I'll try to use my WoT to ping some relevant devs who have been interested in this topic before. Hopefully they won't consider this spam :P @vitorpamplona @staab @mmalmi Hey guys, what do you think about these three DVMs? What would make you consider integrating them into your clients? What's the biggest obstacle? |
|
Instead of making one DVM that does many things based on arguments, like the algorithm choice here, I would remove the parameter and make a separate DVM for each algo. So, for clients, they just need to get all the DVMs that reply user lists and pick the algo by the DVM name in the DVM profile. Clients don't decide which algo to use or which parameters to use. Users do. So, It's better if each DVM pubkey sets up their own parameters in your structure. |
|
Thanks for reviewing @vitorpamplona
I see your point, but I don't see the notion of a user "DVM profile" playing out (if that's what u meant). In my mind algorithms are easily swappable. For example, a user is on the "discovery tab", which calls Also, having one kind per algo means we would run out of DVM kinds very quickly. |
|
Not one kind per algo, one DVM per algo (different pubkeys) using the same kind. |
|
For instance, here is what we show for DVMs that return posts. Each one of them takes the exact same input from the App but run different algos. The user chooses. The DVM operator then has to explain to the user why his/her algo is better. |
This is an important point. And we should envision systems that place no upper limit on the number of algos, no upper limit on the complexity of the available parameters. Whatever changes to this PR that would best align with that, I'm in favor. |
This. Params are very close to a smell and most often than not should just be different pubkeys; it's impossible to capture in a structure all the different nuances of algorithms. |
|
While Vitor's list-based UI approach to DVMs solves for note discovery, web of trust DVMs are not meant to be used this way (except perhaps Parameters are not smell. Smell is UX that confuses the user.
The way we envision these DVMs working is integrated into relevant sections in the app. For example Whether parameters or options for displaying the response, these DVMs could have a place in the client settings to allow users to configure them. Simple things should be simple, complex things should be possible – that's good UX in my opinion. Even in the case of a |
|
@vitorpamplona you recently seemed to be in favor of tags/params, you no longer think the same? nostr-protocol/nips#1534 (comment) |
|
The user param on VerifyReputation is fine. Clients will always have to show the list of DVMs available to run that verification. So, on a Profile Visit, for instance, the client will ask which DVM to use for the first time and the re-use that setting for the future visits. And then we need to add a settings page to change that option. But the list of DVMs will always be there. The user will always choose which service provider and verification algorithm they want to run when visiting profiles. And that's the DVM discovery screen for that service.
The Trusted Assertions framework is similar, you choose a pubkey for the |
Yes, absolutely. I was making reference to the DVM list that currently is in Amethyst that works without context. Our DVMs need context.
Yes. As well as other parameters.
The algo is just another parameter. If the sentiment here is one pubkey per algo, fair enough we can consider it. The advantage of a single pubkey for all DVMs, though, is the reputation of our service provider. You know it's Vertex because it's signed with its main key, that carries social signal. If you have other pubkeys how do you do that? A hack would be for Vertex to follow those pubkeys, but it's not clear enough to the user. |
I don't think keeping all algos in one key makes sense from a reputation perspective. People don't care about the brand that has all the algos, they care about one algo in particular. The algo has the reputation, not the collection of algos. In fact, I think mixing algos that a user likes with algos that are less interesting to that user dilutes the reputation of the main key. |
|
I can't agree with this. People absolutely care about brands and there's books written about it. Brands have better and worse products. This doesn't mean that each product has to be a brand. Companies can tell product success by measuring sales. Same could happen here but since our algos are free, zaps/v4v could work as a proxy. |
|
My point is that each DVM (each algo) is a separate product and it should be named as such. Primal has dozens of DVMs. No one cares about the set of Primal DVMs. They care only about that specific DVM that works for them. By making them select "Primal" and then an algo, you are forcing them to consider and accept the brand first, not the product first as usual. You are free to name all your DVMs as "Brand: Algo", but my experience is that this doesn't go very well. |
Interested - what have you seen going wrong?
True if you assume the interface is the currently designed in Amethyst. I don't want to show "Brand: Algo". I want to show "Algo" and have my pubkey with profile picture underneath. Imagine how cool it would be to tap on the vendor's profile and run a Also, when users browse other DVMs in other parts of clients, or even new ones, having a discrete but recognizable profile picture of the vendor can save the user from wasting time determining trust. |
The difference between Primal's DVMs, which all have "Primal" at the end, and @believethehype's DVMs which don't have anything in common is irrelevant for users. In fact, people like @believethehype's DVMs better and see Primal stuff as centralization. And they are all different keys already. Imagine if it was just one.
Great, but then it will just be a bunch of your profile pictures with different names. And remember, anybody can create a new pubkey, use your profile picture and add to the list. So in the end, the user will have to "follow" the DVMs they like and trust. And instead of following a 100s of algos from the same brand, my take is that they want to follow just the thing they actually use. |
Is this based on hard data? Because everyone talks bad about Primal centralization all the time for anything remotely related to them, if I had to guess I'd say this is not specific to Primal DVMs or the fact they use "Primal" in the DVM name.
Yeah this is exactly what I'm saying I don't want. Here's an example: People follow Vertex. Vertex has many DVMs. DVMs show their title, and below it, in a little container, the Vertex profile picture. Profile can now be passed through a With the current approach we are completely disconnecting DVM reputations from the web of trust. Idk, it would be good to at least have a tag where the vendor's pubkey is included. |
I think this is a key point. DVMs that return content are easy to try because the stakes are relatively low. However, A good proxy for knowing if a DVM will work properly when you need it is the reputation of the operator. Note that this is EXACTLY the WoT that we are all trying to encode in the algos! So my point is that when the stakes are high, people will choose the brand first and then the product. Second, if we have a pubkey-per-algo DVM, the number of options will skyrocket once we add context. For example, |
Neither of these is the right way to slice kinds. Kinds should vary based on interface not implementation. So, many DVMs may implement support using a single kind — but that kind should have a singe semantic meaning, and a pretty small surface area for possible parameters. If the algorithm is itself a parameter, that doesn't imply anything either way necessarily, but if the algorithm implies a different type of result set, then it should be different kind. |
I think there might be some confusion in the meaning of "algorithm", so I'll try to first clarify what we've been talking about so I can better understand your point. First, the 3 kinds ( |
|
My inclination would be to split those up into different kinds. NIP 89 handlers can advertise support for multiple kinds though, so you could still have one DVM listing but for multiple jobs types. |
@vitorpamplona are you using NIP-89 to discover DVMs? |
|
Yes, but only the ones that return a feed of posts. Each return type needs its own screen.. so... it takes a while to code. :( |
Can we direct focus back to the DVMs being discussed here? Post feeds are a completely different usecase (low stakes, non-configurable, and don't require context). I think we all are interested in discussing DVM design starting from the UX perspective. As an example, a client like Amethyst could optionally integrate the On Zapstore, for example, I intend to enable swapping/configuring DVMs in Settings. But otherwise they are transparent in the UI. |
We have been working on a few different web of trust services that, as of now, are fully operational.
We are starting with this draft spec and would like to use this PR as a conversation starter for others to give feedback.