BEP 52 - Define client announce behavior when joining both swarms of a hybrid torrent

[X-Coder264]

This is a follow-up to the discussion here (started from this comment).

TL;DR

BEP 52 currently states

For interoperability with BEP 3 a torrent can be created to contain the necessary data for both formats. Implementations supporting both formats can join both swarms by calculating the new and old infohashes and downloading them to the same storage.

It says that a client can join both swarms, but it doesn't specify the way how should the client announce its traffic to the tracker in such a case.

For example let's say a peer uploaded 100 MB in the v1 swarm and 200 MB in the v2 swarm. How must a client announce this to the tracker? I see two possibilities:

a) /announce?info_hash=<v1_hash>&uploaded=100 MB and /announce?info_hash=<v2_hash>&uploaded=200 MB

or

b) Both announces having the same (total) amount of uploaded/downloaded data (uploaded=300 MB in this example)

I'm opening this issue as I think that this should be clearly specified in the BEP as the choice between the two has great impact on both the client and the tracker's implementation. For example for libtorrent, keeping separate stats (the a) option) would add significant complexity to the code. Also obviously, if one client implementation goes with the a) option and another with b) that's gonna cause big headaches to trackers who care about those numbers (basically all private trackers).

These are very good reasons as to why I think that this should be specified in the BEP as I think there should be no room open for different interpretations (aka different implementations).

[the8472]

What is easiest for the client depends on how it's implemented, i.e. whether it's a) two separate torrents which happen to be backed by the same storage (plus minimal crosschecks for completion) or b) handled as a single torrent with hybrid-aware logic in the wire protocol, announces, hash-checking etc.

Superficially a) seems simpler. But apparently, given the linked issue this is not the case in practice?

Either way, I have no strong preference for either since I honestly do not care much how exactly ratio accounting works and since private tracker operators rarely provide feedback for protocol design.

If you can settle on one and maybe rope in some tracker and client implementers into the discussion we can add that to spec or write an additional, informational BEP that documents ratio accounting common practices.

BEP52 is still a draft and basically waiting for implementation experience, so feedback like this is welcome.

[arvidn]

would it be reasonable to only announce stats to the v2 info-hash, if it's a v2 torrent?

[ssiloti]

@the8472 When implementing BEP 52 in libtorrent I found it made much more sense to use approach b.

From a UI perspective we want to present a single torrent rather than two. That's vastly easier if the two swarms are hidden behind a single torrent object. It turns out that pretty much all of the state in the existing torrent class makes sense to share between both swarms, so I just added hybrid-aware logic to the handful of places where it's needed.

[the8472]

@arvidn hybrid torrents are the concern here. In that case you want to announce both to get peer lists for both.

[X-Coder264]

What is easiest for the client depends on how it's implemented, i.e. whether it's a) two separate torrents which happen to be backed by the same storage (plus minimal crosschecks for completion) or b) handled as a single torrent with hybrid-aware logic in the wire protocol, announces, hash-checking etc.

I didn't even consider option a) as far as torrent client implementation goes and I don't know how hard it'd be to do that (especially from a torrent client GUI perspective where these "two" torrents should still be shown as one so as to not confuse the users). Option b) as @ssiloti mentioned in the linked issue seems to be somewhat complex to implement.

... private tracker operators rarely provide feedback for protocol design. If you can settle on one and maybe rope in some tracker and client implementers into the discussion we can add that to spec

The whole reason I raised this issue in the first place was because I started working on a torrent tracker implementation from scratch recently. This is a heavy WIP and the only reason the repo is public atm is because I'd have to pay for Travis CI otherwise. The announce implementation part is done, the reason it's WIP is that I still have to add a bunch of features that a user would expect from a private tracker. I was also a "operator"/main dev of a private tracker for years so you could say I have experience on the topic (at least from the tracker side of the story). Anyway, the tracker is currently fully BEP 3, 7, 23, 31 and 48 compliant. As I'm writing it from scratch I thought I could make it BEP 52 compliant as well, especially since BEP 52 has been in draft state for a year and it's highly unlikely that anything big is gonna change about it before it goes from draft to "accepted" state and so it doesn't make sense to ship a new tracker implementation from scratch without BEP 52 full compliance as support for BEP 52 requires quite some breaking changes from a tracker perspective (different database table schema etc.).

would it be reasonable to only announce stats to the v2 info-hash, if it's a v2 torrent?

That's exactly what I was getting at. I thought since option b) from my first post is a lot easier to implement from a tracker perspective as the data being reported to the tracker is cumulative so we just take the maximum of the two values, but that also means that one of the two announces to the tracker is redundant and if there is gonna be a lot of hybrid torrents on the tracker it's gonna mean that for each such hybrid torrent there's gonna be twice the needed amount of announces towards the tracker which could cause problems if we are talking about a high traffic/load tracker. Making the announce only with the v2 hash if it's a v2 torrent seems to be a great solution - I don't see any drawbacks atm.

@arvidn hybrid torrents are the concern here. In that case you want to announce both to get peer lists for both.

Is there any reason why the tracker shouldn't send both v1 and v2 peers info back to the client even if the client is only v1/BEP 3 compliant? The v1 peers should communicate normally just like they have been doing before BEP 52, and peers with v2 torrent clients should also be able to talk to such v1 only peers, no?

[the8472]

Is there any reason why the tracker shouldn't send both v1 and v2 peers info back to the client even if the client is only v1/BEP 3 compliant?

The thing is the client can't assume that the tracker will do it. For backwards-compatibility and minimal adoption friction it has to assume the tracker is BEP52-oblivious.

[X-Coder264]

@the8472 I don't quite understand what you are trying to say. What "backwards-compatibility and adoption friction" problems is gonna cause if a v2 client announces only with the v2 info_hash? In order for the torrent to be v2 and have a v2 info_hash it means that the user already had to get a v2 .torrent file from the tracker which means that the tracker is already BEP 52 compliant.

[the8472]

You are assuming that torrents are somehow tied to trackers. That assumption is incorrect.

[X-Coder264]

I still fail to see what you are trying to get at or in what case would that present a "backwards-compatibility and adoption friction" problem, but if that option is off the table then I guess option b) from my original post is the way to go as it seems to be the easiest to implement from both client and the tracker side.

btw, does that mean that even when a v2 client announces to the v2 hash, the tracker should return only v2 peers (or also v1)?

[the8472]

To be able to use any tracker when creating a torrent and not concerning yourself whether it supports BEP52 you just have to assume it will only return peers that have announced themselves under that exact infohash, i.e. there will not be any cross-sharing for the v1 and v2 subsets in a hybrid torrent. This in turn means a client has to announce to both the v1 and v2 infohash on a hybrid torrent.

[X-Coder264]

Ok, I guess that answered even the question that I added afterwards in my previous reply 😄

[arvidn]

@arvidn hybrid torrents are the concern here. In that case you want to announce both to get peer lists for both.

I suggested announcing stats to the v2 hash, but obviously still announcing to v1, just no stats.

[X-Coder264]

I suggested announcing stats to the v2 hash, but obviously still announcing to v1, just no stats.

@arvidn Why would you want to do that? From a tracker perspective that doesn't make any sense. We'd still need to "handle" the announce request so there's no benefit there. It actually only breaks things, as every private tracker atm validates the request before actually sending back any kind of a response to the client (e. g. fields like uploaded, downloaded, left etc. are all required to be in the request, otherwise the client gets a failure reason response back with a proper validation message of what was wrong with the request). If those fields wouldn't be required anymore as you propose to not send them when announcing to the v1 hash, that means that the validation code would have to be changed as well.

btw, I've implemented full BEP 52 support on my tracker and all tests are passing so everything behaves the way I expect it to (the tracker only returns peers that have announced themselves under that exact info_hash back to the client and as far as recording traffic, it expects that the announces from the client are as they are described in my first post above under the b) paragraph (two identical announces with the only difference being the info_hash that's being announced to).

Does anybody know if there is any BEP 52 compliant client out there which I can download to actually test the tracker in a real life scenario?

[arvidn]

was there any resolution on this? I think status quo is to announce every listen socket (e.g. IPv4 and IPv6) and one for each bittorrent version the torrent participates in (e.g. v1 and v2).

Does anybody know if there is any BEP 52 compliant client out there which I can download to actually test the tracker in a real life scenario?

libtorrent master implements bittorrent v2. However, it hasn't been released yet, so you would have to build client_test from source.

[X-Coder264]

was there any resolution on this?
libtorrent master implements bittorrent v2. However, it hasn't been released yet, so you would have to build client_test from source.

I know libtorrent master implements bittorrent v2, but I have never built it as I haven't had the time yet to read how that's done (and it's a low priority for me as I've fully covered the tracker with tests so everything should behave as expected and AFAIK there isn't any popular torrent client out there with v2 support so there is no rush). So I guess I'll wait for something like a qBittorrent master branch build with libtorrent master (which could take some time as AFAIK it isn't compatible yet with libtorrent master).

I think status quo is to announce every listen socket (e.g. IPv4 and IPv6) and one for each bittorrent version the torrent participates in (e.g. v1 and v2).

This sounds like a bunch of announces for each torrent, some trackers with a lot of traffic could literally be hammered.

[arvidn]

This sounds like a bunch of announces for each torrent, some trackers with a lot of traffic could literally be hammered.

If you have both an IPv6 and IPv4 swarm, those would probably be separate trackers (as in, separate processes, possibly running on separate machines).

If you publish hybrid torrents, it's hard to avoid treating v1 and v2 as separate swarms, since they have separate info-hashes.

Do you have any suggestions for improvements?

We could develop a multi-announce protocol, where you can announce multiple torrents per HTTP request. It would be complicated though, and probably not a significant improvement over supporting HTTP keep-alive.

[arvidn]

I also meant to say: I think status quo is option (b). It would be possible to implement (a), with some complexity and some additional space requirements.

[X-Coder264]

If you have both an IPv6 and IPv4 swarm, those would probably be separate trackers (as in, separate processes, possibly running on separate machines).

If both the server and client support BEP 7 then there is no need for separate processes. All new(er) trackers support BEP 7 and since BEP 52 is a lot newer than BEP 7 I don't think it's a realistic case of there being a tracker that supports BEP 52 and not 7.

EDIT: I've just seen #100 :(

If you publish hybrid torrents, it's hard to avoid treating v1 and v2 as separate swarms, since they have separate info-hashes.

That's correct. My tracker does indeed treat them as separate swarms.

Do you have any suggestions for improvements?

We could develop a multi-announce protocol, where you can announce multiple torrents per HTTP request. It would be complicated though, and probably not a significant improvement over supporting HTTP keep-alive.

I don't atm, but I think we should delay implementing such complex solutions until the need for it actually arises in practice. I've just raised my concerns here as a food for thought.

I also meant to say: I think status quo is option (b). It would be possible to implement (a), with some complexity and some additional space requirements.

I agree as I've implemented (b) on my tracker too since it's a lot easier to implement (b) than (a) on the tracker side as well.

[arvidn]

@X-Coder264 do you want to propose a change to BEP 52? To specify option (b)

[X-Coder264]

@arvidn Yes, I do. I'll send a PR later today, unless you want to be the one to do it 😛

[X-Coder264]

@arvidn PR #102 sent.