Selfplay vs matches, etc

[roy7]

With the new slower networks we're really lagging behind on network matches getting done. I wonder if we should reduce the impact of "fast clients only" or go the other way and increase it further. I also wonder if we need to adjust the # of games buffer size.

Right now server says we've done 171 match games in the past hour and log file shows we have about 38 matches running. This is with stats 28 : 45 (38.36%). There are 5 nets in networks/ that haven't started yet. Current match running was put into the queue 7 hours ago.

Someone like Big Red can't run match games but we are assigning self play games to multiple IP addresses. So we could be assigning more matches presently than we are if that's wise.

[billyswong]

2018-04-10 10:50 | 8c67ecdc  VS  5838db67 | 273 : 220 (55.38%) | 493 / 400 | PASS
...
2018-04-09 14:15 | 1629cbfd  VS  95da93c3 | 325 : 183 (63.98%) | 508 / 400 | PASS

Seeing the 93-games and 108-games surplus, maybe we can prolong the wait before we assume a client is down and request more match games.

[ricksu978]

I would suggest to have autogtp report back how long did it take to generate a self-play/match game on a user's machine.

With this valuable data in hand, we can further analyse and optimize the value for MATCH_EXPIRE_TIME, QUEUE_BUFFER and PESSIMISTIC_RATE for better task allocation.

[marcocalignano]

The suggestion was already made but the heuristic method the server is using seems to be good enough.

[ricksu978]

autogtp does report time spent on each game, however this data is only available on the user's machine (Please correct me if I'm wrong)

My suggestion is to have autogtp submitting the game plus the time spent to the server.

By collecting this data on server side, we'll have better idea of the impact from upgrading the network.

Right now, we only know it'll take longer for a game to be generated, but by how long? we don't know.

[marcocalignano]

Yes like I said this suggestion was already made.

[ricksu978]

thanks @marcocalignano do you have the issue reference? so that I can subscribe and follow the issue.

[roy7]

autogtp added support to send the info to the server but the server doesn't make use of it yet. If nothing else I should probably just store the speed when it arrives and adjust the list of fast clients on the fly so we don't send matches to clients that will take over 30 minutes to finish.

[roy7]

Actually for some reason I thought it had been added but I guess not looking over https://github.com/gcp/leela-zero/blob/next/autogtp/Management.cpp#L703. @marcocalignano When it's convenient sometime could you add passing the game speed data back to the server on submissions?

[marcocalignano]

@roy7 I am running the test autogtp version with the game stats.
With the request of the new task (URL: http://zero.sjeng.org/get-task/15) it sends two new parameters:

• timexgame (in sec)
• myid (integer)

myid is the id of the thread the performed the last game so you can have statistics loggin the IP and this id. This is useful if (like me) a client has two different GPU one faster and one slower.
If you need I can give you my IP.

The first 8 games will send will be stopped and resumed games so the stats will be off. with the 9th game you'll get the real numbers.

[roy7]

Are these stats stored for matches and self play? When pausing a match you can save the number of seconds spent instead of the timestamp of when it started, and then maybe add that to the time spent when the game is resumed.

That way pausing and resuming a match would still have the correct total amount of time spent, although if the match changed threads in the meantime maybe it's better to just not report stats on resumed matches at all.

[marcocalignano]

@roy7 the stats are for both, anything the thread does.