SoccerNet models

[yossibiton]

Thanks for the great work and code.
Are you going to publish the SoccerNet models as well (or at least the training parameters you used for your papers) ?

[jhong93]

Hi, thank you for the question! All of the training parameters are in the supplemental material PDF (available on the website).

We will consider uploading weights, but we initially refrained from doing so to avoid becoming another feature that gets concatenated with the other commonly used features. The scripts to pre-process the SoccerNet dataset are in the repo.

[zachpvin]

Hey @jhong93, I'm too interested in the trained SoccerNet model if you don't mind. I trained according to the parameters mentioned in the paper but can only reach around 57 mAP for the loose and 41 mAP for the tight metric for the 200MF regnet model. I wonder what I did wrong. I hope you don't mind that I've personally emailed you @ cs.standford.edu to request for the trained weight if you think that it's not inconvenience for you! Thank you for open-sourcing the repo btw!

[jhong93]

Hi, I've updated the repository with a few of the SoccerNet models; hopefully the config information is helpful.

Otherwise, it could be a dataset pre-processing issue. I would hope that print_dataset_stats.py matches up at least as a weak sanity check.

=== soccernetv2 ===
Categories: 17
Train:
        orig videos: 600
        videos: 600
        events: 66460
        frames: 3439393
        events / frames (%): 1.93
        min frame (of first event): 0
        max frame (of last event): -21
        Label counts:
                Ball out of play : 19097
                Clearance : 4749
                Corner : 2884
                Direct free-kick : 1379
                Foul : 7084
                Goal : 995
                Indirect free-kick : 6331
                Kick-off : 1516
                Offside : 1265
                Penalty : 96
                Red card : 34
                Shots off target : 3214
                Shots on target : 3463
                Substitution : 1700
                Throw-in : 11391
                Yellow card : 1238
                Yellow->red card : 24
Val:
        orig videos: 200
        videos: 200
        events: 21447
        frames: 1144064
        events / frames (%): 1.87
        min frame (of first event): 0
        max frame (of last event): -50
        Label counts:
                Ball out of play : 6253
                Clearance : 1516
                Corner : 953
                Direct free-kick : 439
                Foul : 2176
                Goal : 371
                Indirect free-kick : 1907
                Kick-off : 536
                Offside : 417
                Penalty : 36
                Red card : 13
                Shots off target : 984
                Shots on target : 1182
                Substitution : 560
                Throw-in : 3718
                Yellow card : 378
                Yellow->red card : 8
Test:
        orig videos: 200
        videos: 200
        events: 22551
        frames: 1150191
        events / frames (%): 1.96
        min frame (of first event): 0
        max frame (of last event): 0
        Label counts:
                Ball out of play : 6460
                Clearance : 1631
                Corner : 999
                Direct free-kick : 382
                Foul : 2414
                Goal : 337
                Indirect free-kick : 2283
                Kick-off : 514
                Offside : 416
                Penalty : 41
                Red card : 8
                Shots off target : 1058
                Shots on target : 1175
                Substitution : 579
                Throw-in : 3809
                Yellow card : 431
                Yellow->red card : 14
Overall:
        has train/test orig video overlap: False
        num frames: 5733648
        num events: 110458
        event %: 1.9264872904649883

[zachpvin]

Many thanks for the weights, it's super helpful in diagnosing the issue!

[phuc16102001]

Hi @jhong93, currently I am reproducing your model using the SoccerNetV2 dataset.

However, I did not receive the result as the paper report. Particular, for the test set, I use the model soccer_rny008gsm_gru_rgb and the result (w/o NMS) is:

{
  "test_split": {
    "Average-mAP (loose)": 51.710384220920616,
    "Shown only (loose)": 57.345404094772704,
    "Unshown only (loose)": 22.07061351534247,
    "Average-mAP (tight)": 45.13071156736224,
    "Shown only (tight)": 50.76184780989905,
    "Unshown only (tight)": 16.12071242855153
  }
}

Also, the nearly same result has been shown for the challenge split with model soccer_challenge_rny008gsm_gru_rgb:

{
  "challenge_split": {
    "Average-mAP (loose)": 48.224891761119245,
    "Shown only (loose)": 54.50310379766974,
    "Unshown only (loose)": 37.3107990125073,
    "Average-mAP (tight)": 45.45153077812651,
    "Shown only (tight)": 52.18741683411739,
    "Unshown only (tight)": 33.10071957727419
  }
}

The inference phase is run on the frame extracted from the 224p video.

Can you help me diagnose the problem? Are there any different settings (from default setting) in the frame-extracting phase? Moreover, in the prediction file, I see that the FPS is "2.0833333333333335" so is that correct or it must be exactly "2"?

Thank you!

[phuc16102001]

Thanks a lot for your help @jhong93, I have found that the problem is that we must use the "prediction.recall.json" file instead of "prediction.json".