Give Server a chance to validate a predicted object instantiation

[tedbarth]

There are two ways to spawn a synced NetworkObject (CSP object that use [Replicate] and [Reconcile] methods / NetworkTransform), triggered by an input on the client:

1. Predicted spawn on the client (no delay for client until spawn; not any server-side validation checking if spawn is legal)
2. Client invokes ServerRPC, which then spawns the object server-sided (client has to wait full round-trip until local spawn; full control of server-side validation)

I propose to add a way to allow merging/rollbacking of client-side predicted spawned objects depending on validation done in a server-sided [ServerRpc] method. This would allow to combine CSP (Replicate and Reconcile), client-sided predicted spawning and server validation with the advantages:

• No client-side delay for spawning
• Client-side Prediction with reconciliation
• Validation to encounter desynchronization (caused by lag, packet loss, packet disorder or physic engine nondeterminism) and Cheating

Something like:

1. Client predictably spawns CSP object (with [Replicate] and [Reconcile] and PredictedObject / NetworkTransform) locally while advancing it independently of any server side object
2. Client calls FooServerRpc(objectId), while passing the objectId(s) for validation on server side
3. [Some latency]
4. Server receives FooServerRpc(objectId) call and reads it as the client's wish to spawn an object, validates it and, depending on the result, either spawns the object locally using the passed (predicted) objectId and informs all other clients to do the same or informs the owner to despawn the object, because action is not valid (to give the client a chance to sync again)
5. [Some latency]
6. The client is informed about a newly spawned object with a known objectId and merges the existing one with what comes from the server or is informed about a local object to be removed from scene (all transparently done by FN)

Use cases for validation:

• The client has predictably and wrongfully created an object using an outdated state while the server already has the new state that would not allow a spawning. The client should be informed to remove the stray object to get in sync again.
• The client has predictably and wrongfully created an object using a desynchronized state, e.g. not received state update or physics came out with different results on client and server. The client should be informed to remove the stray object.

Example code:
If in the following example actions of other clients can cause a depletion of the ammo, a desynchronization can occur and the client may fire a missile while the server already knows that the ammo is depleted. The server would block the missile spawn for itself and other clients to fight desynchronization artifacts and cheating. Without the proposed solution the missile would still live on on the client, causing even more desynchronization. Instead, we give the client a chance to get in sync again by telling it to despawn the wrongfully spawned stray missile.

[SyncVar] int ammo = 10;

private void Update()
{
    if (IsOwner) {
      if (Input.GetKeyDown(KeyCode.Space)) {
         if (CanFire()) {
             int objectId = Fire();
             FireServerRpc(objectId);
         }
      }
    }
}

[ServerRpc]
private void FireServerRpc(predictedObjectId)
{
    if (!CanFire()) {
        // Tell the clients or at least the ServerRpc caller to despawn the stray missile with the given ID.
        Despawn(predictedObjectId);
        return;
    }

    Fire(predictedObjectId);

    // Spawn the missile on the network while using the given objectId locally and on the clients. When the
    // spawn command arrives on the ServerRPC-calling client it will bind the server side object to the already
    // existing, predictably spawned object, identified by this objectId, on the client side instead of creating
    // a new object.
    Spawn(missile, predictedObjectId); 
}

private int Fire()
{
    ammo--;
    var missile = Instantiate(
          missileCspPrefab,
          weaponSocket.transform.position,
          transform.rotation);

    NetworkObject nob = missile.GetComponent<NetworkObject>();
    return nob.ObjectId;
}

private bool CanFire()
{
    return ammo > 0;
}

The idea of having a ServerRPC method on the server doing the validation is to have full control over a successful application of the action or a possible rollback, while knowing the exact usecase including all involved objects (actually only their objectIds) that were predictably spawned on the client: One validation and rollback for all at once.

The proposed solution is fully compatible with the projectile lag compensation example of FN, which currently does work with two independently spawned, possibly desynchronizing projectile objects - With no reconciliation support at all - on client and server.

Discussion: https://discord.com/channels/424284635074134018/1034477094731784302/1084178038910103563

[tedbarth]

Just had a thought about it again:
If you in addition to introducing a Despawn(int objectId) will introduce a way to receive a custom validation result from a ServerRpc where the client still knows the context the code is much easier to read and gives the client easier control of what to rollback. If you don't do so, the only way to react (while not know why it happend) is in the OnStopClient() on the client for every object that needs to be rolled back.

So I also propose to introduce asynchronous response values for ServerRpcs in addition to or as replacement for Despawn(int objectId):

private void Update()
{
    if (IsOwner) {
      if (Input.GetKeyDown(KeyCode.Space)) {
         if (CanFire()) {
             int objectId = Fire();
             bool accepted = await FireServerRpc(objectId);
             if (!accepted) {
                 Despawn(objectId); // Could also be replaced by the game's own registry of spawned objects by objectId if run on client
                 RollbackEvenMoreOrSomeStateThatIsntPartOfTheLaunchedGameobject();
             }
         }
      }
    }
}

[ServerRpc]
private Task<bool> FireServerRpcAsync(predictedObjectId)
{
    if (!CanFire()) {
        // Tell the clients or at least the ServerRpc caller to despawn the stray missile with the given ID.
        return false; // We could also return more complex data
    }

    Fire(predictedObjectId);
    ObserversFire(ammo);

    return true;
}

[tedbarth]

I tried to workaround this in my 'fire projectile' example, by

1. Client: Spawning a client projectile (without NetworkObject or any NetworkBehavior)
2. Client: Calling [ServerRpc] fireServerRpc(int projectileInstanceId) on the client
3. [ Latency ]
4. Server (inside of fireServerRpc(...)): Call [TargetRpc] deletePrespawnedProjectileTargetRpc(NetworkConnection c, projectileInstanceId)
5. Server (inside of fireServerRpc(...)): Validate if spawn is possible
6. Server (inside of fireServerRpc(...)): If valid spawn new projectile for all clients, this time with a different prefab with CSP
7. [ Latency ]
8. Client (inside of deletePrespawnedProjectileTargetRpc(...)): Remove projectile identified by given instance ID from scene
9. Client (inside of deletePrespawnedProjectileTargetRpc(...)): Merge just removed prespawned projectile with the new spawned one

That theoretically works but is practically an impossible task:

• You have to create a separate variant of the object you want to prespawn including components that does not have any NetworkBehaviour component and can work like your CSP controller (handling inputs) that does not rely on Network stuff. Reason for this is that the object is not updated in any way (and starts inactive) because it requires to be spawned first.
• It is a hell of a work to merge the state of the prespawned and server-spawned object, including Animations, half played Audio, Position. The latter is really hard, because I need to make the GraphicalObject of the server-spawned CSP projectile smoothly adjust from client-side position.
• You must delete the prespawned object on client side before spawning server side to avoid having two objects simultaneously (including collision problems, etc.), while still holding it to later on merge their state

After thinking about it for too long, I think the best and easiest would be to:

• Let ServerObjects::ReadPredictedSpawn call a validation callback (at latest before WriteResponse(true)) to give the server a possibility to validate/cancel the spawn
• Let ClientObjects::ParsePredictedSpawnResult call a callback to give the client the chance to clean up the predicted mess and get back in sync again

But that in turn would result in having to validate on the server per spawned object instead of per action, which easily becomes hard to validate on server / rollback on client when multiple objects are spawned als a result of an illegal action (e.g. shooting multiple RB projectiles as a triple-shot action). This mainly because you lose the action context.

[FirstGearGames]

Just letting you know I'm aware of this and I've not had a chance to review your comments yet. I will do so by 3.6.1 release though.

[FirstGearGames]

I reviewed your notes but I'm waiting until Prediction v2 before I visit future prediction tasks since version 1 is no longer receiving updates.

[FirstGearGames]

Will this work? #402
This will let the client write data normally, eg: writer.Write("Hello");
and the server can parse and apply it before any of the tryspawn, ect are called.

[tedbarth]

So you send data along with the Spawn command from client. The server has a callback for this and can read/apply the data.

The important missing piece is that the server can block the prespawed object, so that a) the object is not even instantiated and b) the client is informed to remove the prespawned (denied) object to get in sync again.

[FirstGearGames]

Accepted but closing out as 402 will cover this.

[FirstGearGames]

This was resolved via Read/WritePayload for NetworkBehaviours.