Understanding pruning_steps

[iamanigeeit]

Hello @tianyic ,

I see that pruning_steps determines how many steps we take before pruning:
https://github.com/tianyic/only_train_once/blob/7e930f6ae6cab71659fb921671f6f3921828d7c4/only_train_once/optimizer/hesso.py#L29

1. Is it correct to say that if pruning_periods = 1, then pruning_steps = how many steps before we prune once? If it makes more sense to prune after every epoch, i will set pruning_steps = steps_per_epoch.

2. What does pruning_periods do?

Thanks in advance!

[tianyic]

@iamanigeeit

1. Yes, your understanding is correct. It makes sense to set pruning_steps = steps_per_epoch .
2. pruning_periods controls how many groups to be pruned incrementally. For example, if target_group_sparsity=0.7, pruning_periods=10, pruning_steps=200, then every pruning_steps /pruning_periods =20 steps will yield target_group_sparsity / pruning_periods =0.07 groups as redundant and project them onto zero.