Config for fine-tuning

[laetitialalla]

Hey all,
glad to see so many evolutions and improvements with RasterVision :)

I am using Rastervision 0.13 for car detection on RGB satellite images. I trained a fasterRCNN on xView data (as explained in your tutorial) and would like to fine-tune this model on a different dataset.

I saw in your documentation 2 methods for fine tuning :

• 1. Using model-bundle
     As explained here in your doc : "Using model-bundles for fine-tuning : We can also re-construct the Learner in order to continue training, perhaps on a different dataset" by adding source_bundle_uri to the final ObjectDetectionConfig :

ObjectDetectionConfig(
        root_uri=root_uri,
	[...]
	source_bundle_uri= base_uri + "model-bundle.zip")

• 2. Using the weights directly
     As explained here in your doc : "The last-model.pth model file in the train directory can be used as a pretrained model in a new pipeline" by adding init_weights to the ModelConfig :

model = ObjectDetectionModelConfig(backbone=Backbone.resnet50, init_weights=base_uri + "last-model.pth")

My questions are :

• Are these 2 methods strictly equivalent ? I'm not sure what exactly happens with method 1.
• Based on a comment in your code here, it seems important to provide the bundle saved in train/model-bundle.zip and not in bundle/model-bundle.zip. Is there a difference between those 2 bundles except for the config.json files ? Are weights the same ?
• Similarly, in the folder "train", is there a difference between the weights in last-model.pth and the weights in model.pth inside the model-bundle.zip ?
  Could it be that the weights are saved in model.pth at each epoch, and last-model.pth is only the final weigts once the training is complete ? What happens if we planned 40 epochs but want to stop the training after 20 (if, for example, the loss start increasing) ? Is it possible to find the weigts somewhere, even if the training is not complete ?

Thanks a lot for your help !

Laetitia

[AdeelH]

Hello again!

You are right that there currently are multiple ways of doing this.

My questions are :

• Are these 2 methods strictly equivalent ? I'm not sure what exactly happens with method 1.

They should be equivalent, yes. Though, to be very honest, I am not very happy with 1 (the source_bundle_uri thing) and would like to remove it in the future.

• Based on a comment in your code here, it seems important to provide the bundle saved in train/model-bundle.zip and not in bundle/model-bundle.zip. Is there a difference between those 2 bundles except for the config.json files ? Are weights the same ?

The weights are the same. Technically, the config in the one in bundle/ is an RVPipelineConfig subclass, while the other one is a LearnerPipelineConfig.

• Similarly, in the folder "train", is there a difference between the weights in last-model.pth and the weights in model.pth inside the model-bundle.zip ?

They are the same.

Could it be that the weights are saved in model.pth at each epoch, and last-model.pth is only the final weigts once the training is complete ?

The weights are saved after every epoch to last-model.pth (i.e. the file is overwritten after every epoch). Once the training is complete, and the bundle is being created, the last-model.pth file is copied to the bundle as model.pth.

What happens if we planned 40 epochs but want to stop the training after 20 (if, for example, the loss start increasing) ? Is it possible to find the weigts somewhere, even if the training is not complete ?

I'm assuming you mean that you manually kill the job in the middle of training. In that case, the weights from the last completed epoch will be in last-model.pth.

Hope this helps. Please feel free to ask more questions.

P.S. I would strongly recommend upgrading to Raster Vision v0.20.

[laetitialalla]

Hi @AdeelH,

thanks for your answer ! I was able to make my code more straightforward =)

But now I'm wondering why last-model.pth is overwritten after every epoch. Wouldn't it be convenient to keep the weights at each epoch, in order to select the best model at the end of the training ?

For exemple, I had a training that looked this this :


At epoch 30, the map50 starts decreasing... So it seems I'm over-training. Hence, I would like to keep the weights from epoch30, but right now it's not possible since only the last-model.pth is available :) Is there a reason for that ?

Also, I'm wondering why the val_loss is not accessible for Object Detection, only the train_loss is returned. If I'm not mistaken, the function validate_end in Learners definition returns :

• metrics (map and map50) for object_detection_learner
• metrics (conf_mat_matrix) AND val_loss for semantic_segmentation_learner and classifictaion_learner
  I'm wondering why the val_loss is not relevant for Object Detection ?

Thanks a lot for you help @AdeelH !

PS : You convinced me ! I am upgrading to Raster Vision v0.20, but I have an issue... I guess I will open another discussion.

[AdeelH]

Also, I'm wondering why the val_loss is not accessible for Object Detection, only the train_loss is returned.

This is a quirk of TorchVision's object detection models. During training, they return only losses, and during inference, only predictions (boxes, labels, scores).

But now I'm wondering why last-model.pth is overwritten after every epoch. Wouldn't it be convenient to keep the weights at each epoch, in order to select the best model at the end of the training ?

Checkpointing after every epoch or every few epochs would indeed be a useful feature to have. I think RV used to have this functionality some versions ago, but it was removed for some reason. Please feel free to create a feature request for this!

(As an aside, one should be careful about how much stock to put in the model's performance on the validation set. If your validation set is too small or not representative of the test set or the data in production, then you run the risk of overfitting the validation set.)

PS : You convinced me ! I am upgrading to Raster Vision v0.20,

🚀

[laetitialalla]

Feature request is done :)

I had not realized it came form TorchVision's fasterRCNN... Thanks !