Evaluation on Emu Edit benchmark

[Eureka-Maggie]

Hi,

Noticed that there are known issues with the Emu edit benchmark: some image-caption pairs seem incorrect (e.g., 'a train station in city') or identical source and target captions. So I was wondering how to calculate clip_T metric. How did you process the benchmark dataset?

Looking forward to your reply.

[staoxiao]

@Eureka-Maggie, we just use the original caption without any modification.

[Eureka-Maggie]

Thank you for your reply. So there are some unpaired output caption and generated images when calculating clip-t metrics.

[staoxiao]

Yes, there is some noise in emu-edit.

[zc1023]

Hello, I am reproducing the results on the emu dataset. But the result I measured is much higher than yours. Can you provide the specific details? For example, seed, resolution, and clip models.

[staoxiao]

@zc1023 , the clip model is openai/clip-vit-large-patch14 and resolution is 512.