Evaluation

[ChicyChen]

Dear authors,

Could you please provide more details on how you evaluate different benchmarks (CV-Bench, BLINK, RoboSpatial, etc) for different models (Qwen-2.5-VL-7B, SpaceLLaVA, RoboPoint, etc)? I try to reproduce the results in the paper, but find big differences. For example, the results I got for SpaceLLaVA using the official eval codes for RoboSpatial is much lower than what the paper reports. I would really appreciate your help.

Thanks!

[Zhoues]

@ChicyChen

Thank you very much for your attention to our work, especially the results in our experimental tables.

The scores of SpaceLLaVA on RoboSpatial are indeed expected to be very low, as it fundamentally lacks the ability to output points. Therefore, we actually performed a certain degree of fine-tuning on SpaceLLaVA to enable it to output point coordinates (in fact, I'm also curious why in the RoboPoint paper, SpaceLLaVA was able to partially succeed on RoboRefIt and Where2Place).

In other words, I should have added a footnote to SpaceLLaVA in Table 2 in the main paper to clarify that we applied additional fine-tuning to the model. I sincerely apologize for the confusion this may have caused.

For the other benchmarks (e.g., CV-Bench, BLINK), the protocols used are consistent with the official repositories. The usage of RoboPoint also follows the official codebase. As for making Qwen-2.5-VL-7B output point coordinates, it also requires using the prompt provided by the official repository to achieve decent results.