Request for Assistance with Replication

[WentaoTan]

Hello! Our team has been working on reproducing SkyThought, which we find to be groundbreaking and insightful. We have successfully replicated the results using the Qwen2.5-32B model based on the llama-factory code, achieving significant performance improvements.

However, we encountered some challenges when attempting the same training with the llama3.3-70B model. Contrary to our expectations, we did not observe a notable performance boost, and there was even a slight decline in performance on the math500 benchmark.

We greatly appreciate any guidance or insights. Thank you!

[hxdtest]

Have you tested eval scripts? issue I can't reproduce Qwen/QwQ-32B-Preview accuracy on AIME with eval scripts? Can you reproduce Qwen/QwQ-32B-Preview accuracy on AIME with eval scripts?

[WentaoTan]

We tried to replicate the performance of QwQ but failed because our resources could not support very long output and QwQ output was very long.

[richardliaw]

Hi @WentaoTan, we're currently working on more extensive experiments on different model sizes and architectures, and have not tested on 70b yet.

Can you share some of the numbers you've gotten so far?