Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

PDF | | Poster | Slides

Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu

• Fine-tuning LLMs for specific tasks often encounters challenges in balancing performance and preserving general instruction-following abilities. In this work, we posit that the distribution gap between task datasets and the LLMs serves as the primary underlying cause. To address the problem, we introduce Self-Distillation Fine-Tuning to bridge the distribution gap by guiding fine-tuning with a distilled dataset generated by the model itself.

Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework

PDF | | Poster | Slides | Project Page

Zhaorui Yang*, Bo Pan*, Han Wang*, Yiyao Wang, Xingyu Liu, Luoxuan Weng, Yingchaojie Feng, Haozhe Feng, Minfeng Zhu, Bo Zhang, Wei Chen

• Existing deep research works primarily focus on generating text-only content, leaving the automated generation of interleaved texts and visualizations underexplored. In this work, we propose a structured textual representation for visualizations, and introduce an agentic framework that automatically generates comprehensive multimodal reports from scratch with interleaved texts and visualizations.

CoSDA: Continual Source-Free Domain Adaptation

PDF |

Haozhe Feng*, Zhaorui Yang*, Hesun Chen*, Tianyu Pang, Chao Du, Minfeng Zhu, Wei Chen, Shuicheng Yan

• In this work, we investigate the mechanism of catastrophic forgetting of previous Source-Free Domain Adaptation (SFDA) approaches. We observe that there is a trade-off between adaptation gain and forgetting loss. Motivated by the findings, we propose CoSDA, which outperforms SOTA approaches in continuous adaptation.