I think I do once code a LLaVA Trainer with LMMs-Eval. Here is the original code The basic idea is that using lmms-eval to build request and then use trl's unwrap for generation to ask model to ...