question about how to finetune a modified model

[4vicii]

thx for ur repo

I am working with the Llama 3.2 Vision model, which is a multimodal large model consisting of an image_encoder and a causal model, similar to the LLaVA model structure. I plan to freeze the Llama 3.2 Vision model and add an additional causal model to encode the image tokens extracted by the image_encoder, and train this additional causal model.

how do i train this modified model by ur repo?