Hi @John6666 @BenjaminB, Thank you so much for your amazing feedback! I appreciate it! The previous topic closed and I couldn’t add to it so I created this new one. By trial and error, I’m now using the following filter and it stopped the crash. I’m not sure if it’s working 100% for the fine-tuning. I’m still checking. Training Loss is still high. I don’t know if impacted by the filter or other factors e.g. dtype=torch.bfloat16 conversion … target_modules=r"model\\.language_model\\..*\\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)",

Hi, I think it has something to do with the caching of generated results. This problem only arises with Gemma models, from what I experienced. I had a similar problem a month ago. I was able to fix it by deploying the fix by @BenjaminB github.com/huggingface/peft #### FIX Prompt learning issue with 4d attention mask `main` ← `BenjaminBossan:fix-prompt-tuning-4d-attention-mask` opened 11:30AM - 27 Mar 25 UTC BenjaminBossan +100 -12 Resolves #2452 Some causal language models in transformers have 4d attention …masks at the input preparation stage. So far, we have assumed 2d attention masks, which results in an error in that case. This PR fixes the situation. My first attempt was to transform the 2d prefix attention mask (from the virtual tokens) into a 4d attention mask before concatenating them. However, this was error prone and I was unsure if my approach would generalize to other model architectures than the one tested (gemma), as it involved using private transformers methods (`model._prepare_4d_causal_attention_mask_with_cache_position`). The simpler approach was thus to just create a 2d attention mask and let the model handle it. The test suite has been extended to include a tiny gemma model. To prevent the test suite from ballooning, I removed another model. Specifically, this was GPT neox, which from HF download stats seems to be one of the least popular architectures from our test suite. I also extended the default parameters in `constants.py` for the different PEFT methods to support gemma. Unfortunately, some tests are failing with gemma. When they were unrelated to changes in this PR, I chose to just skip those tests, as I consider them out of scope for this PR. It seems that it will be fixed in release 0.15.3. From now on, you can install the package from the main branch. python -m pip install -U git+https://github.com/huggingface/peft.git@main

As BenjaminB also mentioned, it seems unusual that this model size and training parameters require this amount of RAM. Therefore, there may be issues with the library or model, and the cause may lie in parts other than the visible parameters. However, if this were a common issue, there would likely be many reports of it. It may be specific to certain conditions, such as a multi-GPU environment, a specific version of the library, or specific hardware. GitHub ### Fine tuning GPU memory requirements · ggml-org llama.cpp · Discussion #2904 This question is more focused on on full fine tune memory requirements rather than low memory / efficient inference, but I'm hoping it'll be relevant / helpful to community members here especially ...