This is a very rare error, but it may just be that there is no metadata. huggingface.co ### SeaLLMs/SeaLLM-7B-Hybrid · Seems like metadata is not in the safetensors files Running AutoModel.from_pretrained("SeaLLMs/SeaLLM-7B-Hybrid") gets the following error messages: github.com/ml-explore/mlx #### [BUG] Saved safetensors are missing metadata format pt and cannot be loaded through `transformers` library opened 01:37PM - 26 Feb 24 UTC closed 11:18PM - 26 Feb 24 UTC alexweberk enhancement **Issue description** When uploading safetensors files as part of the `mlx_lm.f…use` step, all the weights files with `.safetensors` extensions are missing the optional metadata for format attribute. As a result, the uploaded weights cannot be loaded when used by `transformers` library users. (`mlx` loads them without a problem.) **To Reproduce** Run LoRA fine-tuning, then run fusing script: ```bash !python -m mlx_lm.fuse \ --model google/gemma-7b-it \ --adapter-file checkpoints/600_adapters.npz \ --upload-repo alexweberk/gemma-7b-it-trismegistus \ --hf-path google/gemma-7b-it ``` After the upload, I tried running: ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "alexweberk/gemma-7b-it-trismegistus" tokenizer = AutoTokenizer.from_pretrained(repo_id) model = AutoModelForCausalLM.from_pretrained(repo_id) model.to("mps") input_text = format_prompt(system_prompt, question) input_ids = tokenizer(input_text, return_tensors="pt").to("mps") outputs = model.generate( **input_ids, max_new_tokens=256, ) print(tokenizer.decode(outputs0])) ``` Which gives the full error message below: ``` --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In14], line 7 4 repo_id = "alexweberk/gemma-7b-it-trismegistus" 6 tokenizer = AutoTokenizer.from_pretrained(repo_id) ----> 7 model = AutoModelForCausalLM.from_pretrained(repo_id) 8 model.to('mps') 10 input_text = format_prompt(system_prompt, question) File ~/miniforge3/envs/py311/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:561, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 559 elif type(config) in cls._model_mapping.keys(): 560 model_class = _get_model_class(config, cls._model_mapping) --> 561 return model_class.from_pretrained( 562 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs 563 ) 564 raise ValueError( 565 f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n" 566 f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}." 567 ) File ~/miniforge3/envs/py311/lib/python3.11/site-packages/transformers/modeling_utils.py:3502, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs) 3493 if dtype_orig is not None: 3494 torch.set_default_dtype(dtype_orig) 3495 ( 3496 model, 3497 missing_keys, 3498 unexpected_keys, 3499 mismatched_keys, 3500 offload_index, 3501 error_msgs, -> 3502 ) = cls._load_pretrained_model( 3503 model, 3504 state_dict, 3505 loaded_state_dict_keys, # XXX: rename? 3506 resolved_archive_file, 3507 pretrained_model_name_or_path, 3508 ignore_mismatched_sizes=ignore_mismatched_sizes, 3509 sharded_metadata=sharded_metadata, 3510 _fast_init=_fast_init, 3511 low_cpu_mem_usage=low_cpu_mem_usage, 3512 device_map=device_map, 3513 offload_folder=offload_folder, 3514 offload_state_dict=offload_state_dict, 3515 dtype=torch_dtype, 3516 hf_quantizer=hf_quantizer, 3517 keep_in_fp32_modules=keep_in_fp32_modules, 3518 ) 3520 # make sure token embedding weights are still tied if needed 3521 model.tie_weights() File ~/miniforge3/envs/py311/lib/python3.11/site-packages/transformers/modeling_utils.py:3903, in PreTrainedModel._load_pretrained_model(cls, model, state_dict, loaded_keys, resolved_archive_file, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, _fast_init, low_cpu_mem_usage, device_map, offload_folder, offload_state_dict, dtype, hf_quantizer, keep_in_fp32_modules) 3901 if shard_file in disk_only_shard_files: 3902 continue -> 3903 state_dict = load_state_dict(shard_file) 3905 # Mistmatched keys contains tuples key/shape1/shape2 of weights in the checkpoint that have a shape not 3906 # matching the weights in the model. 3907 mismatched_keys += _find_mismatched_keys( 3908 state_dict, 3909 model_state_dict, (...) 3913 ignore_mismatched_sizes, 3914 ) File ~/miniforge3/envs/py311/lib/python3.11/site-packages/transformers/modeling_utils.py:507, in load_state_dict(checkpoint_file) 505 with safe_open(checkpoint_file, framework="pt") as f: 506 metadata = f.metadata() --> 507 if metadata.get("format") not in "pt", "tf", "flax"]: 508 raise OSError( 509 f"The safetensors archive passed at {checkpoint_file} does not contain the valid metadata. Make sure " [510 "you save your model with the `save_pretrained` method." [511 ) [512 return safe_load_file(checkpoint_file) AttributeError: 'NoneType' object has no attribute 'get' ``` The error seems to stem from the safetensors files missing the metadata for {"format": "pt"} when they are loaded by `AutoModelForCausalLM.from_pretrained()`. A quick work around was to separately resave the files one by one using the below script for each of the safetensors files, and then uploading them to Huggingface. ``` from safetensors import safe_open from safetensors.torch import save_file safetensor_path = "lora_fused_model/model-00001-of-00004.safetensors" # ... fname, ext = safetensor_path.split("/")[-1].split(".") tensors = dict() with safe_open(safetensor_path, framework="pt", device="cpu") as f: for key in f.keys(): tensors[key] = f.get_tensor(key) save_file(tensors, f"lora_fused_model/{fname}-with-format.{ext}", metadata={"format": "pt"}) ``` However, it would be nice to be able to quickly upload and have the model available for a wider audience more easily. The source code led me to `mx.save_safetensors()` which led me to file the issue on this repo. https://github.com/ml-explore/mlx-examples/blob/47dd6bd17f3cc7ef95672ea16e443e58ce5eb1bf/llms/mlx_lm/utils.py#L479 **Expected behavior** Since there are many `transformers` users in the ecosystem, it would be beneficial to be able to seamlessly train and upload model weights to Huggingface and have other users use them through `transformers`. **Desktop (please complete the following information):** - OS Version: [e.g. MacOS 14.3] - MacBook Pro M3 Max 128GB - mlx==0.4.0 - mlx-lm==0.0.13 - transformers==4.38.1

The fact that it’s the same with both PPO and DPO means that, although I don’t know the reason, I think the model weights are probably not being overwritten. For example, `requires_grad=False` may be set. `traineble=True` may also be necessary. github.com/huggingface/peft #### PeftModel is_trainable=True causes generate output to be garbage. opened 12:49PM - 21 Mar 24 UTC closed 03:03PM - 08 May 24 UTC o1lo01ol1o ### TL:DR When loading a pretrained base model and then subsequently loading …a trained PeftModel adaptor on `mistralai/Mistral-7B-Instruct-v0.2`, `model.generate()` behaves as expected. However, if the `PeftModel` has is_trainable set to `True`, the output is garbage. Example: ```python base_model = "mistralai/Mistral-7B-Instruct-v0.2" model = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16) peft_dir = "./foo" device = "cuda" model = PeftModel.from_pretrained(model, peft_dir, is_trainable=True).to(device) inputs = tokenizer.encode("[INST]Sing a nice song.[/INST]", return_tensors="pt").to(device) single_output = model.generate(inputs, max_length=5500, do_sample=True) print(tokenizer.decode(single_output[0])) ``` ```console ID' is isnstyle ;ement L, all Emb be, ", encoded....ified, :,get " A,mall >ves Gget,itia +x ' Revals.,! -- and,fc.' 2; A contract The{ thethe.PL I .; type [ I it are; =>;as........ Ch * <'. , yourli ' my a: all Consult:,, N;ly.olid, Al'ep|;s aest is&,x ' and2 =id-- Itly{: Agreement''Contract ,, nt All; fontia I N'ies Cap FAs many{ the new ,lyown A friend The face, I., I a,w=the Not;:um Ald --, & =xt-- ; ' a and;.0foria {'ia_msI forS .,, spliting want Develop work (, - (c theie1fssart --ics This -sF:: ". _--, $,(1;,icper- " " I 1, I- ``` I expected to be able to generate (or, at a minimum, something like generate) during a training procedure. See below. ### More Context I've implemented (I think) a simplified greedy search as part of a training procedure by following the `_greedy_search` implementation in the `GenerationMixin` class. I'm not completely surprised that there is an issue with calling `generate` since the `@torch.nograd()` annotation annotates the `generate` function. However, I would be surprised if there were a fundamental reason a greedy search could not be performed during training. My implementation, like `generate(),` however, produces garbage during training and I need to know why. My implementation follows: ```python def simplified_greedy_search(model, tokenizer, input_ids, max_length, debug:bool = True): # Initialize variables generated_ids = [] logits_sequence = [] eos_token_id = tokenizer.eos_token_id model_kwargs = {} if debug: # debugging logs show that .generate produces the same type of garbage as this function. test_input = tokenizer.decode(input_ids[0]) with open("simplified_greedy_search_input.log", "a") as f: f.write(test_input + "\n") # use model.generate to generate the sequence test = model.generate(input_ids, max_length=max_length, do_sample=True) test_tokens = tokenizer.decode(test[0]) # append this to a log file with open("simplified_greedy_search.log", "a") as f: f.write(extract_content(test_tokens) + "\n") # Generate tokens until max_length or EOS token is reached while len(generated_ids) < max_length: # Prepare inputs model_inputs = model.prepare_inputs_for_generation(input_ids, **model_kwargs) # Forward pass outputs = model(**model_inputs, return_dict=True) # Get the last token logits next_token_logits = outputs.logits[:, -1, :] # Store the logits logits_sequence.append(next_token_logits) # Get the most probable token next_token_id = torch.argmax(next_token_logits, dim=-1) # Check if EOS token is generated if next_token_id.item() == eos_token_id: break # Add the generated token to the sequence generated_ids.append(next_token_id.item()) # Update the input_ids input_ids = torch.cat([input_ids, next_token_id.unsqueeze(0)], dim=-1) # Update the model_kwargs for the next iteration model_kwargs = model._update_model_kwargs_for_generation( outputs, model_kwargs, is_encoder_decoder=model.config.is_encoder_decoder ) # concatentate the generated ids into a tensor generated_ids = torch.tensor(generated_ids).to(input_ids.device).reshape((1,-1)) # concatenate the logits into a tensor logits_sequence = torch.stack(logits_sequence, dim=1).to(input_ids.device) return generated_ids, logits_sequence ``` ### Version info Name: trl Version: 0.7.11 Name: peft Version: 0.9.0 Name: transformers Version: 4.38.2 Name: torch Version: 2.2.1 ### Who can help? @pacman100 @younesbelkada @sayakpaul ### Information - [X] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder - [X] My own task or dataset (give details below) ### Reproduction ```python base_model = "mistralai/Mistral-7B-Instruct-v0.2" model = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16) peft_dir = "./foo" device = "cuda" model = PeftModel.from_pretrained(model, peft_dir, is_trainable=True).to(device) inputs = tokenizer.encode("[INST]Sing a nice song.[/INST]", return_tensors="pt").to(device) single_output = model.generate(inputs, max_length=5500, do_sample=True) print(tokenizer.decode(single_output[0])) ``` ### Expected behavior I expect that generate does not generate garbage on trainable models. PPO Training does not improve SFT model outputs (Metrics identical before and after PPO) 🤗Transformers > Hi All, I’m currently facing an issue where applying PPO training using trl==0.11.3 on top of my SFT model seems to make no difference at all in the final evaluation metrics. Here’s a snapshot of the comparison before and after PPO: === Summary Metrics === | model | exact_match | rouge1_f1 | rouge2_f1 | rougeL_f1 | bleu | meteor | inference_time_sec | |---------------------------|-------------|-----------|-----------|-----------|----------|---------|-------------------… model = PeftModel.from_pretrained(model, peft_dir, is_trainable=True).to(device)

Hello everyone, I’m running into a puzzling situation where my SFT and DPO evaluations produce **exactly the same** n-gram metrics—even after fine-tuning via DPO. I expected DPO to alter the model’s behavior (and thus change BLEU/ROUGE/etc.), but instead both runs yield: model | exact_match | rouge1_f1 | rouge2_f1 | rougeL_f1 | bleu | meteor | inference_time_s ---|---|---|---|---|---|---|--- SeaLLMs-v3-7B | 0 | 0.715663 | 0.652622 | 0.709211 | 0.558454 | 0.732766 | ~58s **(DPO)** | 0 | 0.715663 | 0.652622 | 0.709211 | 0.558454 | 0.732766 | ~60s * * * ### 1. My workflow 1. **SFT training** via TRL’s `SFTTrainer` * QLoRA (r=16, α=32, dropout=0.05), bf16, 3 epochs * Saved adapter in `sft_output_SeaLLMs-v3-7B/` 2. **Preference dataset creation** (pairwise “chosen vs rejected”) → cleaned JSONL 3. **DPO training** via TRL’s `DPOTrainer` base_model.config.use_cache = False base_model.enable_input_require_grads() base_model.gradient_checkpointing_enable() model = PeftModel.from_pretrained(base_model, sft_output_dir, ...) trainer = DPOTrainer(model=model, args=dpo_args, train_dataset=..., processing_class=tokenizer) trainer.train() model.save_pretrained("dpo_output_SeaLLMs-v3-7B/") 4. **Evaluation Notebooks** * **SFT_Evaluation.ipynb** loads `PeftModel.from_pretrained("sft_output_…")` * **DPO_Evaluation.ipynb** loads `PeftModel.from_pretrained("dpo_output_…")` * Both run 4-bit quantized inference (`BitsAndBytesConfig`), batch-generate, then compute EM / ROUGE-1/2/L / BLEU / METEOR on the _same_ held-out test set. * * * ### 2. Environment * Transformers 4.40.0 * TRL 0.11.3 * PEFT 0.15.0 * bitsandbytes (4-bit NF4 quant) * Python 3.10 * Evaluate library from * GPU: A100 (4-bit inference on GPUs 3,4,5) * * * ### 3. Questions 1. **Why are the SFT & DPO metrics identical?** Is there a scenario where DPO doesn’t actually modify the n-gram outputs, or am I accidentally evaluating the same checkpoint twice? 2. **Adapter loading sanity** * Should I be calling `model.merge_and_unload()` before eval? * Any quick tricks to diff the state-dict of the SFT vs DPO adapter? 3. **Debugging DPO updates** How can I inspect reward/loss signals or gradient norms during DPO training to confirm that the policy is truly being updated? 4. **Best practices for “before vs after” sampling** Do you recommend any lightweight workflow/snippet for sampling a few prompts pre- and post-DPO to spot qualitative changes? I’d really appreciate any pointers, example snippets, or pitfalls to watch out for. Thank you!

Hi all, I’m encountering a critical issue when running `DPOTrainer` on a multi-GPU A100 server (8x A100 40GB) using `trl==0.17.0` and `transformers==4.51.3`. The training fails on all ranks with the same `RuntimeError` during the `.backward()` call in FP16 mode. * * * ### Setup Summary * **Base model** : `SeaLLMs-v3-7B`, loaded in 4-bit using `BitsAndBytesConfig` * **LoRA adapter** : loaded from `../sft/sft_output_SeaLLMs-v3-7B` * **DPOTrainer config** : `fp16=True`, `disable_dropout=True` * **Device setup** : 8 GPUs (`CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`) * **Launcher** : `accelerate launch --mixed_precision="fp16" train_dpo.py` * **Library Used:** transformers==4.51.3 trl==0.17.0 peft==0.15.2 accelerate==1.7.0 torch==2.7.0+cu118 bitsandbytes==0.45.5 datasets==3.6.0 * * * ### The Error Once training begins, all ranks fail with this same error: 0%| | 0/120 [00:00<?, ?it/s][rank3]: Traceback (most recent call last): [rank3]: File "/raid/home/llmsosmed/test-amriz/TA/dpo/train_dpo.py", line 106, in <module> [rank3]: trainer.train() [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train [rank3]: return inner_training_loop( [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/transformers/trainer.py", line 2560, in _inner_training_loop [rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch) [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/transformers/trainer.py", line 3782, in training_step [rank3]: self.accelerator.backward(loss, **kwargs) [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/accelerate/accelerator.py", line 2469, in backward [rank3]: self.scaler.scale(loss).backward(**kwargs) [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward [rank3]: torch.autograd.backward( [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/torch/autograd/__init__.py", line 353, in backward [rank3]: _engine_run_backward( [rank3]: File "/raid/home/llmsosmed/rlaif/lib/python3.10/site-packages/torch/autograd/graph.py", line 824, in _engine_run_backward [rank3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass [rank3]: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn ### Minimal Code Snippet from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig from peft import PeftModel from trl import DPOTrainer, DPOConfig from datasets import load_dataset import torch, os from accelerate import PartialState # Setup os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7" model_name = "SeaLLMs-v3-7B" sft_output_dir = f"../sft/sft_output_{model_name}" base_cache_dir = f"../model_cache/{model_name}" output_dir = f"dpo_output_{model_name}" preference_dataset = f"dpo_preference_dataset_{model_name}_clean.jsonl" # Tokenizer tokenizer = AutoTokenizer.from_pretrained(sft_output_dir, local_files_only=True) tokenizer.pad_token = tokenizer.eos_token tokenizer.padding_side = "left" # 4-bit quant config quant_cfg = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True, ) # Base model device_string = PartialState().process_index base_model = AutoModelForCausalLM.from_pretrained( base_cache_dir, device_map={"": device_string}, quantization_config=quant_cfg, local_files_only=True, ) base_model.config.use_cache = False # LoRA adapter model = PeftModel.from_pretrained( base_model, sft_output_dir, torch_dtype=torch.float16, device_map={"": device_string}, ) model.eval() # Dataset train_dataset = load_dataset("json", data_files={"train": preference_dataset}, split="train") # DPO Config dpo_args = DPOConfig( output_dir=output_dir, num_train_epochs=3.0, per_device_train_batch_size=4, gradient_accumulation_steps=1, learning_rate=1e-6, logging_steps=100, save_steps=500, fp16=True, save_safetensors=True, disable_dropout=True, ) # Trainer trainer = DPOTrainer( model=model, args=dpo_args, train_dataset=train_dataset, processing_class=tokenizer, ) trainer.train() ### Dataset Format (Preference JSONL) { "prompt": "...\\nQuestion: {question}\\nAnswer:", "chosen": "...", "rejected": "..." } ... Where: * `"prompt"` contains the instruction, context, and question. * `"chosen"` is the preferred model response. * `"rejected"` is the less preferred alternative response. All examples follow this pairwise preference format for DPO training. ### Any guidance? Any insight into why this might be happening (especially in the backward pass with LoRA + 4bit + DPO) would be really appreciated. Thank you in advance!