Meta is upgrading its AI chatbot with calendar integration, guided deep research, and daily briefings — positioning it as a direct rival to ChatGPT, Gemini, and Claude. The update is powered by Meta's new Llama Spark 1.1 model and signals the company's broader push toward what Zuckerberg calls 'pers

OpenAI scientist Shengjia Zhao, co-creator of ChatGPT and GPT-4, joins Meta as Superintelligence chief. He will lead Meta’s AGI efforts, including Llama models and next-gen AI research.

For Llama 3.2 model, they don’t seem to be equivalent. get_peft_model and model.add_adapter produces different class structures. See below. model.add_adapter(LoraConfig) gives the following class LlamaForCausalLM( (model): LlamaModel( (embed_tokens): Embedding(128256, 2048) (layers): ModuleList( (0-15): 16 x LlamaDecoderLayer( (self_attn): LlamaFlashAttention2( (q_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (k_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=512, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=512, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (v_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=512, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=512, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (o_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (rotary_emb): LlamaRotaryEmbedding() ) (mlp): LlamaMLP( (gate_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (up_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (down_proj): lora.Linear( (base_layer): Linear(in_features=8192, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=8192, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (act_fn): SiLU() ) (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05) (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05) ) ) (norm): LlamaRMSNorm((2048,), eps=1e-05) (rotary_emb): LlamaRotaryEmbedding() ) (lm_head): Linear(in_features=2048, out_features=128256, bias=False) whereas using get_peft_model produces the following class: PeftModelForCausalLM( (base_model): LoraModel( (model): LlamaForCausalLM( (model): LlamaModel( (embed_tokens): Embedding(128256, 2048) (layers): ModuleList( (0-15): 16 x LlamaDecoderLayer( (self_attn): LlamaFlashAttention2( (q_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (k_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=512, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=512, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (v_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=512, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=512, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (o_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (rotary_emb): LlamaRotaryEmbedding() ) (mlp): LlamaMLP( (gate_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (up_proj): lora.Linear( (base_layer): Linear(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (down_proj): lora.Linear( (base_layer): Linear(in_features=8192, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=8192, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (act_fn): SiLU() ) (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05) (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05) ) ) (norm): LlamaRMSNorm((2048,), eps=1e-05) (rotary_emb): LlamaRotaryEmbedding() ) (lm_head): Linear(in_features=2048, out_features=128256, bias=False) ) ) )

Hi, I’m working on fine-tuning a multimodal LLM to video datasets and due to the large-scale nature of the model and huge datasets, I decided to use deepspeed for GPU memory efficiency, with ZeRO-3. (Previously, I used FSDP but it doesn’t solve the CUDA OOM issue.) To start with, the model I’m trying to fine-tune is as follows: class CambrianLlamaModel(CambrianMetaModel, LlamaModel): config_class = CambrianConfig def __init__(self, config: LlamaConfig): super(CambrianLlamaModel, self).__init__(config) # more code... class CambrianLlamaForCausalLM(LlamaForCausalLM, CambrianMetaForCausalLM): config_class = CambrianConfig def __init__(self, config): super(LlamaForCausalLM, self).__init__(config) self.model = CambrianLlamaModel(config) self.pretraining_tp = config.pretraining_tp self.vocab_size = config.vocab_size self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False) # Initialize weights and apply final processing self.post_init() # more code ... model = CambrianLlamaForCausalLM.from_pretrained( model_args.input_model_filename, **bnb_model_from_pretrained_args, ) In the fine-tuning script, I specify a `LLaVATrainer` instance: trainer = LLaVATrainer( model=model, tokenizer=tokenizer, args=training_args, callbacks=callbacks, deepspeed=training_args.deepspeed, **data_module, ) trainer.train() And my deepspeed config is as follows: { "compute_environment": "LOCAL_MACHINE", "debug": false, "deepspeed_config": { "gradient_accumulation_steps": 1, "gradient_clipping": 8.0, "offload_optimizer_device": "cpu", "offload_param_device": "cpu", "zero3_init_flag": true, "zero3_save_16bit_model": true, "zero_stage": 3 }, "zero_optimization": { "stage": 3, "offload_optimizer": { "device": "cpu" }, "offload_param": { "device": "cpu" }, "overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1e9, "reduce_bucket_size": "auto", "stage3_prefetch_bucket_size": 8493465, "stage3_param_persistence_threshold": "auto", "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_gather_16bit_weights_on_model_save": true }, "train_micro_batch_size_per_gpu": 1, "distributed_type": "DEEPSPEED", "downcast_bf16": "no", "dynamo_config": { "dynamo_backend": "INDUCTOR" }, "enable_cpu_affinity": false, "machine_rank": 0, "main_training_function": "main", "mixed_precision": "bf16", "num_machines": 1, "num_processes": 3, "rdzv_backend": "static", "same_network": true, "tpu_env": [], "tpu_use_cluster": false, "tpu_use_sudo": false, "use_cpu": false } During the fine-tuning process, there’s an error related to the dimension of weights in a conv layer: Traceback ... File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/transformers/models/dinov2/modeling_dinov2.py", line 162, in forward result = forward_call(*args, **kwargs) File "user/LongVidLLaMA/./longvu/language_model/cambrian_llama.py", line 304, in forward embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl ) = self.prepare_inputs_labels_for_multimodal( File "user/LongVidLLaMA/./longvu/cambrian_arch.py", line 830, in prepare_inputs_labels_for_multimodal return self._call_impl(*args, **kwargs) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl image_aux_features_dino = self.encode_images( File "user/LongVidLLaMA/./longvu/cambrian_arch.py", line 609, in encode_images image_aux_features_chunk = vision_tower_aux(chunk) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return forward_call(*args, **kwargs) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward return self._conv_forward(input, self.weight, self.bias) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward return self._call_impl(*args, **kwargs) File "user/Python-3.10.12/thesis_longvu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return F.conv2d(input, weight, bias, self.stride, RuntimeError: weight should have at least three dimensions I’ve done some initial search and found out that this error could stem from ZeRO stage 3 where parameters of the model are also sharded across GPUs. More particularly, the library seems to partition the model parameters into 1D (or 2D) buffers in order to save memory. However, for certain modules such as convolution layers the original shape metadata may not be preserved or automatically restored. In my DINOV2 module, the patch embedding is implemented with a convolution layer that expects its weight to be in 3 or 4 dimensions, but instead, it is coming in as a flattened tensor (e.g. only 2 dimensions). This mismatch causes the error from `torch.nn.functional.conv2d` (which requires a weight tensor with at least 3 dimensions). When I tried wrapping the model loading code with `deepspeed.zero.Init(enabled=False)`: class DinoVisionTower(BaseVisionTower): ... def load_model(self, device_map=None): # self.vision_tower = Dinov2Model.from_pretrained(self.vision_tower_name) with deepspeed.zero.Init(enabled=False): self.vision_tower = Dinov2Model.from_pretrained(self.vision_tower_name) ... The error above still persists. Hence, I would love to hear your experience if you’ve encountered this scenario before when using deepspeed ZeRO-3. Thanks in advance.

Suppose I have Llama-3.2-3B model as follow: > LlamaForCausalLM( > (model): LlamaModel( > (embed_tokens): Embedding(128256, 3072) > (layers): ModuleList( > (0-27): 28 x LlamaDecoderLayer( > (self_attn): LlamaSdpaAttention( > (q_proj): Linear(in_features=3072, out_features=3072, bias=False) > (k_proj): Linear(in_features=3072, out_features=1024, bias=False) > (v_proj): Linear(in_features=3072, out_features=1024, bias=False) > (o_proj): Linear(in_features=3072, out_features=3072, bias=False) > (rotary_emb): LlamaRotaryEmbedding() > ) > (mlp): LlamaMLP( > (gate_proj): Linear(in_features=3072, out_features=8192, bias=False) > (up_proj): Linear(in_features=3072, out_features=8192, bias=False) > (down_proj): Linear(in_features=8192, out_features=3072, bias=False) > (act_fn): SiLU() > ) > (input_layernorm): LlamaRMSNorm((3072,), eps=1e-05) > (post_attention_layernorm): LlamaRMSNorm((3072,), eps=1e-05) > ) > ) > (norm): LlamaRMSNorm((3072,), eps=1e-05) > (rotary_emb): LlamaRotaryEmbedding() > ) > (lm_head): Linear(in_features=3072, out_features=128256, bias=False) > ) How do I separate attention heads from (q_proj), (k_proj), and (v_proj) matrices? Suppose #head is known, my main concern is should I do this row-wise or column-wise?

Digging through the source code of LLaVA can feel like a tongue twister. There are a bunch of class names like LlavaMetaModel, LlamaModel…