Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the opt...

Researchers at Salesforce introduced BLIP-3-Video, a vision-language model that captures temporal data in videos using only 32 tokens, delivering efficiency without compromising accuracy.

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the opt...

Explore this post and more from the machinelearningnews community

Join the discussion on this paper page

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the…

Hello, after running both PIP lines, and creating a python file (test_blip2.py), when running the test script you suggested, i get the following error: Traceback (most recent call last): File “/workspace/test_blip.py”, line 1, in from lavis.models import load_model_and_preprocess ModuleNotFoundError: No module named ‘lavis’ Thoughts?

Perhaps lavis has not been installed… By the way, for BLIP-3o, it may be more reliable to use torchrun rather than lavis (according to the official GitHub). github.com/JiuhaiChen/BLIP3o #### Regarding the finetuning opened 08:09PM - 15 Jun 25 UTC kaneyxx Thanks for the amazing work! I am planning to fine-tune your model on my privat…e dataset. My dataset consists of triplets: {training target image, text prompt, special condition embedding}. This special embedding is extracted from another domain using a separate encoder (dim=768). So, basically I have two ocnditions now. I’m wondering how to adapt to your current SFT pipeline. Could you provide some guidance or instructions? Thank you!

Use a prebuilt Docker image Try: FROM pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime RUN pip install git+https://github.com/salesforce/LAVIS.git@main && pip install accelerate transformers bitsandbytes datasets Validate BLIP-3 with a simple test: from lavis.models import load_model_and_preprocess model, vis_processors, txt_processors = load_model_and_preprocess( name="blip2_t5_instruct", model_type="pretrain_flant5xl", is_eval=True, device="cuda" ) Start with instruct_finetune.py Don't launch the full training loop until you're sure the dataset and model initialize cleanly. Format your dataset properly Use HuggingFace-style datasets avoid raw filepaths on Runpod. Structure matters. Notes: bitsandbytes will silently fail if GPU doesn’t support it. Check transformers version if tokenizer fails to load. Set HF_HOME=/workspace/hf_cache in your Docker to avoid cache collisions. Let me know your dataset format if you need a single-image test script to verify captioning before tuning. Solution provided by Triskel Data Deterministic AI.

Hello! I’ve been trying to fine-tune the multimodal model Blip3-o using Runpod and ChatGPT as a guide but feel like I keep going in circles having problems with dependencies. Never been able to run a test.py or even my dataset and get an output. It’s always something. Any guidance from someone that has been able to pull this off? Thanks!