Video-LLaVA: Learning United Visual Representation by Alignment Before Projection If you like our project, please give us a star ⭐ on GitHub for latest update.
Open-Sora Plan: Open-Source Large Video Generation Model
Bin Lin and Yunyang Ge and Xinhua Cheng and Zongjian Li and Bin Zhu and Shaodong Wang and Xianyi He and Yang Ye and Shenghai Yuan and Liuhan Chen and Tanghui Jia and Junwu Zhang and Zhenyu Tang and Yatian Pang and Bin She and Cen Yan and Zhiheng Hu and Xiaoyi Dong and Lin Chen and Zhang Pan and Xing Zhou and Shaoling Dong and Yonghong Tian and Li Yuan
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, Li Yuan
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan
Video-LLaVA exhibits remarkable interactive capabilities between images and videos, despite the absence of image-video pairs in the dataset.
💡 Simple baseline, learning united visual representation by alignment before projection With the binding of unified visual representations to the language feature space, we enable an LLM to perform visual reasoning capabilities on both images and videos simultaneously. 🔥 High performance, complementary learning with video and image Extensive experiments demonstrate the complementarity of modalities, showcasing significant superiority when compared to models specifically designed for either images or videos.
Highly recommend trying out our web demo by the following command, which incorporates all features currently supported by Video-LLaVA. We also provide online demo in Huggingface Spaces.
python -m videolla.serve.gradio_web_server demo.mp4 CLI Inference CUDA_VISIBLE_DEVICES=0 python -m videolla.serve.cli --model-path "LanguageBind/Video-LLaVA-7B" --file "path/to/your/video.mp4" --load-4bit
Warning
🚨 Upgrade transformers for quick access. pip install -U transformersIf you need to install then do
python -m pip install import import numpy as np from transformers import VideoLlaProcessor, VideoLlaForConditionalGeneration def read_video_py(container, indices): frames = [] container.seek(0) start_index = indices[0] end_index = indices[-1] for i, frame in enumerate(container.decode(video=0)): if i > end_index: break if i >= start_index and i in indices: frames.append(frame) return np.stack([x.to_ndarray(format="rgb24") for x in frames]) model = VideoLlaForConditionalGeneration.from_pretrained("LanguageBind/Video-LLaVA-7B-hf") processor = VideoLlaProcessor.from_pretrained("LanguageBind/Video-LLaVA-7B-hf") prompt = "USER: Why is this video funny? ASSISTANT:" video_path = "YOUR-LOCAL-VIDEO-PATH" container = .open(video_path) # sample uniformly 8 frames from the video total_frames = container.streams.video[0].frames indices = np.arange(0, total_frames, total_frames / 8).astype(int) clip = read_video_py(container, indices) inputs = processor(text=prompt, videos=clip, return_tensors="pt") # Generate generate_ids = model.generate(**inputs, max_length=80) print(processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]) >>> 'USER: Why is this video funny? ASSISTANT: The video is funny because the baby is sitting on the bed and reading a book, which is an unusual and amusing sight.' outdatedWe open source all codes. If you want to load the model (e.g. LanguageBind/Video-LLaVA-7B) on local, you can use the following code snippets.
Inference for image import torch from videolla.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN from videolla.conversation import conv_templates, SeparatorStyle from videolla.model.builder import load_pretrained_model from videolla.utils import disable_torch_init from videolla.mm_utils import tokenizer_image_token, get_model_name_from_path, KeywordsStoppingCriteria def main(): disable_torch_init() image = 'videolla/serve/examples/extreme_ironing.jpg' inp = 'What is unusual about this image?' model_path = 'LanguageBind/Video-LLaVA-7B' cache_dir = 'cache_dir' device = 'cuda' load_4bit, load_8bit = True, False model_name = get_model_name_from_path(model_path) tokenizer, model, processor, _ = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device, cache_dir=cache_dir) image_processor = processor['image'] conv_mode = "lla_v1" conv = conv_templates[conv_mode].copy() roles = conv.roles image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'] if type(image_tensor) is list: tensor = [image.to(model.device, dtype=torch.float16) for image in image_tensor] else: tensor = image_tensor.to(model.device, dtype=torch.float16) print(f"{roles[1]}: {inp}") inp = DEFAULT_IMAGE_TOKEN + '\n' + inp conv.append_message(conv.roles[0], inp) conv.append_message(conv.roles[1], None) prompt = conv.get_prompt() input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda() stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2 keywords = [stop_str] stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids) with torch.inference_mode(): output_ids = model.generate( input_ids, images=tensor, do_sample=True, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=[stopping_criteria]) outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:]).strip() print(outputs) if __name__ == '__main__': main() Inference for video import torch from videolla.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN from videolla.conversation import conv_templates, SeparatorStyle from videolla.model.builder import load_pretrained_model from videolla.utils import disable_torch_init from videolla.mm_utils import tokenizer_image_token, get_model_name_from_path, KeywordsStoppingCriteria def main(): disable_torch_init() video = 'videolla/serve/examples/sample_demo_1.mp4' inp = 'Why is this video funny?' model_path = 'LanguageBind/Video-LLaVA-7B' cache_dir = 'cache_dir' device = 'cuda' load_4bit, load_8bit = True, False model_name = get_model_name_from_path(model_path) tokenizer, model, processor, _ = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device, cache_dir=cache_dir) video_processor = processor['video'] conv_mode = "lla_v1" conv = conv_templates[conv_mode].copy() roles = conv.roles video_tensor = video_processor(video, return_tensors='pt')['pixel_values'] if type(video_tensor) is list: tensor = [video.to(model.device, dtype=torch.float16) for video in video_tensor] else: tensor = video_tensor.to(model.device, dtype=torch.float16) print(f"{roles[1]}: {inp}") inp = ' '.join([DEFAULT_IMAGE_TOKEN] * model.get_video_tower().config.num_frames) + '\n' + inp conv.append_message(conv.roles[0], inp) conv.append_message(conv.roles[1], None) prompt = conv.get_prompt() input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda() stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2 keywords = [stop_str] stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids) with torch.inference_mode(): output_ids = model.generate( input_ids, images=tensor, do_sample=True, temperature=0.1, max_new_tokens=1024, use_cache=True, stopping_criteria=[stopping_criteria]) outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:]).strip() print(outputs) if __name__ == '__main__': main() 🗝️ Training & ValidatingThe training & validating instruction is in TRAIN_AND_VALIDATE.md.
👍 Acknowledgement LLaVA The codebase we built upon and it is an efficient large language and vision assistant. Video-ChatGPT Great job contributing the evaluation code and dataset. 🙌 Related Projects LanguageBind An open source five modalities language-based retrieval framework. Chat-UniVi This framework empowers the model to efficiently utilize a limited number of visual tokens. 🔒 License The majority of this project is released under the Apache 2.0 license as found in the LICENSE file. The service is a research preview intended for non-commercial use only, subject to the model License of LLaMA, Terms of Use of the data generated by OpenAI, and Privacy Practices of ShareGPT. Please contact us if you find any potential violation. ✏️ CitationIf you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📝.
@article{lin2023video, title={Video-LLaVA: Learning United Visual Representation by Alignment Before Projection}, author={Lin, Bin and Zhu, Bin and Ye, Yang and Ning, Munan and Jin, Peng and Yuan, Li}, journal={arXiv preprint arXiv:2311.10122}, year={2023} } @article{zhu2023languagebind, title={LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment}, author={Zhu, Bin and Lin, Bin and Ning, Munan and Yan, Yang and Cui, Jiaxi and Wang, HongFa and Pang, Yatian and Jiang, Wenhao and Zhang, Junwu and Li, Zongwei and others}, journal={arXiv preprint arXiv:2310.01852}, year={2023} } ✨ Star History