DSAIE-vision-LLM-Project

This Project is for DSAIE Course Project!

We want to build sewer defect detction with Vision Language Model(VLM) Visual Question Answering(VQA) system.

Vision Language Model(VLM)

We use different VLM likes:

Model Name	Finish	Runing Environmrnt
Phi-3.5-vision-instruct	[x]	runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
GPT-4o	[x]	API
GPT-4omini	[x]	API
~~llama -3.2v~~	As Meta banned the model in EU
Qwen2-VL-7B-Instruct	[x]	runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04
MiniCPM-V 2.6	[x]	runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04
Llama-3.2V-11B-cot (LLaVA-CoT)	[]
moondream2	[]

We build the system with Prompt Engineering and RAG to finish VQA task.

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
.ipynb_checkpoints		.ipynb_checkpoints
results		results
.gitignore		.gitignore
MiniCPM_requirements.txt		MiniCPM_requirements.txt
Qwen2-VL-7B-Instruct.ipynb		Qwen2-VL-7B-Instruct.ipynb
README.md		README.md
dsaie-sewerwork.ipynb		dsaie-sewerwork.ipynb
gpt-4o.ipynb		gpt-4o.ipynb
gpt4o_mini.ipynb		gpt4o_mini.ipynb
gpt_requirements.txt		gpt_requirements.txt
huggingface_login.txt		huggingface_login.txt
llama-3.2v.ipynb		llama-3.2v.ipynb
llama_requirements.txt		llama_requirements.txt
minicpm-v 2.6.ipynb		minicpm-v 2.6.ipynb
phi-3.5.ipynb		phi-3.5.ipynb
phi_requirements.txt		phi_requirements.txt
qwen_requirement.txt		qwen_requirement.txt