Ollama model support

[msettlemeyer]

Are the models listed in the description the only ones that can be used with this project? Sorry, I'm new to hosting LLMs.

[rjbutler]

I've used Mistral:7b and Mistral NeMo which seemed to give better results than Llama3.2. And Qwen2.5 which was in between. They all still create new tags even when instructed not to. I haven't tried it with OpenAI because, well, privacy. I'm guessing it would also ignore the instruction to only use provided tags too, although maybe not as egregiously.

[clusterzx]

I mean yeah, care about privacy is important. But I trust that they don't lie in their terms of service.

Many big big big corporations use the paid api and feed sensitive information to it.

Always remember OpenAI API is not ChatGPT. It says the data will not be used or touched for anything and only kept for 30 days.

Maybe give it a try with non sensitive data just for testing.

[rjbutler]

I use it for other things, but most of my docs contain personal and financial information. Given their alleged willingness to use copyrighted material without permission when training their data, and general noise in the industry about running out of good data for training models going forward I will err on the side of distrust.

[clusterzx]

But regarding you initial post I can confirm I use mistral over llama aswell. Seemed to give the best results so far.

[clusterzx]

Are the models listed in the description the only ones that can be used with this project? Sorry, I'm new to hosting LLMs.

You can use all models listed on Ollama.com and also all models from huggingface.

[gima84]

For my first 15 documents "phi4" gives the best result with slightly modified standard template - at least for Correspondent and Title.

[mamema]

can you post your modifications of your template here?

[mamema]

i've launched phi4 and got this: Huh! Damn:

error: 'model requires more system memory (137.6 GiB) than is available (108.9 GiB)'

[gima84]

Strange - I am using a Mac mini with 64 RAM and its working fine there.

[mamema]

yeah, seems to be a bug: ollama/ollama#7942

[gima84]

Regarding mistral: Which mistral model are you using?
I have the following on my testplan:
14GB mistral:7b-instruct-fp16
53GB mistral-large:123b-instruct-2407-q3_K_S
59GB mistral-large:123b-instruct-2407-q3_K_M
69GB mistral-large:123b-instruct-2407-q4_0

[bat1939]

For those with lower end system with no GPU or a small GPU, I have been liking the results with the model granite3.1-moe:3b.