Model testing (ollama) - and some tips and tricks

[mamema]

i was trying to optimize the outcome of tagging and title assignment, so i was thinking what can i do and made some tests:

• even though, the solution and the example prompt is in english, results are better if you translate your prompt to the document language you're analyzing
• even though we're having a playground to test prompting, which is great, i had tremendous success to ask the "best machine" (openai) we have to optimize prompting for another machine (ollama). My thinking was: i'm a human and i haven't developed those machines, so i do not know what they really need in good prompting. Refining the outcome of my prompts with AI are measurable.

Additional to the default prompt i had three major requirements: Use the year of the document as tag. Don't use my name within tags or correspondents. Use my language for tags.
Your prompt may don't need those requirements, but for me this was a great way to differentiate the following list of models.

Every model got the same prompt to work with.

Mistral7B:

• good Tag selection
• but never assigned the Year as Tag, which is a requirement
• out of 14 documrnts 8 were not tagged at all
• generated tags called "tag1","tag2","tag3"
• used english tags even though forbidden

llama2:7b

• tag selection not as good as mistral
• also no year tag attached
• Correspondent MyName was used, even though forbidden in prompt
• Correspondent "xxxxxxxx" assigned. useless. Document is OCRed
• generated tags called "tag1","tag2","tag3"
• out of 14 documrnts 8 were not tagged at all

Mistral-NeMo

• good Tag selection
• only partly on some documrnts assigned the Year as Tag, which is a requirement
• out of 14 documrnts 8 were not tagged at all
• generated tags called "tag1","tag2","tag3"
• My Name assigned as tag, even though forbidden
• out of 14 documrnts 4 were not tagged at all

phi4

• constantly tagged the year
• good tagging
• tagged with My Name which isn't allowed
• good on medical term tagging
• out of 14 documents 6 were not tagged at all
• used english tag terms even though forbidden in the prompt

gemma2:9b

• year gets extracted
• tag quality is good
• 4 out of 14 documents are not tagged at all

For me, the winner is "gemma2:9b", your mileage may vary.

[Snify89]

Please give GLM4 a try. Especially for German users, this is the best option IMHO.

[Atomique]

Which one do you prefer exactly? Would like to try this model

[mamema]

i cannot confirm this. For a document, which was processed fine by gemma, glm4 came up with this correspondent:
"2023 !ai-processed Karmaoa theges wijiltH 2 > Bir oy. Be: Rlleyoe rast 1 he Stes u ie et Re ae Je Karo oQas Fersen tee / << . = a Qans fe Ver, uye Ou strehck oe oe u: Ban ns Warren rn"

[Snify89]

i cannot confirm this. For a document, which was processed fine by gemma, glm4 came up with this correspondent: "2023 !ai-processed Karmaoa theges wijiltH 2 > Bir oy. Be: Rlleyoe rast 1 he Stes u ie et Re ae Je Karo oQas Fersen tee / << . = a Qans fe Ver, uye Ou strehck oe oe u: Ban ns Warren rn"

Thank you for testing this.
Could you provide the prompt please, the document language, context size etc. (ollama). Can you chat with the model in ollama cli/openwebui? The output looks weird.
I shall test it myself but I haven't setup anything yet.

[mamema]

yes the output looks weird, cause the tested document is an edge case with partly handwritten text. i use this document with a cause, as it differentiates the models very good for my needs, gemma can handle it, glm4 not. The rest what glm4 is delivering is on par with gemma. i don't see better titles or tags or correspondents with glm4 in comparision to gemma

[SenseiMarv]

Out of curiosity: Why did you pick such an old version of llama compared to the other models? I would be interested to see the results of llama3.1:8b, especially compared to gemma2:9b as according to Meta it succeeds that model in benchmarks. Maybe you can have a look if you find the time :)

[SenseiMarv]

Ah, thanks for checking! 3.2 has only a size up to 3B - I think this explains why it couldn't hold up to the other models. 3.1 also offers bigger sizes, so that's probably better comparable to models like gemma2.

[mamema]

okay i'll test that

[mamema]

tested now with llama3.1:8b, i would say it's on par with gemma2, i did not see a clear winner between those two

[SenseiMarv]

Thanks for checking, as these two models were pretty close in the benchmarks as well, it matches my expectations.

[Mehyar-ALS]

Many thanks for sharing your findings, this is very helpful.

Will it be possible to update the main post with the new findings? Just to keep it updated as many might not go through the comments.

[rjbutler]

@mamema are you letting it suggest tags, or are you using existing tags only? If the latter, which models do better at not hallucinating new tags?

Also, are you using the default Ollama context window size?

[mamema]

i´m using open webui. there in the model settings for ollama.....

[Atomique]

But that is the setting for open webui, sending its requests to ollama, isnt it? Doesnt sound like that paperless-ai will use this setting. Or did you somehow configure open webui as host?

[mamema]

frankly i didn´t know. perhaps you´ve found something. need to be investigated

[Atomique]

Which URL did you configure in paperless-ai? The url from your ollama or did you configure an url from owui?

[mamema]

ollama