sentence splitting implementation

[doobidoo]

To implement sentence splitting for sentences that are too long, we need to break them down into smaller parts while ensuring that the resulting chunks do not exceed the specified chunk size. Here's an updated version of the splitIntoChunks method that includes sentence splitting for overly long sentences:

[splitbrain]

Thanks for approaching this, but I think it needs some clean up

[splitbrain]

my previous concerns have not been addressed yet.

[doobidoo]

Sorry, for not getting back earlier. Regarding the sentence splitting. What is the max chunk size respectively where is it settled.
If that is taken into the equation the sentence splitter could basically have a treshold on where to split if no separator can be found.

[doobidoo]

I have made some reasonable adjustments to tackle the large bodies. It is certainly not perfect, however it will not break the code and the embedding service will complain anyway if the payload is to big. Here is an excerpt of my test:

⚠ Sentence too long, splitting it into smaller parts
✗ Failed to get embedding for chunk of page intern:editors:bpmn: Gemini API error: Request payload size exceeds the limit: 10000 bytes.
✓ intern:editors:bpmn split into 1 chunks
ℹ Reusing chunks for faq:sql:memory
⚠ Sentence too long, splitting it into smaller parts
✗ Failed to get embedding for chunk of page intern:editors:loremipsumlongtext: Gemini API error: Request payload size exceeds the limit: 10000 bytes.
✓ intern:editors:loremipsumlongtext split into 1 chunks
ℹ Assigning clusters to chunks...
✓ Chunk 11300 assigned to cluster 14