Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ollama/Mistral Fixes and Improvements #18

Open
lazydog2 opened this issue Jul 30, 2024 · 0 comments
Open

Ollama/Mistral Fixes and Improvements #18

lazydog2 opened this issue Jul 30, 2024 · 0 comments

Comments

@lazydog2
Copy link

lazydog2 commented Jul 30, 2024

I wasn't able to get the movies.run.ts example to work as is and had to change "template:" to "prompt:" here:
https://github.com/hrishioa/wishful-search/blob/master/src/ollama.ts#L24

I was also experiencing UND_ERR_HEADERS_TIMEOUT errors because the LLM was taking longer to respond than the fetch default of 30 seconds so I made the following changes:
Added the "undici" dependency to package.json and ran npm install:

"dependencies": {
    "@anthropic-ai/sdk": "^0.16.1",
    "@azure/openai": "^1.0.0-beta.9",
    "openai": "^4.24.7",
    "sql.js": "^1.8.0",
    "undici": "^6.19.0",
  },

Added the following to the beginning of ollama.ts:
import { fetch, Agent } from 'undici';
And replaced:

// Using fetch to initiate a POST request that will return an SSE stream
  const response = await fetch(`http://127.0.0.1:${port}/api/generate`, {
    method: 'POST',
    headers: headers,
    body: requestBody,
  });

With:

// Using fetch to initiate a POST request that will return an SSE stream
  const response = await fetch(`http://127.0.0.1:${port}/api/generate`, {
    method: 'POST',
    headers: headers,
    body: requestBody,
    dispatcher: new Agent({
      headersTimeout: 10 * 60e3,  // 10 minutes
      bodyTimeout: 10 * 60e3,     // 10 minutes
    }),
  });

I also found that when the response from the LLM contained a curly brace, the json parsing would fail so I replaced the following in ollama.ts:

      // Attempt to parse all complete JSON objects in the buffer
      while (true) {
        const openingBraceIndex = textBuffer.indexOf('{');
        const closingBraceIndex = textBuffer.indexOf('}');

        // Check if we have a complete object
        if (openingBraceIndex !== -1 && closingBraceIndex !== -1) {
          // Extract and parse the JSON object
          const jsonString = textBuffer.slice(
            openingBraceIndex,
            closingBraceIndex + 1,
          );

With (there may be better ways to achieve this):

      const regex = /{.*?"done":(?:true|false).*?}/g;
      const jsonStrings = textBuffer.match(regex);
      if (jsonStrings == null) {
        continue;
      } else if (jsonStrings.length == 0) {
        continue;
      }

      // Attempt to parse all complete JSON objects in the buffer
      for (let i = 0; i < jsonStrings.length; i++) {
        const currentJsonString = jsonStrings[i];
	if (currentJsonString == null) {
            break;
        }
        const closingBraceIndex = currentJsonString.length;

        // Check if we have a complete object
        if (closingBraceIndex !== -1) {
          const jsonString = currentJsonString;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant