fix: use character-based token estimation

[devin-ai-integration]

Token Counting Improvements

This PR implements the token counting changes from PR #54, replacing the simplified 1-token-per-message approach with a more accurate character-based estimation.

Changes

• Replace Vercel's 1-token-per-message with character-based estimation (chars/4)
• Remove token categories (PROMPT, ACCEPTED, REJECTED)
• Add token tracking for content length in read operations
• Add comprehensive tests for token counting

Implementation Details

• Token estimation: Math.ceil(Buffer.byteLength(content, 'utf-8') / 4)
• Removed completion_tokens_details from response
• Added tests for various message lengths and multiple messages
• Updated all tools to use consistent token tracking

Testing

• Added tests for various message lengths
• Added tests for multiple messages
• Verified token counts match character-based estimation
• Verified token counts increase with message length

Link to Devin run: https://app.devin.ai/sessions/7576e99c63674da3bba8eb30ee7b44df
Requested by: [email protected]

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring