Reduce size of swarms : Tick token using torch

[jmikedupont2]

We currently seem to have a dependency in the api server on the ticktoken that uses torch,
this adds many gbs into the docker image. Can we please remove or refactor that?

[kyegomez]

@jmikedupont2 this will be difficult because we need to cound the tokens the agents uses.

[kyegomez]

@jmikedupont2 solution i can think of is : setup a tiktoken api that counts the number of tokens in each request and returns the number of tokens. But this will take some time. How much is it adding to the server?

[kyegomez]

@jmikedupont2 i'm removing 2 more packages to slim down the size ;)

[jmikedupont2]

what if we move the token count to a separate litellm proxy server?
and that would hide the openai secrets from the agents, it could be more secure.

[kyegomez]

oken count to a separate litellm proxy server?
and that would hide the openai secrets from the agents, it could be more secure.

that could work well, would we need an api for it?

[evelynmitchell]

A litellm proxy server would run in the same environment as the swarms api server, and would wrap the incoming and outgoing api calls to the swarms api server.

user -> request -> litellm server supabase(user_usage += len(request)) -> swarms api call -> litellm server supabase(user_usage += len(answer)) -> answer -> user

The instructions to set up litellm are at: https://docs.litellm.ai/docs/proxy/deploy

[jmikedupont2]

https://github.com/jmikedupont2/openlightllm I have a fork of litellm that I would like to continue working on that removes all the non-open source mess.

[jmikedupont2]

I might have some terraform for that as well somewhere