Add curator to handle inference for the model being evaluated

[RyanMarten]

No description provided.

[RyanMarten]

Ok we are going to fix two things in curator to simplify this (can do another PR later when these fixes are released)

1. Allow for passing in list[messages] directly in llm() instead of requiring to make a dataset

• Allow lists of messages as simple input bespokelabsai/curator#371

2. Fixing the rate limit issue with anthropic models so we don't have to have manual if statements setting rate limits

• Support separate rate limits for input and output tokens bespokelabsai/curator#239

[RyanMarten]

Testing with

  python -m eval.eval \
        --model curator  \
        --tasks alpaca_eval \
        --model_name "gemini/gemini-1.5-flash" \
        --annotator_model "gpt-4o-mini-2024-07-18" \
        --apply_chat_template False \
        --model_args 'tokenized_requests=False' \
        --output_path logs

[RyanMarten]

Testing with

 python -m eval.eval \
        --model curator  \
        --tasks alpaca_eval \
        --model_name "claude-3-5-haiku-20241022" \
        --annotator_model "gpt-4o-mini-2024-07-18" \
        --apply_chat_template False \
        --model_args 'tokenized_requests=False' \
        --debug \
        --output_path logs

Working!

[RyanMarten]

It would be better to pass backend_params (e.g. {"max_requests_per_minute": 2_000, "max_tokens_per_minute": 4_000_000}) via --model_args but couldn't really figure it out how to access these args in the class

@jmercat have you done this before? tried adding a

    @classmethod
    def create_from_arg_string(

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/ibm_watsonx_ai.py#L72-L73

But ran into bugs, so just hardcoded the rate limits for gemini in the class itself