MakerSuite does not work -- instructions ignored.

[dosstx]

Hello, I am evaluating Google Palm 2 API and I am using MakerSuite for testing. I must be doing something wrong, but I am not understanding how the "context" prompt input works in MakerSuite. I simply write the following context: "Do not talk about baseball." and also provided an example user/model input stating not to talk about baseball, but when I run the test and ask "What is baseball?", the model responds back talking about baseball:

So, what am I doing wrong?

[MarkDaoust]

I don't think you're doing anything wrong. System instructions aren't strictly enforced, the model owners know this should be improved. This may be better with the Gemini models, but I don't have data on this.

[MarkDaoust]

Quick test with the gemini model:

response = model.generate_content(
"""Important: You must not mention baseball in your response.

What are some popular sports in the united states? Name a few where you hit a ball with a stick of some kind.""")

response.text

- Golf: Players use various clubs to hit balls into a series of holes on a course, with the fewest strokes winning.
- Hockey: Players use sticks to shoot a puck into a net, while other players try to prevent them from doing so.
- Tennis: Players use rackets to hit a ball over a net, with points awarded for successful hits and faults for missed hits or out-of-bounds shots.
- Lacrosse: Players use long-handled sticks with netted ends to catch, carry, and throw a ball, aiming to score by throwing it into a goal.
- Cricket: Players use wooden bats to hit a hard leather ball bowled by an opponent, with runs scored by running between wickets or hitting the ball out of bounds.

and

response = model.generate_content(
"""Important: You must not mention baseball in your response.

How do you play baseball?""")

response.text

I cannot answer this question as I am not supposed to mention baseball in my responses.