job chat: claude issues

[josephjclark]

An issue to share issues found in Claude's responses in job chat

[josephjclark]

#141 - Bad use of fn()(state) in the each each callback

[josephjclark]

Claude doesn't seem to know what adaptor is being used

[josephjclark]

From Aisha: don't always assume the answer is job code - could just be regular js

This was my question to the AI: Given a JSON data, and given a list of columns in an sql table, give me an insert query. 

The AI assistant immediately assumed i wanted an OpenFn code for the request i made. I just wanted it to structure a json object into an sql insert query format so i can try it on the db. I'm not sure if the AI assuming I wanted an OpenFn code by default was an intended feature but wanted to point that out

#160

[josephjclark]

We've had two reports that the AI doesn't know what the user's job code looks like.

This could be a lightning issue but more likely it's the model being wild

Sorry I don't have any evidence to hand here

[josephjclark]

Collections! I'll bet there's nothing in the prompt to explain the collections API. This is a great case where a good RAG would really help

See #143

[josephjclark]

From Jack: My experience with the AI Assistant so far is that it has been pretty verbose/excessive. ,Any time I run through what I've made with Taylor he usually re-writes it with a much simpler expression which achieves what I actually want to do, without the bells and whistles (which I guess Claude wants to give us out of the box)?

#157

[josephjclark]

Question: Please open a previous session

Answer: I apologize, but I don't have access to any previous sessions or conversations. I'm an AI language model designed to provide information and assistance based on the current conversation only. Each interaction starts fresh.

Is there something specific from a previous conversation that you'd like to discuss or ask about? I'd be happy to help you with any questions or topics related to OpenFn job writing or DHIS2 integration to the best of my current knowledge and capabilities.

---

The assistant is a bit out of line here. It doesn't understand its context properly - not every interaction starts fresh. Actually, we use sessions with memory, so Claude does have some recall and context.

Also, while it's true that Claude can't open a session itself, it could (in a perfect world) tell users how to do it.

Related: what if users ask "what can you see?". #156

We should improve Claude's context about how it is being used in our app.

[josephjclark]

Terrible handling of DHIS2. Here's some sample code it gave:

// Configuration for source DHIS2 instance
alterState(state => ({
  ...state,
  configuration: {
    username: 'source_username',
    password: 'source_password',
    baseUrl: 'https://source.dhis2.org/api'
  }
}));

// Fetch data from source DHIS2
get('resourceType', {
  // Add query parameters here
});

// Transform data if needed
fn(state => {
  const sourceData = state.data;
  // Add your data transformation logic here
  return { ...state, transformedData: sourceData };
});

// Configuration for destination DHIS2 instance
alterState(state => ({
  ...state,
  configuration: {
    username: 'destination_username',
    password: 'destination_password',
    baseUrl: 'https://destination.dhis2.org/api'
  }
}));

// Send data to destination DHIS2
create('resourceType', state => state.transformedData);

The big issues here are:

• Total mishandling of credentials (twice??) job chat: understand how credentials work #158
• Use of alterState, which I'm sure we explicitly tell it not to job chat: sometimes users alterState despite the prompt saying not to #159

[aleksa-krolls]

hey @hunterachieng I'm not clear on what the status is here/ next steps. Is there any AI work you need to wrap up in this coming sprint?

[hunterachieng]

@aleksa-krolls I have worked on an AI issue that is currently under review. I was to tackle this after that was reviewed but we can slot it for another day this week.