[Security Solution] [Elastic AI Assistant] Updates ES|QL Knowledge Base docs for 8.13 #176353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spong
added
release_note:enhancement
chore
Team: SecuritySolution
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
A documentation preview will be available soon.
Request a new doc build by commenting
If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here. |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
To update your PR or re-run it, just comment with: cc @spong |
|
Observation: Using the Automatic retries produce results (instead of an error), but they are not very useful, per the following example: |
In contrast, when a user configures RAG on alerts to send the model more alerts than can fit in context, the assistant displays the following error: |
|
@andrew-goldstein, thanks for pairing and reviewing the behavior above with me. I've updated the description with results from running the evaluator over a subset of the With regards to why the Based on these results, I think we should hold on updating these KB docs until we implement some prompt size management at the tool layer to improve resiliency and consistency here. We can continue the conversation over on https://github.com/elastic/security-team/issues/8087. |
|
Closing this PR as we're going to work the enhancements outlined over in https://github.com/elastic/security-team/issues/8087#issuecomment-1933161936 first. Hopefully by then we'll be able to ship these KB docs via integrations, and so the next KB update PR will be to remove these files instead of updating... 😀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Updates bundled ES|QL Knowledge Base docs for
8.13. Previous update was performed in #169593.Update process
To update, I deleted the
x-pack/plugins/elastic_assistant/server/knowledge_base/documentationfolder contents, then copied the latest Elasticsearch ESQL documentation files over. Then ran the below script from that directory via terminal to rename all files/directories fromkebab-casetosnake_caseas required by the Kibana repo:For the language files, I checked those manually since there are only 4 (no updates in the last 5 months).
Evaluation Results
Evaluation results of first 10 test queries from `esql_dataset.json`
[ { "@timestamp": "2024-02-07T21:06:10.599Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "My logs data (ECS) is in `logs-*`. Show me a query that gets the average CPU per host, limit it to the top 10 results, in 1m buckets, and only include the last 15m.", "reference": "FROM logs-*\n| WHERE @timestamp >= NOW() - 15 minutes\n| EVAL bucket = DATE_TRUNC(1 minute, @timestamp)\n| STATS avg_cpu = AVG(system.cpu.total.norm.pct) BY bucket, host.name\n| LIMIT 10", "prediction": "I'm sorry for any confusion, but your last comment did not contain a specific query or question regarding log data. Could you please provide more information or elaborate on your issue or question? Then I could provide a more accurate response.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "My logs data (ECS) is in `logs-*`. Show me a query that gets the average CPU per host, limit it to the top 10 results, in 1m buckets, and only include the last 15m.", "prediction.keyword": "I'm sorry for any confusion, but your last comment did not contain a specific query or question regarding log data. Could you please provide more information or elaborate on your issue or question? Then I could provide a more accurate response.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry for any confusion, but your last comment did not contain a specific query or question regarding log data. Could you please provide more information or elaborate on your issue or question? Then I could provide a more accurate response.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry for any confusion, but your last comment did not contain a specific query or question regarding log data. Could you please provide more information or elaborate on your issue or question? Then I could provide a more accurate response.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "FROM logs-*\n| WHERE @timestamp >= NOW() - 15 minutes\n| EVAL bucket = DATE_TRUNC(1 minute, @timestamp)\n| STATS avg_cpu = AVG(system.cpu.total.norm.pct) BY bucket, host.name\n| LIMIT 10", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:06:10.599Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation", "totalAgents": 1, "totalInput": 10, "totalRequests": 10, "input": null, "reference": null, "prediction": null, "connectorName": null, "connectorName.keyword": null, "evaluationDuration": 259106, "evaluationEnd": 1707339970599, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": 1707339711493, "input.keyword": null, "prediction.keyword": null, "predictionResponse.status": null, "ConnectorId": null, "predictionResponse.value.data": null, "predictionResponse.value.data.keyword": null, "predictionResponse.value.status": null, "reference.keyword": null, "runName.keyword": "8.13 ESQL Query Generation" }, { "@timestamp": "2024-02-07T21:05:50.883Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "From employees, I want to sort the documents by salary, and then return 10 results per page, and then see the second page", "reference": "Pagination is not supported", "prediction": "Based on our investigation, it is found that ES|QL does not support pagination directly. Although you can't return 10 results per page or see the second page, you can use the SORT command to order the documents by salary. An example query might be: `FROM employees | SORT salary DESC` which would return the employee documents sorted by salary in descending order.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "From employees, I want to sort the documents by salary, and then return 10 results per page, and then see the second page", "prediction.keyword": "Based on our investigation, it is found that ES|QL does not support pagination directly. Although you can't return 10 results per page or see the second page, you can use the SORT command to order the documents by salary. An example query might be: `FROM employees | SORT salary DESC` which would return the employee documents sorted by salary in descending order.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on our investigation, it is found that ES|QL does not support pagination directly. Although you can't return 10 results per page or see the second page, you can use the SORT command to order the documents by salary. An example query might be: `FROM employees | SORT salary DESC` which would return the employee documents sorted by salary in descending order.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on our investigation, it is found that ES|QL does not support pagination directly. Although you can't return 10 results per page or see the second page, you can use the SORT command to order the documents by salary. An example query might be: `FROM employees | SORT salary DESC` which would return the employee documents sorted by salary in descending order.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "Pagination is not supported", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:05:27.221Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query", "reference": "FROM employees\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \"MMMM yyyy\")\n| SORT hire_date\n| KEEP emp_no, hire_date_formatted\n| LIMIT 5", "prediction": "The information gathered is as follows: to get the 5 earliest hired employees from the 'employees' table, showing only their employee number and the month and year of their hire date in the format 'September 2019', you would use the following ES|QL query: \n\nFROM employees\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \"MMMM yyyy\")\n| SORT hire_date\n| KEEP emp_no, hire_date_formatted\n| LIMIT 5 \n\nThis query sorts the employees by hire date in ascending order, keeps only the employee number and the formatted hire date, and provides the first 5 records.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query", "prediction.keyword": "The information gathered is as follows: to get the 5 earliest hired employees from the 'employees' table, showing only their employee number and the month and year of their hire date in the format 'September 2019', you would use the following ES|QL query: \n\nFROM employees\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \"MMMM yyyy\")\n| SORT hire_date\n| KEEP emp_no, hire_date_formatted\n| LIMIT 5 \n\nThis query sorts the employees by hire date in ascending order, keeps only the employee number and the formatted hire date, and provides the first 5 records.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"The information gathered is as follows: to get the 5 earliest hired employees from the 'employees' table, showing only their employee number and the month and year of their hire date in the format 'September 2019', you would use the following ES|QL query: \\n\\nFROM employees\\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \\\"MMMM yyyy\\\")\\n| SORT hire_date\\n| KEEP emp_no, hire_date_formatted\\n| LIMIT 5 \\n\\nThis query sorts the employees by hire date in ascending order, keeps only the employee number and the formatted hire date, and provides the first 5 records.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"The information gathered is as follows: to get the 5 earliest hired employees from the 'employees' table, showing only their employee number and the month and year of their hire date in the format 'September 2019', you would use the following ES|QL query: \\n\\nFROM employees\\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \\\"MMMM yyyy\\\")\\n| SORT hire_date\\n| KEEP emp_no, hire_date_formatted\\n| LIMIT 5 \\n\\nThis query sorts the employees by hire date in ascending order, keeps only the employee number and the formatted hire date, and provides the first 5 records.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "FROM employees\n| EVAL hire_date_formatted = DATE_FORMAT(hire_date, \"MMMM yyyy\")\n| SORT hire_date\n| KEEP emp_no, hire_date_formatted\n| LIMIT 5", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:04:46.791Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "For standard Elastic ECS compliant packetbeat data view, create an ES|QL query that shows the top 10 unique domains by doc count", "reference": "FROM packetbeat-*\n| STATS doc_count = COUNT(destination.domain) BY destination.domain\n| SORT doc_count DESC\n| LIMIT 10", "prediction": "I'm sorry for the confusion, but without the specific details from the tool response, I'm unable to provide you with the exact ESQL query. However, generally, to show the top 10 unique domains by document count in an Elastic ECS compliant packetbeat data view, you'd typically use a query like: 'SELECT domain, COUNT(*) AS count FROM <your_index> GROUP BY domain ORDER BY count DESC LIMIT 10'. Please replace '<your_index>' with your specific index name. Remember, the actual query can vary depending on your specific data structure and the exact fields available in your ECS compliant packetbeat data view.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "For standard Elastic ECS compliant packetbeat data view, create an ES|QL query that shows the top 10 unique domains by doc count", "prediction.keyword": "I'm sorry for the confusion, but without the specific details from the tool response, I'm unable to provide you with the exact ESQL query. However, generally, to show the top 10 unique domains by document count in an Elastic ECS compliant packetbeat data view, you'd typically use a query like: 'SELECT domain, COUNT(*) AS count FROM <your_index> GROUP BY domain ORDER BY count DESC LIMIT 10'. Please replace '<your_index>' with your specific index name. Remember, the actual query can vary depending on your specific data structure and the exact fields available in your ECS compliant packetbeat data view.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry for the confusion, but without the specific details from the tool response, I'm unable to provide you with the exact ESQL query. However, generally, to show the top 10 unique domains by document count in an Elastic ECS compliant packetbeat data view, you'd typically use a query like: 'SELECT domain, COUNT(*) AS count FROM <your_index> GROUP BY domain ORDER BY count DESC LIMIT 10'. Please replace '<your_index>' with your specific index name. Remember, the actual query can vary depending on your specific data structure and the exact fields available in your ECS compliant packetbeat data view.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry for the confusion, but without the specific details from the tool response, I'm unable to provide you with the exact ESQL query. However, generally, to show the top 10 unique domains by document count in an Elastic ECS compliant packetbeat data view, you'd typically use a query like: 'SELECT domain, COUNT(*) AS count FROM <your_index> GROUP BY domain ORDER BY count DESC LIMIT 10'. Please replace '<your_index>' with your specific index name. Remember, the actual query can vary depending on your specific data structure and the exact fields available in your ECS compliant packetbeat data view.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "FROM packetbeat-*\n| STATS doc_count = COUNT(destination.domain) BY destination.domain\n| SORT doc_count DESC\n| LIMIT 10", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:04:15.640Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "I want to see a query for metrics-apm*, filtering on metricset.name:transaction and metricset.interval:1m, showing the average duration (via transaction.duration.histogram), in 50 buckets.", "reference": "FROM metrics-apm*\n| WHERE metricset.name == \"transaction\" AND metricset.interval == \"1m\"\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50, <start-date>, <end-date>)\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket", "prediction": "Based on the information obtained previously, here is the ESQL query you might need:\n\n```\nFROM metrics-apm*\n| WHERE metricset.name == \"transaction\" AND metricset.interval == \"1m\"\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50)\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket;\n```\n\nThis query pulls from a certain metrics set where the name is \"transaction\" and interval is \"1m\". It creates 50 buckets based on the histogram of transaction duration values and computes the average duration for each bucket. Please make sure to replace the placeholder dates when you use this query.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "I want to see a query for metrics-apm*, filtering on metricset.name:transaction and metricset.interval:1m, showing the average duration (via transaction.duration.histogram), in 50 buckets.", "prediction.keyword": "Based on the information obtained previously, here is the ESQL query you might need:\n\n```\nFROM metrics-apm*\n| WHERE metricset.name == \"transaction\" AND metricset.interval == \"1m\"\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50)\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket;\n```\n\nThis query pulls from a certain metrics set where the name is \"transaction\" and interval is \"1m\". It creates 50 buckets based on the histogram of transaction duration values and computes the average duration for each bucket. Please make sure to replace the placeholder dates when you use this query.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on the information obtained previously, here is the ESQL query you might need:\\n\\n```\\nFROM metrics-apm*\\n| WHERE metricset.name == \\\"transaction\\\" AND metricset.interval == \\\"1m\\\"\\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50)\\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket;\\n```\\n\\nThis query pulls from a certain metrics set where the name is \\\"transaction\\\" and interval is \\\"1m\\\". It creates 50 buckets based on the histogram of transaction duration values and computes the average duration for each bucket. Please make sure to replace the placeholder dates when you use this query.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on the information obtained previously, here is the ESQL query you might need:\\n\\n```\\nFROM metrics-apm*\\n| WHERE metricset.name == \\\"transaction\\\" AND metricset.interval == \\\"1m\\\"\\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50)\\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket;\\n```\\n\\nThis query pulls from a certain metrics set where the name is \\\"transaction\\\" and interval is \\\"1m\\\". It creates 50 buckets based on the histogram of transaction duration values and computes the average duration for each bucket. Please make sure to replace the placeholder dates when you use this query.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "FROM metrics-apm*\n| WHERE metricset.name == \"transaction\" AND metricset.interval == \"1m\"\n| EVAL bucket = AUTO_BUCKET(transaction.duration.histogram, 50, <start-date>, <end-date>)\n| STATS avg_duration = AVG(transaction.duration.histogram) BY bucket", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:03:36.458Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "Generate an ES|QL query that will look for all process events for the process \"curl.exe\". Calculate the sum of outbund bytes for this process by the destination address. Output the results in KB, also sorted by KB in descending order. Limit to the top 10 results.", "reference": "from logs-*\n| where process.name == \"curl.exe\"\n| stats bytes = sum(destination.bytes) by destination.address\n| eval kb = bytes/1024\n| sort kb desc\n| limit 10\n| keep kb,destination.address", "prediction": "Apologies for any misunderstanding. Based on my knowledge, without using any external tools, here is a rough ESQL query for your request. Please consult with a database or ESQL expert for further assistance and precision:\n\nSELECT \ndest_address, \nSUM(outbound_bytes/1024.0) AS total_outbound_kb \nFROM \nprocess_events \nWHERE \nprocess_name='curl.exe'\nGROUP BY \ndest_address \nORDER BY \ntotal_outbound_kb DESC \nLIMIT 10;\n\nThis query will look at 'process_events' for 'curl.exe', sum up the outbound bytes (converted to KB), and sort the result by the total outbound data for each destination address in descending order, while limiting the output to the top 10 results.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "Generate an ES|QL query that will look for all process events for the process \"curl.exe\". Calculate the sum of outbund bytes for this process by the destination address. Output the results in KB, also sorted by KB in descending order. Limit to the top 10 results.", "prediction.keyword": "Apologies for any misunderstanding. Based on my knowledge, without using any external tools, here is a rough ESQL query for your request. Please consult with a database or ESQL expert for further assistance and precision:\n\nSELECT \ndest_address, \nSUM(outbound_bytes/1024.0) AS total_outbound_kb \nFROM \nprocess_events \nWHERE \nprocess_name='curl.exe'\nGROUP BY \ndest_address \nORDER BY \ntotal_outbound_kb DESC \nLIMIT 10;\n\nThis query will look at 'process_events' for 'curl.exe', sum up the outbound bytes (converted to KB), and sort the result by the total outbound data for each destination address in descending order, while limiting the output to the top 10 results.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Apologies for any misunderstanding. Based on my knowledge, without using any external tools, here is a rough ESQL query for your request. Please consult with a database or ESQL expert for further assistance and precision:\\n\\nSELECT \\ndest_address, \\nSUM(outbound_bytes/1024.0) AS total_outbound_kb \\nFROM \\nprocess_events \\nWHERE \\nprocess_name='curl.exe'\\nGROUP BY \\ndest_address \\nORDER BY \\ntotal_outbound_kb DESC \\nLIMIT 10;\\n\\nThis query will look at 'process_events' for 'curl.exe', sum up the outbound bytes (converted to KB), and sort the result by the total outbound data for each destination address in descending order, while limiting the output to the top 10 results.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Apologies for any misunderstanding. Based on my knowledge, without using any external tools, here is a rough ESQL query for your request. Please consult with a database or ESQL expert for further assistance and precision:\\n\\nSELECT \\ndest_address, \\nSUM(outbound_bytes/1024.0) AS total_outbound_kb \\nFROM \\nprocess_events \\nWHERE \\nprocess_name='curl.exe'\\nGROUP BY \\ndest_address \\nORDER BY \\ntotal_outbound_kb DESC \\nLIMIT 10;\\n\\nThis query will look at 'process_events' for 'curl.exe', sum up the outbound bytes (converted to KB), and sort the result by the total outbound data for each destination address in descending order, while limiting the output to the top 10 results.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "from logs-*\n| where process.name == \"curl.exe\"\n| stats bytes = sum(destination.bytes) by destination.address\n| eval kb = bytes/1024\n| sort kb desc\n| limit 10\n| keep kb,destination.address", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:02:46.550Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "Generate an ES|QL query that will filter for file creation events. Count the number of file interactions by the process interacting with the file, and the host name. From the process name field, parse and output two new fields that represent the process and the process extension separately. Calculate the length of the process name and filter for events where the length is greater than 15 characters. Sort the result based on the process length and filecount in descending order. Limit the results to the top 10.", "reference": "from logs-*\n| where event.category == \"file\" and event.action == \"creation\"\n| stats filecount = count(file.name) by process.name,host.name\n| dissect process.name \"%{process}.%{extension}\" \n| eval proclength = length(process.name)\n| where proclength > 10 \n| sort filecount,proclength desc\n| limit 10 \n| keep host.name,process.name,filecount,process,extension,fullproc,proclength", "prediction": "Based on the information retrieved, you wanted to generate an ES|QL query for file creation events. This entails steps such as counting file interactions, processing and sorting data, and limiting results to the top 10. Yet, because you've forgotten the responses provided by the tools, It's advisable to rerun the tool for creating the ES|QL query. This information is crucial as it helps in crafting an appropriate response to your query.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "Generate an ES|QL query that will filter for file creation events. Count the number of file interactions by the process interacting with the file, and the host name. From the process name field, parse and output two new fields that represent the process and the process extension separately. Calculate the length of the process name and filter for events where the length is greater than 15 characters. Sort the result based on the process length and filecount in descending order. Limit the results to the top 10.", "prediction.keyword": "Based on the information retrieved, you wanted to generate an ES|QL query for file creation events. This entails steps such as counting file interactions, processing and sorting data, and limiting results to the top 10. Yet, because you've forgotten the responses provided by the tools, It's advisable to rerun the tool for creating the ES|QL query. This information is crucial as it helps in crafting an appropriate response to your query.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on the information retrieved, you wanted to generate an ES|QL query for file creation events. This entails steps such as counting file interactions, processing and sorting data, and limiting results to the top 10. Yet, because you've forgotten the responses provided by the tools, It's advisable to rerun the tool for creating the ES|QL query. This information is crucial as it helps in crafting an appropriate response to your query.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Based on the information retrieved, you wanted to generate an ES|QL query for file creation events. This entails steps such as counting file interactions, processing and sorting data, and limiting results to the top 10. Yet, because you've forgotten the responses provided by the tools, It's advisable to rerun the tool for creating the ES|QL query. This information is crucial as it helps in crafting an appropriate response to your query.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "from logs-*\n| where event.category == \"file\" and event.action == \"creation\"\n| stats filecount = count(file.name) by process.name,host.name\n| dissect process.name \"%{process}.%{extension}\" \n| eval proclength = length(process.name)\n| where proclength > 10 \n| sort filecount,proclength desc\n| limit 10 \n| keep host.name,process.name,filecount,process,extension,fullproc,proclength", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:02:10.817Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "Generate an ES|QL query that will filter all logs for those containing windows event codes, count them by host name, and enrich the codes with a description of what the code means (via an enrichment policy called \"win_events\"). The results should be sorted by the count of each code in descending order", "reference": "from logs-*\n| where event.code is not null\n| stats event_code_count = count(event.code) by event.code,host.name\n| enrich win_events on event.code with EVENT_DESCRIPTION\n| where EVENT_DESCRIPTION is not null and host.name is not null\n| rename EVENT_DESCRIPTION as event.description\n| sort event_code_count desc\n| keep event_code_count,event.code,host.name,event.description\"", "prediction": "error", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "Generate an ES|QL query that will filter all logs for those containing windows event codes, count them by host name, and enrich the codes with a description of what the code means (via an enrichment policy called \"win_events\"). The results should be sorted by the count of each code in descending order", "prediction.keyword": "error", "predictionResponse.status": "rejected", "ConnectorId": null, "predictionResponse.value.data": null, "predictionResponse.value.data.keyword": null, "predictionResponse.value.status": null, "reference.keyword": "from logs-*\n| where event.code is not null\n| stats event_code_count = count(event.code) by event.code,host.name\n| enrich win_events on event.code with EVENT_DESCRIPTION\n| where EVENT_DESCRIPTION is not null and host.name is not null\n| rename EVENT_DESCRIPTION as event.description\n| sort event_code_count desc\n| keep event_code_count,event.code,host.name,event.description\"", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:02:10.153Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "Generate an ES|QL query that will parse the DNS registered domain from a DNS query, count the number distinct DNS queries being made per DNS registered domain and filter for when the distinct count is greater than 5. The query should sort the results by the distinct count of queries in descending order.", "reference": "from logs-*\n| grok dns.question.name \"%{DATA}\\\\.%{GREEDYDATA:dns.question.registered_domain:string}\"\n| stats unique_queries = count_distinct(dns.question.name) by dns.question.registered_domain, process.name\n| where unique_queries > 5\n| sort unique_queries desc", "prediction": "error", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "Generate an ES|QL query that will parse the DNS registered domain from a DNS query, count the number distinct DNS queries being made per DNS registered domain and filter for when the distinct count is greater than 5. The query should sort the results by the distinct count of queries in descending order.", "prediction.keyword": "error", "predictionResponse.status": "rejected", "ConnectorId": null, "predictionResponse.value.data": null, "predictionResponse.value.data.keyword": null, "predictionResponse.value.status": null, "reference.keyword": "from logs-*\n| grok dns.question.name \"%{DATA}\\\\.%{GREEDYDATA:dns.question.registered_domain:string}\"\n| stats unique_queries = count_distinct(dns.question.name) by dns.question.registered_domain, process.name\n| where unique_queries > 5\n| sort unique_queries desc", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" }, { "@timestamp": "2024-02-07T21:02:09.505Z", "evaluationId": "475479cf-608d-41e4-911b-81ab47fad639", "runName": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor", "totalAgents": null, "totalInput": null, "totalRequests": null, "input": "Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called \"follow_up\" that contains a value of \"true\", otherwise, it should contain \"false\". The user names should also be enriched with their respective group names.", "reference": "FROM logs-*\n| WHERE NOT CIDR_MATCH(destination.ip, \"10.0.0.0/8\", \"172.16.0.0/12\", \"192.168.0.0/16\")\n| STATS destcount = COUNT(destination.ip) by user.name, host.name\n| ENRICH ldap_lookup_new ON user.name\n| WHERE group.name IS NOT NULL\n| EVAL follow_up = CASE(\n destcount >= 100, \"true\",\n \"false\")\n| SORT destcount desc\n| KEEP destcount, host.name, user.name, group.name, follow_up", "prediction": "I'm sorry, but I'm unable to provide the response to your last comment as no information was given from the ESQLKnowledgeBaseTool. I would typically use the tool to construct the required ES|QL query, but in this case, I don't have any response from the tool to work with.", "connectorName": "Azure OpenAI", "connectorName.keyword": "Azure OpenAI", "evaluationDuration": null, "evaluationEnd": null, "evaluationId.keyword": "475479cf-608d-41e4-911b-81ab47fad639", "evaluationStart": null, "input.keyword": "Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called \"follow_up\" that contains a value of \"true\", otherwise, it should contain \"false\". The user names should also be enriched with their respective group names.", "prediction.keyword": "I'm sorry, but I'm unable to provide the response to your last comment as no information was given from the ESQLKnowledgeBaseTool. I would typically use the tool to construct the required ES|QL query, but in this case, I don't have any response from the tool to work with.", "predictionResponse.status": "fulfilled", "ConnectorId": "azure-open-ai", "predictionResponse.value.data": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry, but I'm unable to provide the response to your last comment as no information was given from the ESQLKnowledgeBaseTool. I would typically use the tool to construct the required ES|QL query, but in this case, I don't have any response from the tool to work with.\"\n}\n```", "predictionResponse.value.data.keyword": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"I'm sorry, but I'm unable to provide the response to your last comment as no information was given from the ESQLKnowledgeBaseTool. I would typically use the tool to construct the required ES|QL query, but in this case, I don't have any response from the tool to work with.\"\n}\n```", "predictionResponse.value.status": "ok", "reference.keyword": "FROM logs-*\n| WHERE NOT CIDR_MATCH(destination.ip, \"10.0.0.0/8\", \"172.16.0.0/12\", \"192.168.0.0/16\")\n| STATS destcount = COUNT(destination.ip) by user.name, host.name\n| ENRICH ldap_lookup_new ON user.name\n| WHERE group.name IS NOT NULL\n| EVAL follow_up = CASE(\n destcount >= 100, \"true\",\n \"false\")\n| SORT destcount desc\n| KEEP destcount, host.name, user.name, group.name, follow_up", "runName.keyword": "8.13 ESQL Query Generation - Azure OpenAI + DefaultAgentExecutor" } ]Results and tracing on LangSmith are here.
As alluded to in @andrew-goldstein's comments below, the updated KB documents now contain more data than before, which is resulting in the context window being exceeded within the
ESQLKnowledgeBaseToolchain. Instead of bubbling up the error (like with RAG on alerts), the agent completes the request as if the tool was not successful, responding e.g.Based on the above results (where 3/10 were successful, 2/10 were error, and the remaining 5/10 blew the context size and fell back to a non-tool response), it seems best to implement some prompt size management at the tool layer to improve resiliency and consistency here. Will continue conversation on https://github.com/elastic/security-team/issues/8087
Test Instructions
To test, setup the Knowledge Base as detailed in the docs. You should see the below
INFO/DEBUGlog messages indicating the knowledge base documents were ingested and created successfully. (Note: if already installed, you must delete and re-initialize, please see elastic/security-docs#4773 for adding this note to the KB docs)Now you can ask an ES|QL query generation question to ensure functionality, and verify returned knowledge base docs in the
DEBUGlogs by searching forSimilarity search metadata source:, e.g.