Hallucinated incorrect code despite all references being correct

[pamelafox]

Type: Bug

I asked "how to use github models to call gpt-4o?"
All the references had correct Python code, but it provided very wrong code. I think possibly the code might have been the OpenAI v0 version, but its been a while since people have used that. Ideally it'd stick better with the references.
Probably not fixable since it's a hallucination but reporting anyway.

Extension version: 0.3.42
VS Code version: Code 1.96.4 (cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba, 2025-01-16T00:16:19.038Z)
OS version: Darwin arm64 23.6.0
Modes:

System Info

Item	Value
CPUs	Apple M1 Pro (10 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	3, 4, 4
Memory (System)	16.00GB (0.06GB free)
Process Argv	--crash-reporter-id 62586060-13a2-4d6f-9b3f-4b1de1d548e4
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
a9j8j154:30646983
962ge761:30959799
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
945dj816:31013170
dvdeprecation:31068756
dwnewjupytercf:31046870
newcmakeconfigv2:31071590
impr_priority:31102340
nativerepl1:31139838
pythonrstrctxt:31112756
cf971741:31144450
iacca1:31171482
notype1:31157159
5fd0e150:31155592
dwcopilot:31170013
h409b430:31179529

[pamelafox]

[tmeschter]

@pamelafox Version 0.3.42 is pretty old at this point, and we've made some updates that allow @Azure to make better use of the content of the learn.microsoft.com pages. Could you try the latest version and see if it provides a better response to this input?

[JasonYeMSFT]

@pamelafox Please share you latest findings if there is any. Our bot will auto-close the issue in two weeks if there is no new response.

[pamelafox]

Here's what I got today:

You can use GitHub models to call GPT-4o by using the Azure OpenAI Service. Here’s a quick guide to get started:

Set up the Environment:

Install required Python packages:
ipykernel
Configure Parameters:

Set variables for your Azure Search Service and OpenAI endpoints:
"
Using Azure OpenAI Service:

Ensure you have an Azure subscription and an Azure OpenAI Service resource with the GPT-4o model deployed. You can refer to the Azure OpenAI Quickstart Guide for more details.
Call the API:

Utilize the OpenAI client for interaction with GPT-4o using Python or other supported languages.
For code samples and further instructions, see the Azure AI documentation.

I think we might not have documentation about GitHub models, as it keeps directing me to use Azure OpenAI endpoints. So maybe that's the best we can get?

[pamelafox]

It isn't suggesting incorrect code anymore, it's just not able to fully answer the question, as it appears to not know about GitHub models.