[2.x] Upgrade to LangChain v0.3 and Pydantic v2

[dlqqq]

Description

• Closes support langchain>=0.3 and migrate to use pydantic v2 #1003.
• Upgrades to langchain>=0.3.0,<0.4.0 and pydantic>=2.0,<3.0.
• Implements code changes required by this upgrade. These are detailed below in separate sections.
• Removes import-linter tool from the project, as it is no longer needed to block importing from pydantic directly.

cc @michaelchia @krassowski

Demo

Screen.Recording.2025-01-14.at.1.08.29.PM.mov

Notable Pydantic v2 changes

• In BaseModel, the .dict() and .json() methods have been deprecated in favor of .model_dump() and .model_dump_json() respectively.

• All Optional fields no longer default to a value of None in Pydantic v2. The default value of None must be explicitly set in the model definition, otherwise an exception is raised at runtime.

  • I used this regex to find all Optional fields lacking a default: Optional\[[\w\[\]\s,]*\]\s*$. I then added a default value of None to each.

• It is no longer possible to implement an attribute from a parent class via a property while also inheriting from BaseModel. Upstream issue: Failure to override an inherited field with a computed field pydantic/pydantic#7250

  • Therefore, I've had to drop BaseModel from the BoundedChatHistory classes. They now just directly inherit from BaseChatMessageHistory.

• The ModelMetaclass is no longer available from a public module, and should be considered for internal use by Pydantic only. Therefore this PR no longer subclasses ModelMetaclass and reverts LangChain v0.1.0 #572.

  • The original motivation was a bug which caused the name field on each provider to be resolved to ... (a literal Ellipses object). This is documented in Pin langchain-core dependency to prevent Settings UI crash #558.
  • I've verified that this bug no longer occurs, as I am able to open the Jupyter AI settings.

• AFAIK, it is no longer possible to verify server_settings: ClassVar[MappingProxyType] is only set once. I've removed the associated test.

  • Previously this was implemented in the custom metaclass derived from ModelMetaclass used by Pydantic internally (Prevent overriding server_settings on base provider class #825). However, in Pydantic v2, this metaclass is not exported by a public module, and therefore should not be subclassed.
  • Model & field validators can't meet this use-case either, because ClassVar attributes are not treated as fields by Pydantic: no error when using ClassVar and Field pydantic/pydantic#10169
  • I believe the long term solution is to avoid using ClassVar attributes as much as possible. The behavior of ClassVars isn't as well-defined as the behavior of other fields. For example, instead of using ClassVars to track globals, we should probably just store them as instance fields on singleton objects, e.g. ConfigManager.

Notable LangChain changes

Surprisingly, I didn't notice any which impact Jupyter AI. Reviewers, please let me know if I missed something.

For reviewers

• Please test the application as much as possible. Both Pydantic & LangChain are used pervasively throughout the codebase, and both have introduced breaking changes since the previous version set.

• What I've tested:

  • I can change my chat & embedding model freely. Haven't tested more advanced scenarios such as typing a custom model ID or adding model fields.
  • Chatting with OpenAI & Anthropic work as expected.
  • All slash commands work as expected.
  • Stopping an AI stream response works as expected.

[dlqqq]

Ah, I forgot to update the docs. I'll open a new issue to track this for fast follow-up, since I've already ported these changes to the main branch in #1201.

New issue: #1200.

[dlqqq]

Hey folks, our team of 5 engineers performed a 30 minute bug bash on this PR and didn't find any new bugs or regressions. Therefore, I will merge this PR into 2.x and include this in the v2.29.0 release. We are hoping to publish this release tomorrow. 👍