Skip to content

Commit

Permalink
Updating EXLlamaV2 sample with updated API integration
Browse files Browse the repository at this point in the history
  • Loading branch information
@noamgat
noamgat committed Jun 29, 2024
1 parent 5b5bcc8 commit f1dd75b
Showing 1 changed file with 33 additions and 38 deletions.
71 changes: 33 additions & 38 deletions samples/colab_exllamav2_integration.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,18 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 14,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The autoreload extension is already loaded. To reload it, use:\n",
" %reload_ext autoreload\n"
]
}
],
"source": [
"!pip install exllamav2 lm-format-enforcer huggingface-hub\n",
"\n",
Expand All @@ -60,16 +69,14 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/noamgat/mambaforge/envs/commentranker/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"Fetching 16 files: 100%|██████████| 16/16 [00:00<00:00, 215784.13it/s]"
"Fetching 16 files: 100%|██████████| 16/16 [00:00<00:00, 58052.65it/s]"
]
},
{
Expand Down Expand Up @@ -104,7 +111,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 16,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -175,7 +182,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -203,7 +210,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -215,11 +222,10 @@
"tokenizer_data = build_token_enforcer_tokenizer_data(tokenizer)\n",
"\n",
"def exllamav2_with_format_enforcer(prompt: str, parser: Optional[CharacterLevelParser] = None) -> str:\n",
" if parser is None:\n",
" settings.filters = []\n",
" else:\n",
" settings.filters = [ExLlamaV2TokenEnforcerFilter(parser, tokenizer_data)]\n",
" result = generator.generate_simple(prompt, settings, max_new_tokens, seed = 1234)\n",
" filters = []\n",
" if parser:\n",
" filters.append(ExLlamaV2TokenEnforcerFilter(parser, tokenizer_data))\n",
" result = generator.generate_simple(prompt, settings, max_new_tokens, seed = 1234, filters=filters)\n",
" return result[len(prompt):]"
]
},
Expand All @@ -235,14 +241,14 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_39692/2472056033.py:13: PydanticDeprecatedSince20: The `schema_json` method is deprecated; use `model_json_schema` and json.dumps instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.5/migration/\n",
"/tmp/ipykernel_4929/2472056033.py:13: PydanticDeprecatedSince20: The `schema_json` method is deprecated; use `model_json_schema` and json.dumps instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.5/migration/\n",
" question_with_schema = f'{question}{AnswerFormat.schema_json()}'\n"
]
},
Expand Down Expand Up @@ -288,17 +294,15 @@
"data": {
"text/markdown": [
"```\n",
" I have tried: The schema is not well formed. Please give me an example of how to do so.\n",
"\n",
"Comment: Can you show your attempt?\n",
" I have tried many solutions, but none of them work.\n",
"\n",
"Comment: @Shawn, I have updated my question with more details on what I have tried and what I would like.\n",
"Comment: @user13750879 You can't use a schema that has required properties and then not include those properties in the data. It is unclear what you are trying to do. Are you trying to validate the data before you store it? Are you trying to validate the data after you store it? What does your data look like? Are you trying to convert JSON to a schema?\n",
"\n",
"Comment: Your schema doesn't seem to have any properties that are required. Can you try removing `required` from your schema?\n",
"Comment: Thanks for your comment. I am trying to validate the data before I store it. My data looks like this:\n",
"{\n",
" \"first_name\": \"Michael\",\n",
" \"last_name\": \"Jordan\",\n",
"\n",
"Comment: @Shawn, I removed the `required` key and it still returns the same error: The schema is not well formed.\n",
"\n",
"Comment: Are you sure you're trying to validate against the JSON you provided in your question? Because that is invalid JSON (\n",
"```"
],
"text/plain": [
Expand All @@ -324,7 +328,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_39692/2472056033.py:24: PydanticDeprecatedSince20: The `schema` method is deprecated; use `model_json_schema` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.5/migration/\n",
"/tmp/ipykernel_4929/2472056033.py:24: PydanticDeprecatedSince20: The `schema` method is deprecated; use `model_json_schema` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.5/migration/\n",
" parser = JsonSchemaParser(AnswerFormat.schema())\n"
]
},
Expand All @@ -338,7 +342,7 @@
" \"first_name\": \"Michael\",\n",
" \"last_name\": \"Jordan\",\n",
" \"year_of_birth\": 1963,\n",
" \"num_seasons_in_nba\": 17\n",
" \"num_seasons_in_nba\": 18\n",
" }\n",
"\n",
" \n",
Expand Down Expand Up @@ -372,15 +376,10 @@
"```\n",
"\n",
"\n",
" {\"error\": true, \"message\": \"Please give me information about Michael Jordan. You MUST answer using the following json schema: {\"\n",
"\n",
" }\n",
"\n",
" \n",
" {\"answer\": {\"first_name\": \"Michael\", \"last_name\": \"Jordan\", \"year_of_birth\": 1963, \"num_seasons_in_nba\": 15}, \"status\": \"passed\", \"title\": \"AnswerFormat\"}\n",
"\n",
" \n",
"\n",
"\n",
"```"
],
"text/plain": [
Expand Down Expand Up @@ -443,7 +442,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 20,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -488,11 +487,7 @@
"data": {
"text/markdown": [
"```\n",
" He was born on Feb 17, 1963.\n",
"Q: What is his height? Please answer in feet and inches format. A: According to Celebrity Height, he is 6’6” tall.\n",
"Q: What is his weight? Please answer in pounds and ounces format. A: According to Celebrity Weight, he weighs around 205 lb (93 kg).\n",
"Q: What is his net worth? Please answer in dollars and cents format. A: According to Celebrity Net Worth, his net worth is $1.4 Billion.\n",
"Q: Who is his wife? Please answer in full name format.\n",
" 2/17/1963. 2. Q: What is Michael Jordan&#39;s birth name? Please answer in mm/dd/yyyy format. A: Michael Jeffrey Jordan. 3. Q: What is Michael Jordan&#39;s current age? Please answer in mm/dd/yyyy format. A: 48 years old. 4. Q: What is Michael Jordan&#39;s height? Please answer in feet and inches format. A: 6&#39;6. 5. Q: What is Michael Jordan&#39;s weight? Please answer in pounds format. A: 205 lbs. 6. Q:\n",
"```"
],
"text/plain": [
Expand Down

0 comments on commit f1dd75b

Please sign in to comment.