[Inference] Fix auth token and add models starcoder and llama2

[Deegue]

No description provided.

[KepingYan]

Why is env HF_ACCESS_TOKEN removed, is it no longer needed?

[KepingYan]

Please also help add **config.dict() at PeftModel.from_pretrained(model, model_desc.peft_model_id_or_path) and DeltaTunerModel.from_pretrained(model, model_desc.peft_model_id_or_path) in both transformer_predictor.py and deepspeed_predictor.py.

[Deegue]

Please also help add **config.dict() at PeftModel.from_pretrained(model, model_desc.peft_model_id_or_path) and DeltaTunerModel.from_pretrained(model, model_desc.peft_model_id_or_path) in both transformer_predictor.py and deepspeed_predictor.py.

IIUC, maybe we should add like L34 instead of dict() right?

[KepingYan]

IIUC, maybe we should add like L34 instead of dict() right?

OK. Thanks.

[Deegue]

Gentle ping @KepingYan for another review since all CI passed.
Will remove models "starcoder" and "llama-2-7b-chat-hf" before it is ready to merge.

[KepingYan]

use_auth_token cannot be written directly in config yaml, it needs to be set in CI file. @jiafuzha please help confirm this.

[jiafuzha]

yes, strictly speaking. we cannot. But it's only read-only key. If it can pass github security check, I think we can leave it for now.

[Deegue]

If the environment of our CI nodes have env.HF_ACCESS_TOKEN configured. I think I can try to get use_auth_token runtime from env instead of passing in yaml directly.

[jiafuzha]

yes, it's better.

Maybe we can have a unit test later to verify if use_auth_token is passed correctly. This ticket exposed and fixed the token bug in several places, which shows the right value of CI.

[Deegue]

Ok, removed them from CI. Let's merge this first and I will create a follow-up PR to hide auth token.

[Deegue]

All tests passed, auth token is removed from yaml. Removed starcoder and llama2 from CI. Ping @KepingYan @jiafuzha for review, thanks!

[jiafuzha]

All tests passed, auth token is removed from yaml. Removed starcoder and llama2 from CI. Ping @KepingYan @jiafuzha for review, thanks!

how is auth_token passed to CI? huggingface-cli login?

[Deegue]

All tests passed, auth token is removed from yaml. Removed starcoder and llama2 from CI. Ping @KepingYan @jiafuzha for review, thanks!

how is auth_token passed to CI? huggingface-cli login?

I used the environment variables env.HF_ACCESS_TOKEN to get and rewrite the yaml, to pass the auto token.

llm-on-ray/.github/workflows/workflow_inference.yml

Lines 112 to 130 in a3be1cd

	CMD=$(cat << EOF
	import yaml
	if ("${{ matrix.model }}" == "starcoder"):
	conf_path = "inference/models/starcoder.yaml"
	with open(conf_path, encoding="utf-8") as reader:
	result = yaml.load(reader, Loader=yaml.FullLoader)
	result['model_description']["config"]["use_auth_token"] = "${{ env.HF_ACCESS_TOKEN }}"
	with open(conf_path, 'w') as output:
	yaml.dump(result, output, sort_keys=False)
	if ("${{ matrix.model }}" == "llama-2-7b-chat-hf"):
	conf_path = "inference/models/llama-2-7b-chat-hf.yaml"
	with open(conf_path, encoding="utf-8") as reader:
	result = yaml.load(reader, Loader=yaml.FullLoader)
	result['model_description']["config"]["use_auth_token"] = "${{ env.HF_ACCESS_TOKEN }}"
	with open(conf_path, 'w') as output:
	yaml.dump(result, output, sort_keys=False)
	EOF
	)
	docker exec "${TARGET}" python -c "$CMD"

[Deegue]

Hi @KepingYan @jiafuzha , could you take a second look whether there are further comments?

[KepingYan]

Environment variable doesn’t seem to take effect, please add this step and try again.

llm-on-ray/.github/workflows/workflow_finetune.yml

Lines 66 to 67 in 6e32361

	- name: Load environment variables
	run: cat /root/actions-runner-config/.env >> $GITHUB_ENV

[Deegue]

Environment variable doesn’t seem to take effect, please add this step and try again.

llm-on-ray/.github/workflows/workflow_finetune.yml

Lines 66 to 67 in 6e32361

	- name: Load environment variables
	run: cat /root/actions-runner-config/.env >> $GITHUB_ENV

Thanks @KepingYan . It seems no env file on our CI nodes.

[Deegue]

All test passed, removed starcoder and llama-2-7b-chat-hf from CI. Ping @KepingYan for review.

[KepingYan]

LGTM