xFinance tuning example and methodology details

[mjsteele12]

Are there any examples to reproduce the xFinance result? I.e., examples of using a TextDataset for the domain knowledge/ unsupervised fine tuning and then InstructionDataset for instruct tuning? I have many questions on the xFinance methodology such as:

1. Was the finetuning done using PEFT/LoRa (for both text and instruct tuning)?
2. How were subsequent fine-tunings completed? i.e. if the answer to 1 is that LoRa was used, how did you continue training the LoRa checkpoints?
3. How many epochs on each text/instruct dataset?
4. What LoRa rank was used?
5. What target_modules were used for each tuning?

I have more questions, but any insight into the XFinance methodology would be greatly appreciated!

[StochasticRomanAgeev]

Hi @mjsteele12,
Thanks for interest in our model!
We have a blogpost about it where we describe the process of data collection, finetuning and evaluation.

[ma-ji]

5. What target_modules were used for each tuning?

Having the same question and found this: https://github.com/stochasticai/xTuring/blob/7d6413207b3a3d8491549873cafa435bbed261b9/src/xturing/engines/llama2_engine.py#L61C16-L61C16