Result on Hallucination with GPT2-XL

[JiaangL]

Hi, thanks for your nice work. I've downloaded the repo and tried to reimplement the results from the paper. But I've got different results on the Hallucination dataset with GPT2-XL. I'm running with the default hyperparameter setting in \config. The outputs are as follows:

......
[2024-03-06 01:27:31,849][trainer][INFO] - Number of edits: 1392
[2024-03-06 01:27:31,849][trainer][INFO] - [+edit results+]TRR: {'UP': 12.314838409423828}
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]ERR: {'HIS': 1.0030531883239746}
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]ES: 1.005843997001648
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]train_time: 0.29427483876546223
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]edit: ["This is a Wikipedia passage about edward synge archbishop of tuam. Edward Synge (1714–1798) was an Irish Anglican prelate who served as the Church of Ireland Archbishop of Tuam from 1781 to 1798. Synge was born in Dublin in 1714, the son of the Rev. Edward Synge, rector of St. Werburgh's Church, Dublin. He was educated at Trinity College, Dublin, and was ordained in 1737. He held livings at St. Werburgh's, Dublin, and at Kilmore, County Meath. He was appointed Dean of Clonfert in 1760 and Dean of St. Patrick's Cathedral, Dublin in 1763.", "This is a Wikipedia passage about edward synge archbishop of tuam. Edward Synge (1714–1798) was an Irish Anglican prelate who served as the Church of Ireland Archbishop of Tuam from 1781 to 1798. Synge was born in Dublin in 1714, the son of the Rev. Edward Synge, rector of St. Werburgh's Church, Dublin. He was educated at Trinity College, Dublin, and was ordained in 1737. He held livings at St. Werburgh's, Dublin, and at Kilmore, County Meath. He was appointed Dean of Clonfert in 1760 and Dean of St. Patrick's Cathedral, Dublin in 1763. In 1781 he was appointed Archbishop of Tuam, a post he held until his death in 1798.", "This is a Wikipedia passage about edward synge archbishop of tuam. Edward Synge (1714–1798) was an Irish Anglican prelate who served as the Church of Ireland Archbishop of Tuam from 1781 to 1798. Synge was born in Dublin in 1714, the son of the Rev. Edward Synge, rector of St. Werburgh's Church, Dublin. He was educated at Trinity College, Dublin, and was ordained in 1737. He held livings at St. Werburgh's, Dublin, and at Kilmore, County Meath. He was appointed Dean of Clonfert in 1760 and Dean of St. Patrick's Cathedral, Dublin in 1763. In 1781 he was appointed Archbishop of Tuam, a post he held until his death in 1798. Synge was a noted scholar and a friend of the philosopher Edmund Burke.", "This is a Wikipedia passage about edward synge archbishop of tuam. Edward Synge (1714–1798) was an Irish Anglican prelate who served as the Church of Ireland Archbishop of Tuam from 1781 to 1798. Synge was born in Dublin in 1714, the son of the Rev. Edward Synge, rector of St. Werburgh's Church, Dublin. He was educated at Trinity College, Dublin, and was ordained in 1737. He held livings at St. Werburgh's, Dublin, and at Kilmore, County Meath. He was appointed Dean of Clonfert in 1760 and Dean of St. Patrick's Cathedral, Dublin in 1763. In 1781 he was appointed Archbishop of Tuam, a post he held until his death in 1798. Synge was a noted scholar and a friend of the philosopher Edmund Burke. He was a strong supporter of the Church of Ireland and was an advocate of the union of the Anglican and Roman Catholic churches."]
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]edit_label: ['A renowned preacher, his works were frequently published and included an exhortation to frequent communion translated into Welsh.', 'Amongst other achievements, he established a dynasty of prominent ecclesiastics and literary figures closely integrated into the Protestant squirearchy in the west of Ireland.', 'He died in office on 23 July 1741, aged 82.', 'His sons were Edward Synge (Bishop of Elphin) and Nicholas Synge (Bishop of Killaloe).']
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]n_edits: 1392
[2024-03-06 01:27:31,850][trainer][INFO] - [+edit results+]ARR: 8.632816314697266
348it [13:37:53, 141.01s/it]

The Locality (TRR) and ARR are different from the original paper (12.31 vs 17.45, 8.63 vs 2.66). Is there anything I did wrong?
Moreover, after checking the data, I find only 516 accurate outputs in Hallucination. Could you double-check the dataset statistics, by checking the shape of the accurate dataset in this line? Thanks in advance!

[BruthYU]

I guess the hyper-parameter settings are changed during experiments on Further Analyses in our paper, I will release some running logs recording the correct training and inference processes.

[BruthYU]

Logs could be downloaded using Google Drive. Thanks for your attention, please feel free to contact us whenever you have other questions.