[SIGCIS-Members] On the Merits of ChatGPT and the like as Historian's Assistants

[Brian Randell]

Hi:

Brian Coghlan and I have been pleasantly surprised by the interest aroused by, and the positive comments we have received on, the little paper "ChatGPT’s Astonishing Fabrications About Percy Ludgate" that we published in the April-June 2023 issue of the IEEE Annals of the History of Computing. The study [1] that we reported on was a temporary diversion from our main interests, but we are curious to know whether the startlingly high level of fabrications (so-called "hallucinations") in the answers we obtained from ChatGPT is typical of its performance when used as a historian's assistant. It would be great if someone with appropriate skills and resources (research students!) could organise and carry out a serious statistically-valid evaluation of the trustworthiness of the answers provided by ChatGPT and the like to a wide range of  questions on the history of computing, carefully checking (i) the existence of any citations listed, (ii) the accuracy of all the factual statements made, and (iii) whether the errors found were likely to be because of errors in the learning data, or due to ChatGPT's text selection and generation strategy.

One of the motivations we had for our study was the enthusiastic and uncritical assessments of ChatGPT's merits as a historian's assistant that we found on the web. We have since been amused and embarrassed to find that the one article we quoted [2] in our paper was almost certainly generated by ChatGPT itself. It turns out that the alleged author, Martin Frackiewicz, has been posting on average close to twenty sizeable articles a day since mid-March on his company's website in praise of ChatGPT.

A much more credible favourable account of the merits of ChatGPT and the like has been provided by Mark Humphries and Eric Story [3]. This is very optimistic about the ability of historians to use such systems in ways that achieve high levels of veracity, and to identify such errors as are nevertheless made. It would be good if such hopes could be validated - or debunked - before large amounts of ChatGPT-generated text on the history of computing was published and added to future sets of learning data.

Cheers

Brian Randell

1. ChatGPT’s Astonishing Fabrications about Percy Ludgate - https://treasures.scss.tcd.ie/miscellany/TCD-SCSS-X.20121208.002/ChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf

2: ChatGPT-4: A Valuable Tool for Historical Research and Analysis - https://ts2.space/en/chatgpt-4-a-valuable-tool-for-historical-research-and-analysis/

3. Today’s AI, Tomorrow’s History: Doing History in the Age of ChatGPT. https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/



—

School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG
EMAIL = Brian.Randell at ncl.ac.uk   PHONE = +44 191 208 7923
URL =  https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sigcis.org/pipermail/members-sigcis.org/attachments/20230718/b0f1d291/attachment.htm>