<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:none;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brian Coghlan and I have been pleasantly surprised by the interest aroused by, and the positive comments we have received on, the little paper "ChatGPT’s Astonishing Fabrications About Percy Ludgate" that we published in the April-June
2023 issue of the IEEE Annals of the History of Computing. The study [1] that we reported on was a temporary diversion from our main interests, but we are curious to know whether the startlingly high level of fabrications (so-called "hallucinations") in the
answers we obtained from ChatGPT is typical of its performance when used as a historian's assistant. It would be great if someone with appropriate skills and resources (research students!) could organise and carry out a serious statistically-valid evaluation
of the trustworthiness of the answers provided by ChatGPT and the like to a wide range of questions on the history of computing, carefully checking (i) the existence of any citations listed, (ii) the accuracy of all the factual statements made, and (iii)
whether the errors found were likely to be because of errors in the learning data, or due to ChatGPT's text selection and generation strategy.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">One of the motivations we had for our study was the enthusiastic and uncritical assessments of ChatGPT's merits as a historian's assistant that we found on the web. We have since been amused and embarrassed to find that the one article
we quoted [2] in our paper was almost certainly generated by ChatGPT itself. It turns out that the alleged author, Martin Frackiewicz, has been posting on average close to twenty sizeable articles a day since mid-March on his company's website in praise of
ChatGPT.<span class="apple-converted-space"> </span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">A much more credible favourable account of the merits of ChatGPT and the like has been provided by Mark Humphries and Eric Story [3]. This is very optimistic about the ability of historians to use such systems in ways that achieve high
levels of veracity, and to identify such errors as are nevertheless made. It would be good if such hopes could be validated - or debunked - before large amounts of ChatGPT-generated text on the history of computing was published and added to future sets of
learning data.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Cheers</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brian Randell</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">1. ChatGPT’s Astonishing Fabrications about Percy Ludgate -<span class="apple-converted-space"> </span><a href="https://treasures.scss.tcd.ie/miscellany/TCD-SCSS-X.20121208.002/ChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf" title="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftreasures.scss.tcd.ie%2Fmiscellany%2FTCD-SCSS-X.20121208.002%2FChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf&data=05%7C01%7Cbrian.randell%40newca"><span style="color:#0078D7">https://treasures.scss.tcd.ie/miscellany/TCD-SCSS-X.20121208.002/ChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf</span></a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">2: ChatGPT-4: A Valuable Tool for Historical Research and Analysis -<span class="apple-converted-space"> </span><a href="https://ts2.space/en/chatgpt-4-a-valuable-tool-for-historical-research-and-analysis/" title="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fts2.space%2Fen%2Fchatgpt-4-a-valuable-tool-for-historical-research-and-analysis%2F&data=05%7C01%7Cbrian.randell%40newcastle.ac.uk%7C012d5084d1964965c49b08db87a98703%7C9c5012c9b61644c2a91766"><span style="color:#0078D7">https://ts2.space/en/chatgpt-4-a-valuable-tool-for-historical-research-and-analysis/</span></a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">3. Today’s AI, Tomorrow’s History: Doing History in the Age of ChatGPT.<span class="apple-converted-space"> </span><a href="https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/" title="Original URL:
https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/
Click to follow link."><span style="color:#0078D7">https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/</span></a></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none;mso-fareast-language:EN-GB">—<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none;mso-fareast-language:EN-GB">School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none;mso-fareast-language:EN-GB">EMAIL = Brian.Randell@ncl.ac.uk PHONE = +44 191 208 7923<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none;mso-fareast-language:EN-GB">URL = https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html</span></p>
</div>
</body>
</html>