<signature id="signaturefinder_x-mail-compose1689890321276">
<div>
Might be of interest to you:
</div><div><br></div><div><a href="https://cacm.acm.org/blogs/blog-cacm/274694-is-bard-better-than-chatgpt/fulltext">Is Bard Better than ChatGPT? | blog@CACM | Communications of the ACM</a></div>
<div>
<br>
</div>
<div>
Best wishes,</div><div>Herbert</div><div><br></div>
</signature>
<blockquote style="margin-right: 0.0px;margin-left: 15.0px;">
----Ursprüngliche Nachricht----
<br>Von : members@lists.sigcis.org
<br>Datum : 18/07/2023 - 22:12 (MS)
<br>An : brian.randell@newcastle.ac.uk
<br>Cc : members@lists.sigcis.org
<br>Betreff : Re: [SIGCIS-Members] On the Merits of ChatGPT and the like as Historian's Assistants
<br>
<br>
<div dir="auto">
I found Donald Knuth’s reflections in this topic informative:
<div>
<a href="https://www-cs-faculty.stanford.edu/~knuth/chatGPT20.txt">https://www-cs-faculty.stanford.edu/~knuth/chatGPT20.txt</a>
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
My two cents:
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
The capacity for hallucination alongside what appears to be appropriate identification of context or nuance casts a veil of fabulism over all the output, which (IMO) makes models like these unreliable in ways that even Procopius was not. Given that problem, a model which more often returns correct information might well be considered *less* trustworthy overall as an assistant.
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
On Tue, Jul 18, 2023 at 12:49 PM Brian Randell via Members <
<a style="cursor: pointer;text-decoration: underline;color: blue;">members@lists.sigcis.org</a>> wrote:
<br>
</div>
</div>
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left-width: 1.0px;border-left-style: solid;padding-left: 1.0ex;border-left-color: rgb(204,204,204);">
<div lang="EN-GB" link="#0563C1" vlink="#954F72" style="overflow-wrap: break-word;">
<div class="m_8284772541529463063WordSection1">
<p class="MsoNormal">Hi:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brian Coghlan and I have been pleasantly surprised by the interest aroused by, and the positive comments we have received on, the little paper "ChatGPT’s Astonishing Fabrications About Percy Ludgate" that we published in the April-June 2023 issue of the IEEE Annals of the History of Computing. The study [1] that we reported on was a temporary diversion from our main interests, but we are curious to know whether the startlingly high level of fabrications (so-called "hallucinations") in the answers we obtained from ChatGPT is typical of its performance when used as a historian's assistant. It would be great if someone with appropriate skills and resources (research students!) could organise and carry out a serious statistically-valid evaluation of the trustworthiness of the answers provided by ChatGPT and the like to a wide range of questions on the history of computing, carefully checking (i) the existence of any citations listed, (ii) the accuracy of all the factual statements made, and (iii) whether the errors found were likely to be because of errors in the learning data, or due to ChatGPT's text selection and generation strategy.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">One of the motivations we had for our study was the enthusiastic and uncritical assessments of ChatGPT's merits as a historian's assistant that we found on the web. We have since been amused and embarrassed to find that the one article we quoted [2] in our paper was almost certainly generated by ChatGPT itself. It turns out that the alleged author, Martin Frackiewicz, has been posting on average close to twenty sizeable articles a day since mid-March on his company's website in praise of ChatGPT.<span class="m_8284772541529463063apple-converted-space"> </span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">A much more credible favourable account of the merits of ChatGPT and the like has been provided by Mark Humphries and Eric Story [3]. This is very optimistic about the ability of historians to use such systems in ways that achieve high levels of veracity, and to identify such errors as are nevertheless made. It would be good if such hopes could be validated - or debunked - before large amounts of ChatGPT-generated text on the history of computing was published and added to future sets of learning data.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Cheers</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brian Randell</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">1. ChatGPT’s Astonishing Fabrications about Percy Ludgate -<span class="m_8284772541529463063apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://treasures.scss.tcd.ie/miscellany/TCD-SCSS-X.20121208.002/ChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxlo51uk7E$" title="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftreasures.scss.tcd.ie%2Fmiscellany%2FTCD-SCSS-X.20121208.002%2FChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf&data=05%7C01%7Cbrian.randell%40newca"><span style="color: rgb(0,120,215);">https://treasures.scss.tcd.ie/miscellany/TCD-SCSS-X.20121208.002/ChatGPTs-AstonishingFabrications-aboutPercyLudgate-CoghlanRandellOBoyle-20230424-1434.pdf</span></a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">2: ChatGPT-4: A Valuable Tool for Historical Research and Analysis -<span class="m_8284772541529463063apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://ts2.space/en/chatgpt-4-a-valuable-tool-for-historical-research-and-analysis/__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxllUwtCeV$" title="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fts2.space%2Fen%2Fchatgpt-4-a-valuable-tool-for-historical-research-and-analysis%2F&data=05%7C01%7Cbrian.randell%40newcastle.ac.uk%7C012d5084d1964965c49b08db87a98703%7C9c5012c9b61644c2a91766"><span style="color: rgb(0,120,215);">https://ts2.space/en/chatgpt-4-a-valuable-tool-for-historical-research-and-analysis/</span></a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">3. Today’s AI, Tomorrow’s History: Doing History in the Age of ChatGPT.<span class="m_8284772541529463063apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxln7isyWK$" title="Original URL:
https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/
Click to follow link."><span style="color: rgb(0,120,215);">https://activehistory.ca/2023/03/todays-ai-tomorrows-history-doing-history-in-the-age-of-chatgpt/</span></a></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 9.0pt;font-family: Helvetica;color: black;">—<u style="font-family: Helvetica;"></u><u style="font-family: Helvetica;"></u></span></p>
<p class="MsoNormal"><span style="font-size: 9.0pt;font-family: Helvetica;color: black;"><u style="font-family: Helvetica;"></u> <u style="font-family: Helvetica;"></u></span></p>
<p class="MsoNormal"><span style="font-size: 9.0pt;font-family: Helvetica;color: black;">School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG<u style="font-family: Helvetica;"></u><u style="font-family: Helvetica;"></u></span></p>
<p class="MsoNormal"><span style="font-size: 9.0pt;font-family: Helvetica;color: black;">EMAIL = <a style="cursor: pointer;text-decoration: underline;color: blue;">Brian.Randell@ncl.ac.uk</a> PHONE = +44 191 208 7923<u style="font-family: Helvetica;"></u><u style="font-family: Helvetica;"></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size: 9.0pt;font-family: Helvetica;color: black;">URL = <a href="https://urldefense.com/v3/__https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxlgDUs0H9$" style="font-family: Helvetica;">https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html</a></span></p>
</div>
</div> _______________________________________________
<br> This email is relayed from members at
<a href="http://sigcis.org" rel="noreferrer">sigcis.org</a>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at
<a href="https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis.org/__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxlq_2gkLS$" rel="noreferrer">https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis.org/__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxlq_2gkLS$</a> and you can change your subscription options at
<a href="https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sigcis.org__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxllcliWn4$" rel="noreferrer">https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sigcis.org__;!!K-Hz7m0Vt54!hwCqcmqZeYw_ZiA1gvEYeZcwiAuicJxRcaqmrqsshs1rFMcb_eTH4-FNydNlg1cciaPHtA6lKPrxllcliWn4$</a>
<br>
</blockquote>
</div>
</div>
<span class="gmail_signature_prefix">-- </span>
<br>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
Adam Hyland (
<i>he/him)</i>
<div>
<a href="https://adampunk.com">adampunk.com</a>
</div>
<div>
UW HCDE PhD Student
</div>
</div>
</div>
<br>
</blockquote>
<br>
<p></p>