Hi folks The piece Brian mentions having written is B. Randell and B. Coghlan, "ChatGPT's Astonishing Fabrications About Percy Ludgate" in IEEE Annals of the History of Computing, vol. 45, no. 02, pp. 71-72, April-June 2023, doi: 10.1109/MAHC.2023.3272989. (it appears to be Open Access at the Xplore link https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10148832). Personally, my view is that no matter how good the answers these machine learning systems give us, they’re not worth the cost. Quite apart from the well-documented energy/water/environmental impact (which must surely be our most important criterion when making any decision, these days), and the social impacts (including educational, where ML has hit me personally the hardest), from a scholarly perspective we have a duty to be critical of our sources—and when they come from a LLM we simply can’t interrogate them properly. As the editor of Annals, I need to follow the IEEE’s publication guidelines on the use of machine learning systems in the production of manuscripts: it is mandatory that authors declare if they have ML-generated text, but not if they have used a tool to check/correct their spelling or grammar. Clearly this is a bit of a strange distinction: if an author accepts a Grammarly-rewrite of a sentence, aren’t they including ML-generated text? What I have not found so far is whether there is any requirement to declare the use of ML-based tools in their research. Again, there may be grey areas—I’m pretty sure Preview on my Mac is automatically OCR’ing text in all my scanned PDFs—that’s probably ML-based but I can’t do much about it—but it feels to me like there there is a distinction between that and asking a chatbot to do your research for you. As a community of scholars we will have to set our own standards here. While my personal view is always going to be anti-ML, I will do my best to listen openly and make sure the community is served as best as possible by Annals. Best, Dr. Troy Kaighin Astarte (they/them / nhw) I often dictate messages due to motor disability; please forgive any oddities resulting. Lecturer, Computer Science / Darlithydd, Cyfrifiadureg Swansea University / Prifysgol Abertawe Editor-in-Chief / Prif Olygydd, IEEE Annals of the History of Computing For students: my drop in hours are on the Intranet<https://fse-intranet.swan.ac.uk/intranet/staff_officehours?selected_staff_id=203842> (office CoFo 407) I fyfyrwyr: mae fy oriau swyddfa ar y fewnrwyd. Meeting booking: via Office Booking<https://outlook.office.com/bookwithme/user/8e101a47e22e4af793d033901758d0e4@Swansea.ac.uk/meetingtype/SVRwCe7HMUGxuT6WGxi68g2?anonymous&ep=mlink>. Zoom office: https://swanseauniversity.zoom.us/my/t.k.astarte Every email has a cost to the climate. Please think before sending short emails. Mae gan bob e-bost gost i’r hinsawdd. Meddyliwch cyn i chi anfon e-byst byr. Yes, I have switched my default message font. Do you like it? On 6 May 2025, at 23:39, Ceruzzi, Paul via Members <members@lists.sigcis.org> wrote: CAUTION: This email originated from outside of Swansea University. Do not click links or open attachments unless you recognise the sender and know the content is safe. RHYBUDD: Daeth yr e-bost hwn o'r tu allan i Brifysgol Abertawe. Peidiwch â chlicio ar atodiadau neu agor atodiadau oni bai eich bod chi'n adnabod yr anfonwr a'ch bod yn gwybod bod y cynnwys yn ddiogel. CAUTION: This email originated from outside of Swansea University. Do not click links or open attachments unless you recognise the sender and know the content is safe. RHYBUDD: Daeth yr e-bost hwn o'r tu allan i Brifysgol Abertawe. Peidiwch â chlicio ar atodiadau neu agor atodiadau oni bai eich bod chi'n adnabod yr anfonwr a'ch bod yn gwybod bod y cynnwys yn ddiogel. Thank you for bringing up this thread. Agree that there has been a great advance in the past year, of which I was unaware. Not as easy as it once was to get the programs to hallucinate. I was chatting with a Google employee the other day. He works in the Reston, Virginia office and was familiar with my book on Tysons Corner ("Internet Alley"). I told him of my Substack posts on the concentration of Data Centers in neighboring Ashburn. He suggested that I write a second edition of the book to cover this topic. I told him that as a retiree I no longer have the energy or stamina to write another book. He said (paraphrasing): "Let Google Gemini write it for you. It will do an excellent job. No problem, as long as you supervise it, and acknowledge how the Second Edition was written." As Jack Benny once said, "I'm thinking it over." Paul Ceruzzi Substack.com/@paulceruzzi<http://substack.com/@paulceruzzi> ________________________________ From: Members <members-bounces@lists.sigcis.org<mailto:members-bounces@lists.sigcis.org>> on behalf of Adam Hyland via Members <members@lists.sigcis.org<mailto:members@lists.sigcis.org>> Sent: Tuesday, May 6, 2025 6:15 PM To: Brian Randell <brian.randell@newcastle.ac.uk<mailto:brian.randell@newcastle.ac.uk>> Cc: Sigcis <members@sigcis.org<mailto:members@sigcis.org>> Subject: Re: [SIGCIS-Members] Perplexity External Email - Exercise Caution Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here ( https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-oxneW37nS9.bA...) and should be available to anyone with the URL. I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom. One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month. -Adam Adam Hyland (he/him) adampunk.com<https://adampunk.com/> UW HCDE PhD Student On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members <members@lists.sigcis.org<mailto:members@lists.sigcis.org>> wrote: Hi: A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate. I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system. Perplexity itself claims: Here's what makes Perplexity different Answers that are accurate and always cited We continuously search the internet and identify the best sources, from academic research to Reddit threads, to provide the perfect answer to any question. Citations in every response Every answer uses cited sources to provide a more accurate and comprehensive answer. If you want to dig deeper, just click the link to the source. See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”: https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA<https://urldefense.com/v3/__https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$> The web version limits the number of questions per day – so far the iPhone App hasn’t. I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS. However, the Wikipedia article about it https://en.wikipedia.org/wiki/Perplexity_AI<https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$> is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards. Cheers Brian _______________________________________________ This email is relayed from members at sigcis.org<http://sigcis.org/>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis... and you can change your subscription options at https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sig... _______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

For those who are interested, my co-author and I contributed a piece to _Management and Organizational History_ that looks at some of the methodological opportunities at the intersection of ML and history: Villamor Martin, M., Kirsch, D. A., & Prieto-Nanez, F. (2023). The promise of machine-learning-driven text analysis techniques for historical research: topic modeling and word embedding. *Management & Organizational History*. https://www.tandfonline.com/doi/abs/10.1080/17449359.2023.2181184 - david k. On Wed, May 7, 2025 at 7:44 AM James Cortada via Members < members@lists.sigcis.org> wrote:

If I may add, I am personally being driven by two realities in my work.

First, AI tools are still not as reliable as sources of accurate information and logic grounded in realities as all the hype would suggest these are. That is my reality in 2025-2026. If all that changes over time then I will revisit how to use AI, but not until then, just as in the 1970s we worried about the accuracy and relevance of children using handheld H-P calculators, etc.

My second reality is that I ultimately and always am personally responsible for whatever I publish--not a publisher, not an editor, not another historian--but me. It is my ethical and moral responsibility to offer up my works, my thoughts, my best efforts adhering to the highest standards of scholarship that I can muster. So, no matter what tools I use, I as a human ultimately am responsible. If I were to invent a footnote, inject some hallucination nonsense purposefully into something I write, or even unknowingly, shame on me because these actions would be dishonest. My second reality is increasingly of concern to me as I slowly evolve my research and writing to topics that are not as monographically narrowly focused, moving toward more tacit knowledge topics where logic, facts, and boundaries with truth become increasingly fuzzy and perhaps less knowledgeable and describable. For such topics I have to eschew the use of AI for the foreseeable future and rely on my aging brain but absolutely current release of Microsoft Word to go about my work.

I think the entire IEEE community is working through similar topics. Read recent issues of *Computer, Edge, *and *Spectrum*--as I do as part of the benefits of my expensive IEEE membership--and you will see that Troy is right (so far) about it not worth the cost of using such tools yet, but by implication, too, that we need to learn about this technology as it unfolds. Most of the history journals and book publishers I work with are also just now beginning to learn even how to spell AI. I consider our discussion as members of *Annals* about the subject ahead of those being held elsewhere and so I find it most interesting for which I thank all of you and Troy.

Jim

On Wed, May 7, 2025 at 3:46 AM Troy Astarte via Members < members@lists.sigcis.org> wrote:

...

Hi folks

The piece Brian mentions having written is B. Randell and B. Coghlan, "ChatGPT's Astonishing Fabrications About Percy Ludgate" in *IEEE Annals of the History of Computing*, vol. 45, no. 02, pp. 71-72, April-June 2023, doi: 10.1109/MAHC.2023.3272989. (it appears to be Open Access at the Xplore link https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10148832).

Personally, my view is that no matter how good the answers these machine learning systems give us, *they’re not worth the cost. *Quite apart from the well-documented energy/water/environmental impact (which must surely be our most important criterion when making any decision, these days), and the social impacts (including educational, where ML has hit me personally the hardest), from a scholarly perspective we have a duty to be critical of our sources—and when they come from a LLM we simply can’t interrogate them properly.

As the editor of *Annals*, I need to follow the IEEE’s publication guidelines on the use of machine learning systems in the production of manuscripts: it is mandatory that authors declare if they have ML-generated text, but not if they have used a tool to check/correct their spelling or grammar. Clearly this is a bit of a strange distinction: if an author accepts a Grammarly-rewrite of a sentence, aren’t they including ML-generated text?

What I have not found *so far *is whether there is any requirement to declare the use of ML-based tools in their research. Again, there may be grey areas—I’m pretty sure Preview on my Mac is automatically OCR’ing text in all my scanned PDFs—that’s probably ML-based but I can’t do much about it—but it feels to me like there there is a distinction between that and asking a chatbot to do your research for you.

As a community of scholars we will have to set our own standards here. While my personal view is always going to be anti-ML, I will do my best to listen openly and make sure the community is served as best as possible by *Annals*.

Best,

Dr. Troy Kaighin Astarte (they/them / nhw)

I often dictate messages due to motor disability; please forgive any oddities resulting.

Lecturer, Computer Science / Darlithydd, Cyfrifiadureg Swansea University / Prifysgol Abertawe Editor-in-Chief / Prif Olygydd, *IEEE Annals of the History of Computing*

For students: my drop in hours are on the Intranet <https://fse-intranet.swan.ac.uk/intranet/staff_officehours?selected_staff_id=203842> (office CoFo 407) I fyfyrwyr: mae fy oriau swyddfa ar y fewnrwyd. Meeting booking: via Office Booking <https://outlook.office.com/bookwithme/user/8e101a47e22e4af793d033901758d0e4@Swansea.ac.uk/meetingtype/SVRwCe7HMUGxuT6WGxi68g2?anonymous&ep=mlink> . Zoom office: https://swanseauniversity.zoom.us/my/t.k.astarte

Every email has a cost to the climate. Please think before sending short emails. Mae gan bob e-bost gost i’r hinsawdd. Meddyliwch cyn i chi anfon e-byst byr.

Yes, I have switched my default message font. Do you like it?

On 6 May 2025, at 23:39, Ceruzzi, Paul via Members < members@lists.sigcis.org> wrote:

*CAUTION:* This email originated from outside of Swansea University. Do not click links or open attachments unless you recognise the sender and know the content is safe.

*RHYBUDD:* Daeth yr e-bost hwn o'r tu allan i Brifysgol Abertawe. Peidiwch â chlicio ar atodiadau neu agor atodiadau oni bai eich bod chi'n adnabod yr anfonwr a'ch bod yn gwybod bod y cynnwys yn ddiogel. *CAUTION:* This email originated from outside of Swansea University. Do not click links or open attachments unless you recognise the sender and know the content is safe.

*RHYBUDD:* Daeth yr e-bost hwn o'r tu allan i Brifysgol Abertawe. Peidiwch â chlicio ar atodiadau neu agor atodiadau oni bai eich bod chi'n adnabod yr anfonwr a'ch bod yn gwybod bod y cynnwys yn ddiogel. Thank you for bringing up this thread. Agree that there has been a great advance in the past year, of which I was unaware. Not as easy as it once was to get the programs to hallucinate.

I was chatting with a Google employee the other day. He works in the Reston, Virginia office and was familiar with my book on Tysons Corner ("Internet Alley"). I told him of my Substack posts on the concentration of Data Centers in neighboring Ashburn. He suggested that I write a second edition of the book to cover this topic. I told him that as a retiree I no longer have the energy or stamina to write another book. He said (paraphrasing): "Let Google Gemini write it for you. It will do an excellent job. No problem, as long as you supervise it, and acknowledge how the Second Edition was written."

As Jack Benny once said, "I'm thinking it over."

Paul Ceruzzi Substack.com/@paulceruzzi <http://substack.com/@paulceruzzi> ------------------------------ *From:* Members <members-bounces@lists.sigcis.org> on behalf of Adam Hyland via Members <members@lists.sigcis.org> *Sent:* Tuesday, May 6, 2025 6:15 PM *To:* Brian Randell <brian.randell@newcastle.ac.uk> *Cc:* Sigcis <members@sigcis.org> *Subject:* Re: [SIGCIS-Members] Perplexity

*External Email - Exercise Caution* Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here (

https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-oxneW37nS9.bA...) and should be available to anyone with the URL.

I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom.

One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month.

-Adam

Adam Hyland (*he/him)* adampunk.com UW HCDE PhD Student

On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members < members@lists.sigcis.org> wrote:

Hi:

A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate.

I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system.

Perplexity itself claims:

Here's what makes Perplexity different

Answers that are accurate and always cited We continuously search the internet and identify the best sources, from academic research to Reddit threads, to provide the perfect answer to any question.

Citations in every response Every answer uses cited sources to provide a more accurate and comprehensive answer. If you want to dig deeper, just click the link to the source.

See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”:

https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0... <https://urldefense.com/v3/__https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$>

The web version limits the number of questions per day – so far the iPhone App hasn’t.

I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS.

However, the Wikipedia article about it https://en.wikipedia.org/wiki/Perplexity_AI <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$> is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards.

Cheers

Brian _______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis... and you can change your subscription options at https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sig...

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

-- James W. Cortada Senior Research Fellow Charles Babbage Institute University of Minnesota jcortada@umn.edu 608-274-6382 _______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

This has happened before: the embrace of spreadsheets like Lotus 1-2-3 by users who were seduced by their power on a PC desktop may have led to the financial excesses of the late 1990s. https://www.researchgate.net/publication/45869772_Spreadsheets_and_the_Finan... Paul Ceruzzi ________________________________ From: Brian E Carpenter <brian.e.carpenter@gmail.com> Sent: Wednesday, May 7, 2025 5:00 PM To: Troy Astarte <t.k.astarte@swansea.ac.uk>; Ceruzzi, Paul <CeruzziP@si.edu> Cc: Sigcis <members@sigcis.org> Subject: Re: [SIGCIS-Members] Perplexity, or ML tools in historical research External Email - Exercise Caution Cherry-picking one sentence from Troy:

What I have not found so far is whether there is any requirement to declare the use of ML-based tools in their research.

Since Google now starts with an "AI summary", even Google searches would need to be declared under such a rule. (Try googling "Does Google AI hallucinate?") DuckDuckGo is heading in the same direction, but more carefully: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspreadprivacy.com%2Fduckassist-launch%2F&data=05%7C02%7CCeruzziP%40si.edu%7Cea9a69451bab48abb56408dd8daa2fec%7C989b5e2a14e44efe93b78cdd5fc5d11c%7C0%7C0%7C638822484297266330%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0OB7RjvvFEsBhJ7x09%2BgqX%2FBszm5Z%2F6OZKtXQDdmAFc%3D&reserved=0<https://spreadprivacy.com/duckassist-launch/> Regards Brian Carpenter

Hi Adam: Thanks – I hadn’t realised that URL links to Perplexity’s answers aren’t transferable. It is certainly the case that Perplexity (which is based on ChatGPT4 I believe) is much more “knowledgeable” about Percy Ludgate than ChatGPT was – not least because it has ingested subsequent additions to the Internet. Indeed, given the huge amount of effort and money being poured into LLM research we surely should expect the latest models to be significantly improved. And Perplexity’s answer to my question “Are the references that Perplexity provides always real ones” was very reasonable, one might even say “thoughtful”, though its cautionary comments have of course to be applied to this answer itself – a nice example of recursion. However, the NY Times had a very interesting, and surely worrying (for the Artificial Intelligentsia, especially) piece yesterday entitled: “A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse: A new wave of “reasoning” systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.” I hope this link to it will work: https://www.nytimes.com/2025/05/05/technology/ai-hallucinations-chatgpt-google.html?unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share And on the subject of “hallucinations” (or rather “fabrications”), I also strongly recommend Gary Marcus’s very recent piece: Why DO large language models hallucinate?<https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate> (https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate) Let me end by quoting from it: “Because LLMS statistically mimic the language people have used, they often fool people into thinking that they operate like people. But they don’t operate like people. They don’t, for example, ever fact check (as humans sometimes, when well motivated, do). They mimic the kinds of things of people say in various contexts. And that’s essentially all they do. . . Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates <https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/> more<https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/> than some its predecesssors<https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/>. The chronic problem with creating fake citations in research papers<https://www.forbes.com/sites/larsdaniel/2025/01/29/the-irony-ai-experts-testimony--collapses-over-fake-ai-citations/> and faked cases in legal briefs<https://www.rawstory.com/lindell-dominion-2671843170/> is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is the rampant problem with numerical errors in financial reports<https://www.vals.ai/benchmarks/finance_agent-04-22-2025>, documented in a recent benchmark. Just how bad is it? One recent study showed rates of hallucinations of between 15% and 60% <https://research.aimultiple.com/ai-hallucination/> across various models on a benchmark of 60 questions that were easily verifiable relative to easily found CNN source articles that were directly supplied in the exam. Even the best performance (15% hallucination rate) is, relative to an open-book exam with sources supplied, pathetic. That same study reports that, “According to Deloitte, 77% of businesses who joined the study are concerned about AI hallucinations”. If I can be blunt, it is an absolute embarrassment that a technology that has collectively cost about half a trillion dollars can’t do something as basic as (reliably) check its output against wikipedia or a CNN article that is handed on a silver plattter. But LLMs still cannot - and on their own may never be able to — reliably do even things that basic.” Cheers Brian -- School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG EMAIL = Brian.Randell@ncl.ac.uk<mailto:Brian.Randell@ncl.ac.uk> PHONE = +44 (0)786 7805578 URL = https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html From: Adam Hyland <achyland@uw.edu> Date: Tuesday, 6 May 2025 at 23:15 To: Brian Randell <brian.randell@newcastle.ac.uk> Cc: Sigcis <members@sigcis.org> Subject: Re: [SIGCIS-Members] Perplexity ⚠ External sender. Take care when opening links or attachments. Do not provide your login details. Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here ( https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-oxneW37nS9.bA...) and should be available to anyone with the URL. I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom. One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month. -Adam Adam Hyland (he/him) adampunk.com<https://adampunk.com/> UW HCDE PhD Student On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members <members@lists.sigcis.org<mailto:members@lists.sigcis.org>> wrote: Hi: A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate. I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system. Perplexity itself claims: Here's what makes Perplexity different Answers that are accurate and always cited We continuously search the internet and identify the best sources, from academic research to Reddit threads, to provide the perfect answer to any question. Citations in every response Every answer uses cited sources to provide a more accurate and comprehensive answer. If you want to dig deeper, just click the link to the source. See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”: https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA<https://urldefense.com/v3/__https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$> The web version limits the number of questions per day – so far the iPhone App hasn’t. I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS. However, the Wikipedia article about it https://en.wikipedia.org/wiki/Perplexity_AI<https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$> is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards. Cheers Brian _______________________________________________ This email is relayed from members at sigcis.org<http://sigcis.org/>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis... and you can change your subscription options at https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sig...

To your second point, what if the old Irish newspapers were no longer around to check? We have 90 years of experience of librarians filming and digitizing newspapers, magazines, etc then throwing away the originals, at least in the USA. That was especially so with American librarians with newspapers beginning in the 1930s when they microfilmed them. Hate to criticize them, but they sometimes did not film all the issues of a newspaper, sometimes missed pages, or they came out blurry. Fast forward to the future if AI invents citations to stuff that no longer exists in their original. I encountered a bit of that similar result with the Google book project a few years ago. I worry about this possibility. How do we address that problem? Meanwhile, I, an old guy who grew up pre-AI, will still look at hard copies or trusted online versions of something (e.g., a journal) and keep hard copies of anything I cite off the Internet (e.g., images) before dropping that info into an endnote, because I don't want to be accused of faking stuff. It's hard enough working with real facts and sources. I suggest others follow a similar path. Call me old fashioned, but it has not hurt my productivity so far. Thanks for posting your thoughts, Brian. Jim Cortada On Fri, May 23, 2025 at 4:22 PM Brian Randell via Members < members@lists.sigcis.org> wrote:

Hi:

My colleague Brian Coghlan added two further points about the dangers AI hallucinations/fabrications pose to historians who are trying to use Generative AI tools to help their investigations:

First, our own investigation very strongly implied the "AI error rate" was inversely proportional to the AI

knowledge of the query topic. So one *very easy* countermeasure would be for AI tools to calculate their

knowledge of the topic and inversely adapt their response to that (be more cautious, not provide citations,

warn users their knowledge is weak, etc). *THAT* would make me somewhat hopeful ...

Secondly, "but then I found" fake citations -- these are a *REALLY* serious issue, especially to very old

sources, e.g. the 100+ years old Irish Times citations I'd to check -- it took a full 2 days of my time plus an

archivist to check them, and he spent the previous day physically retrieving the papers from a remote

archive (and presumably another day returning them) -- I got this effort gratis because of the novelty of

our investigation at that time, but it'd be unlikely to be repeated now without funding to pay for staff effort.

Imagine some years from now when huge numbers of these fake citations have to be checked, simply

because one might be a previously unknown real citation. Who on earth is going to fund that?

Cheers

Brian Randell

--

School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG

EMAIL = Brian.Randell@ncl.ac.uk PHONE = +44 (0)786 7805578

URL = https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html

*From: *Brian Randell <brian.randell@newcastle.ac.uk> *Date: *Wednesday, 7 May 2025 at 14:52 *To: *Adam Hyland <achyland@uw.edu> *Cc: *Sigcis <members@sigcis.org> *Subject: *Re: [SIGCIS-Members] Perplexity

Hi Adam:

Thanks – I hadn’t realised that URL links to Perplexity’s answers aren’t transferable.

It is certainly the case that Perplexity (which is based on ChatGPT4 I believe) is much more “knowledgeable” about Percy Ludgate than ChatGPT was – not least because it has ingested subsequent additions to the Internet.

Indeed, given the huge amount of effort and money being poured into LLM research we surely should expect the latest models to be significantly improved.

And Perplexity’s answer to my question “Are the references that Perplexity provides always real ones” was very reasonable, one might even say “thoughtful”, though its cautionary comments have of course to be applied to this answer itself – a nice example of recursion.

However, the NY Times had a very interesting, and surely worrying (for the Artificial Intelligentsia, especially) piece yesterday entitled: “A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse: A new wave of “reasoning” systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.”

I hope this link to it will work:

https://www.nytimes.com/2025/05/05/technology/ai-hallucinations-chatgpt-google.html?unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share

And on the subject of “hallucinations” (or rather “fabrications”), I also strongly recommend Gary Marcus’s very recent piece:

Why DO large language models hallucinate? <https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate>

( https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate )

Let me end by quoting from it:

“Because LLMS statistically mimic the language people have used, they often *fool people *into thinking that they operate like people. But they don’t operate like people. They don’t, for example, ever *fact check* (as humans sometimes, when well motivated, do). They *mimic the kinds of things of people say in various contexts*. And that’s essentially all they do. . .

Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates <https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/>*more <https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/>* than some its predecesssors <https://www.theleftshift.com/openai-admits-newer-models-hallucinate-even-more/> .

The chronic problem with creating fake citations in research papers <https://www.forbes.com/sites/larsdaniel/2025/01/29/the-irony-ai-experts-testimony--collapses-over-fake-ai-citations/> and faked cases in legal briefs <https://www.rawstory.com/lindell-dominion-2671843170/> is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is the rampant problem with numerical errors in financial reports <https://www.vals.ai/benchmarks/finance_agent-04-22-2025>, documented in a recent benchmark.

Just how bad is it? One recent study showed rates of hallucinations of between 15% and 60% <https://research.aimultiple.com/ai-hallucination/>across various models on a benchmark of 60 questions that were *easily verifiable relative to easily found CNN source articles that were directly supplied in the exam*. Even the best performance (15% hallucination rate) is, relative to an open-book exam with sources supplied, pathetic. That same study reports that, “According to Deloitte, 77% of businesses who joined the study are concerned about AI hallucinations”.

If I can be blunt, it is an absolute embarrassment that a technology that has collectively cost about half a trillion dollars can’t do something as basic as (reliably) check its output against wikipedia or a CNN article that is handed on a silver plattter. But LLMs still cannot - and on their own may never be able to — reliably do even things that basic.”

Cheers

Brian

--

School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG

EMAIL = Brian.Randell@ncl.ac.uk PHONE = +44 (0)786 7805578

URL = https://www.ncl.ac.uk/computing/staff/profile/brianrandell.html

*From: *Adam Hyland <achyland@uw.edu> *Date: *Tuesday, 6 May 2025 at 23:15 *To: *Brian Randell <brian.randell@newcastle.ac.uk> *Cc: *Sigcis <members@sigcis.org> *Subject: *Re: [SIGCIS-Members] Perplexity

⚠ External sender. Take care when opening links or attachments. Do not provide your login details.

Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here (

https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-oxneW37nS9.bA...) and should be available to anyone with the URL.

I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom.

One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month.

-Adam

Adam Hyland (*he/him)*

adampunk.com

UW HCDE PhD Student

On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members < members@lists.sigcis.org> wrote:

Hi:

A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate.

I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system.

Perplexity itself claims:

Here's what makes Perplexity different

Answers that are accurate and always cited

We continuously search the internet and identify the best sources, from academic research

to Reddit threads, to provide the perfect answer to any question.

Citations in every response

Every answer uses cited sources to provide a more accurate and comprehensive answer.

If you want to dig deeper, just click the link to the source.

See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”:

https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0... <https://urldefense.com/v3/__https://www.perplexity.ai/search/did-percy-ludgate-s-work-have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$>

The web version limits the number of questions per day – so far the iPhone App hasn’t.

I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS.

However, the Wikipedia article about it

https://en.wikipedia.org/wiki/Perplexity_AI <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$>

is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards.

Cheers

Brian

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://urldefense.com/v3/__http://lists.sigcis.org/pipermail/members-sigcis... and you can change your subscription options at https://urldefense.com/v3/__http://lists.sigcis.org/listinfo.cgi/members-sig...

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

-- James W. Cortada Senior Research Fellow Charles Babbage Institute University of Minnesota jcortada@umn.edu 608-274-6382

Some thoughts on The Book from the wonderful Carl Sagan seem apposite: “What an astonishing thing a book is. It's a flat object made from a tree with flexible parts on which are imprinted lots of funny dark squiggles. But one glance at it and you're inside the mind of another person, maybe somebody dead for thousands of years. Across the millennia, an author is speaking clearly and silently inside your head, directly to you. Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time. A book is proof that humans are capable of working magic." Dag ----- Dag Spicer Senior Curator Computer History Museum Editorial Board, IEEE Annals of the History of Computing ACM History Committee 1401 N. Shoreline Blvd. Mountain View CA 94043 “History is a vast early warning system.” — Norman Cousins, American journalist (1915-1990). Join our Mailing List here: https://info.computerhistory.org/subscribe On May 24, 2025, at 2:01 AM, Willard McCarty via Members <members@lists.sigcis.org> wrote: An intrusion from an outsider who subscribes... This morning on BBC Radio 4 someone at the Hay-on-Wye Festival (celebrating books) got to talking about the pleasure of reading physical codices. That led me to thinking about the inevitable loss of data (if you will) when a book, any book, is digitised. I couldn't do the interdisciplinary work I do without access to a large number of digitised texts. Indeed I am exceedingly grateful for digital collections and build my own with the help of a good scanner + OCR. But then there's the sustained reading of those I regard as important to spend time with. However such an argument runs, the fact is that digitisation is a particularly lossy form of translation. More than that, it not so much makes the scholar's life easier as quite different. I worry about the 'easier' bit because it entails loss that one quickly stops thinking about--hence the tendency to miss what only hard work will find. Given the competitiveness of the academic life esp. for junior scholars--I'd guess that in this crowd I don't have to spell it out. On the tangible/digital, not either/or but both! All best, Willard McCarty On 23/05/2025 22:39, James Cortada via Members wrote: To your second point, what if the old Irish newspapers were no longer around to check? We have 90 years of experience of librarians filming and digitizing newspapers, magazines, etc then throwing away the originals, at least in the USA. That was especially so with American librarians with newspapers beginning in the 1930s when they microfilmed them. Hate to criticize them, but they sometimes did not film all the issues of a newspaper, sometimes missed pages, or they came out blurry. Fast forward to the future if AI invents citations to stuff that no longer exists in their original. I encountered a bit of that similar result with the Google book project a few years ago. I worry about this possibility. How do we address that problem? Meanwhile, I, an old guy who grew up pre-AI, will still look at hard copies or trusted online versions of something (e.g., a journal) and keep hard copies of anything I cite off the Internet (e.g., images) before dropping that info into an endnote, because I don't want to be accused of faking stuff. It's hard enough working with real facts and sources. I suggest others follow a similar path. Call me old fashioned, but it has not hurt my productivity so far. Thanks for posting your thoughts, Brian. Jim Cortada On Fri, May 23, 2025 at 4:22 PM Brian Randell via Members <members@lists.sigcis.org <mailto:members@lists.sigcis.org>> wrote: Hi:____ __ __ My colleague Brian Coghlan added two further points about the dangers AI hallucinations/fabrications pose to historians who are trying to use Generative AI tools to help their investigations:____ __ __ First, our own investigation very strongly implied the "AI error rate" was inversely proportional to the AI ____ knowledge of the query topic. So one *very easy* countermeasure would be for AI tools to calculate their ____ knowledge of the topic and inversely adapt their response to that (be more cautious, not provide citations, ____ warn users their knowledge is weak, etc). *THAT* would make me somewhat hopeful ... Secondly, "but then I found" fake citations -- these are a *REALLY* serious issue, especially to very old ____ sources, e.g. the 100+ years old Irish Times citations I'd to check -- it took a full 2 days of my time plus an ____ archivist to check them, and he spent the previous day physically retrieving the papers from a remote ____ archive (and presumably another day returning them) -- I got this effort gratis because of the novelty of ____ our investigation at that time, but it'd be unlikely to be repeated now without funding to pay for staff effort.____ Imagine some years from now when huge numbers of these fake citations have to be checked, simply____ because one might be a previously unknown real citation. Who on earth is going to fund that? ____ __ __ Cheers____ __ __ Brian Randell____ __ __ -- ____ __ __ School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG____ EMAIL = Brian.Randell@ncl.ac.uk <mailto:Brian.Randell@ncl.ac.uk> PHONE = +44 (0)786 7805578____ URL = https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546393139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oNnyTHMuMgMucXs1HQaF7fFQ2aPrePKtCUKz%2FgOWL3I%3D&reserved=0 brianrandell.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546414984%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fIctrwGi4ixQuD4q3NTvxNmvkkTHWp7QVNz%2FUgsYM7w%3D&reserved=0 brianrandell.html>____ __ __ __ __ __ __ *From: *Brian Randell <brian.randell@newcastle.ac.uk <mailto:brian.randell@newcastle.ac.uk>> *Date: *Wednesday, 7 May 2025 at 14:52 *To: *Adam Hyland <achyland@uw.edu <mailto:achyland@uw.edu>> *Cc: *Sigcis <members@sigcis.org <mailto:members@sigcis.org>> *Subject: *Re: [SIGCIS-Members] Perplexity____ Hi Adam:____ ____ Thanks – I hadn’t realised that URL links to Perplexity’s answers aren’t transferable.____ ____ It is certainly the case that Perplexity (which is based on ChatGPT4 I believe) is much more “knowledgeable” about Percy Ludgate than ChatGPT was – not least because it has ingested subsequent additions to the Internet. ____ ____ Indeed, given the huge amount of effort and money being poured into LLM research we surely should expect the latest models to be significantly improved. ____ ____ And Perplexity’s answer to my question “Are the references that Perplexity provides always real ones” was very reasonable, one might even say “thoughtful”, though its cautionary comments have of course to be applied to this answer itself – a nice example of recursion. ____ ____ However, the NY Times had a very interesting, and surely worrying (for the Artificial Intelligentsia, especially) piece yesterday entitled: “A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse: A new wave of “reasoning” systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.”____ ____ I hope this link to it will work: ____ https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.nytimes.com%2F2025%2F05%2F05%2Ftechnology%2Fai-hallucinations-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546436545%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gEVbAtuBLymTKWaFwz9h%2Be6vYbq5q2UNlVdPUpDvRE0%3D&reserved=0 chatgpt-google.html? unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.nytimes.com%2F2025%2F05%2F05%2Ftechnology%2Fai-hallucinations-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546455331%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=vRVlqD9IhBHHxMDZHD6gukKl2tzL%2B7HvgQ1MdNMoV7U%3D&reserved=0 chatgpt-google.html? unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share>____ ____ And on the subject of “hallucinations” (or rather “fabrications”), I also strongly recommend Gary Marcus’s very recent piece:____ ____ Why DO large language models hallucinate? <https:// garymarcus.substack.com/p/why-do-large-language-models-hallucinate> ____ (https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgarymarcus.substack.com%2Fp%2Fwhy-do-large-language-models-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546472543%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xPJ2YSxEUEEBw9YRCf7qod1tjpyjd52GxXOWNmzKYdU%3D&reserved=0 hallucinate <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgarymarcus.substack.com%2Fp%2Fwhy-do-large-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546489849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cyClLhEZ5tXUcNiWnOY0%2BTZUQI2OvxYAFj24rnGi3hY%3D&reserved=0 language-models-hallucinate>)____ ____ Let me end by quoting from it:____ ____ “Because LLMS statistically mimic the language people have used, they often /fool people /into thinking that they operate like people. But they don’t operate like people. They don’t, for example, ever /fact check/ (as humans sometimes, when well motivated, do). They /mimic the kinds of things of people say in various contexts/. And that’s essentially all they do. . .____ Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates<https:// https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-models-hallucinate-even-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546506529%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xw5N%2Fc6tdtVnuVi4D03pyaPlcthOWxyKWT02P5Ofjss%3D&reserved=0 more/>/more <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546521577%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bQjrSC%2B7xv8IYTFYlHcrdXFCDt8h9eZuoVhYJ6GqH1U%3D&reserved=0 models-hallucinate-even-more/>/than some its predecesssors <https:// https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-models-hallucinate-even-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546534705%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZLpsZutayii5WFyyxofOWiVMASWx1O0nu4picq8gVdc%3D&reserved=0 more/>. ____ The chronic problem with creating fake citations in research papers <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.forbes.com%2Fsites%2Flarsdaniel%2F2025%2F01%2F29%2Fthe-irony-ai-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546547635%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HD1jw8wsOJghYC6LIxA6MpL%2FI7YU9HEy69mugPUys8o%3D&reserved=0 experts-testimony--collapses-over-fake-ai-citations/> and faked cases in legal briefs <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rawstory.com%2Flindell-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546560792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gyX8uT%2Fug31s5K8zkvG1VujsP%2FX8ZOnsa%2Fdz85tECU0%3D&reserved=0 dominion-2671843170/> is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is therampant problem with numerical errors in financial reports <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.vals.ai%2Fbenchmarks%2Ffinance_agent-04-22-2025&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546576681%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Jy66QrskB3aPCoO3ayT%2FICWZ4FlbqZq5M6SlRgAbyZM%3D&reserved=0>, documented in a recent benchmark.____ Just how bad is it? One recent study showed rates of hallucinations of between 15% and 60%<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresearch.aimultiple.com%2Fai-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546590578%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=seRW1vLe2NuBP15YS%2F%2FmAtHGOCE%2B5838OzedZyVUnN0%3D&reserved=0 hallucination/>across various models on a benchmark of 60 questions that were /easily verifiable relative to easily found CNN source articles that were directly supplied in the exam/. Even the best performance (15% hallucination rate) is, relative to an open-book exam with sources supplied, pathetic. That same study reports that, “According to Deloitte, 77% of businesses who joined the study are concerned about AI hallucinations”.____ If I can be blunt, it is an absolute embarrassment that a technology that has collectively cost about half a trillion dollars can’t do something as basic as (reliably) check its output against wikipedia or a CNN article that is handed on a silver plattter. But LLMs still cannot - and on their own may never be able to — reliably do even things that basic.”____ Cheers____ ____ Brian____ ____ -- ____ ____ School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG____ EMAIL = Brian.Randell@ncl.ac.uk <mailto:Brian.Randell@ncl.ac.uk> PHONE = +44 (0)786 7805578____ URL = https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546603193%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xVmA%2FdBbDaP81326OABeW%2FbePJz4Z8aihr%2F3dSqt7zE%3D&reserved=0 brianrandell.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546615899%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=QnN96y8EPhon8PY3WVdoggoWF0o3j23NTDpPYSGZlBk%3D&reserved=0 brianrandell.html>____ ____ ____ ____ *From: *Adam Hyland <achyland@uw.edu <mailto:achyland@uw.edu>> *Date: *Tuesday, 6 May 2025 at 23:15 *To: *Brian Randell <brian.randell@newcastle.ac.uk <mailto:brian.randell@newcastle.ac.uk>> *Cc: *Sigcis <members@sigcis.org <mailto:members@sigcis.org>> *Subject: *Re: [SIGCIS-Members] Perplexity____ ⚠External sender. Take care when opening links or attachments. Do not provide your login details. ____ Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here (____ https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-have-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546629150%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8QIVbhLoazvcEGTCFS3nRUmDn2neKBJFd7nq8EkVJ1o%3D&reserved=0 oxneW37nS9.bAiYvfV3QVw <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546642455%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0yHeqZbyjt5RHEE6eq%2FZoJ0J1UcyJfR9tBSJQSSQ4sA%3D&reserved=0 ludgate-s-work-have-oxneW37nS9.bAiYvfV3QVw>) and should be available to anyone with the URL.____ ____ I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom.____ ____ One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month.____ ____ -Adam____ ____ Adam Hyland (/he/him)/ ____ adampunk.com <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fadampunk.com%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546655285%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ow8%2FV0Y7rWMGRQhane6Pp%2FcE9Oxg55j0zNPTYgT8bBg%3D&reserved=0>____ UW HCDE PhD Student____ ____ ____ On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members <members@lists.sigcis.org <mailto:members@lists.sigcis.org>> wrote:____ Hi:____ ____ A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate.____ ____ I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system.____ ____ Perplexity itself claims:____ ____ Here's what makes Perplexity different____ ____ Answers that are accurate and always cited____ We continuously search the internet and identify the best sources, from academic research ____ to Reddit threads, to provide the perfect answer to any question.____ ____ Citations in every response____ Every answer uses cited sources to provide a more accurate and comprehensive answer. ____ If you want to dig deeper, just click the link to the source.____ ____ See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”:____ ____ https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546668082%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dzyBnIvPfmq65vIvQjJPwvPZHANmdBEFbwA%2FIIPaVdw%3D&reserved=0 have-2esemV.BRuCo0Q7O_4AxfA <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546681405%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oMVxUq3W6zK%2BIoz3VDH1V5sIJ8LsSD8KEo1aSvqOtm0%3D&reserved=0 __https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546694395%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nItNM3SOH4x4wTqFMVD56VVdYHMtgQiK8RCJ8GBL3i8%3D&reserved=0 have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4- UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$>____ ____ The web version limits the number of questions per day – so far the iPhone App hasn’t.____ ____ I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS.____ ____ However, the Wikipedia article about it ____ https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPerplexity_AI&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546706849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ttz0ePB2g%2BxEWWIKeQcWtVWDB1FkyzqTPxuJo%2BQuZPE%3D&reserved=0 <https:// urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546719376%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0PAz3wNJkFpfXf%2BrFq5X2ndkK9Ag1geNAVatxGlAwgc%3D&reserved=0 Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4- UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$>____ is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards.____ ____ Cheers____ ____ Brian____ _______________________________________________ This email is relayed from members at sigcis.org <http:// sigcis.org/>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546734328%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mNTHHgtSvVSBbyjJnHzo8v9L5h35RTOFbE1wt3YK8Hg%3D&reserved=0 pipermail/members-sigcis.org/__;!!K-Hz7m0Vt54!gyxp0x4- UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lxruqEdl$ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lxruqEdl%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546747965%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f6nX7F1K3268ahsegkgnJjIzdjxAXPnNO0bhNCXmGYs%3D&reserved=0> and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lwGqTJnR%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546760503%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rYIHnjbbpNxLv98QQmLWlAQfCHs92vz1U17hSahAsSw%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lwGqTJnR%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546772708%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xwE53BautyC9Vpn6F9r8qw86O24BApLGqyGucsZqqzA%3D&reserved=0> ____ _______________________________________________ This email is relayed from members at sigcis.org <http:// sigcis.org>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http:// lists.sigcis.org/pipermail/members-sigcis.org/ <http:// lists.sigcis.org/pipermail/members-sigcis.org/> and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546784738%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XtcDc0HkYCyhnQ3mPdaIubbp5Wo0LWrGpSQFJ5%2BKdzE%3D&reserved=0 members-sigcis.org <https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546798030%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nwG2%2BG8yCUGsx0LBjttoVj9Z8%2BwvrOd9AoMgI8j2vAw%3D&reserved=0 sigcis.org> -- James W. Cortada Senior Research Fellow Charles Babbage Institute University of Minnesota jcortada@umn.edu <mailto:jcortada@umn.edu> 608-274-6382 _______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546810380%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7QZaDpifoQT10soobyZ%2FXcy3c7uvAWYc9j%2BOYHKFrx0%3D&reserved=0 and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546822843%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JfGl%2FObIRVGbcND3yPtsObIeULypic3vGckuvEjAiG4%3D&reserved=0 -- Willard McCarty, Professor emeritus, King's College London; Editor, Humanist https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mccarty.org.uk%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546835757%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XXJSH%2F54jEpy4Z1w9jGhQe%2FYvD0shfZbVTaR8uCr%2Bho%3D&reserved=0 _______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546849073%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bLukzhtSLkVWdX9sjeHXmRCsKHF8NBGBmh0cDv7x%2FjI%3D&reserved=0 and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546862281%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AABudduVx2EjcqxTmhQ2LSEg%2FLaLshsxoe%2BHmG0ry38%3D&reserved=0 . CAUTION: External email. Beware of links & attachments.

Ah Yes. Blue Books. Dag, you are a naughty person, because I had not thought about Blue Books for over 50 years! They hurt my hands, they were too slow to capture my thinking, they led me to fear what horrible questions I would have to pontificate about. Best question I ever had to write about came in a modern Russian history class where all the exams were multiple choice until the final. Blue books and 1 Question, for which we had 3 hours to respond: "Was Stalin Necessary?" Everyone wrote in multiple Blue Books, I wrote for 30 minutes arguing why he was necessary to industrialize and modernize the Soviet economy just in time for WWII. I got an A+ with the professor commenting that I had internalized modern Russian history and could think through what it all meant. But I love the old-school solution. Thanks for sharing this posting. Jim On Mon, May 26, 2025 at 2:26 PM Danny Spitzberg via Members < members@lists.sigcis.org> wrote:

Related to what Dag quoted about books... this just out <https://archive.ph/JtpKt> from the Wall Street Journal

They Were Every Student’s Worst Nightmare. Now Blue Books Are Back. Cheating with ChatGPT has become a huge problem for colleges. The solution is painfully old-school.

By Ben Cohen

May 23, 2025 8:00 pm ET

On Sat, May 24, 2025 at 10:11 AM Dag Spicer via Members < members@lists.sigcis.org> wrote:

...

Some thoughts on The Book from the wonderful Carl Sagan seem apposite: “What an astonishing thing a book is. It's a flat object made from a tree with flexible parts on which are imprinted lots of funny dark squiggles. But one glance at it and you're inside the mind of another person, maybe somebody dead for thousands of years. Across the millennia, an author is speaking clearly and silently inside your head, directly to you. Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time. A book is proof that humans are capable of working magic." Dag ----- Dag Spicer Senior Curator Computer History Museum Editorial Board, IEEE Annals of the History of Computing ACM History Committee 1401 N. Shoreline Blvd. Mountain View CA 94043

“History is a vast early warning system.” — Norman Cousins, American journalist (1915-1990).

Join our Mailing List here: https://info.computerhistory.org/subscribe

On May 24, 2025, at 2:01 AM, Willard McCarty via Members < members@lists.sigcis.org> wrote:

An intrusion from an outsider who subscribes...

This morning on BBC Radio 4 someone at the Hay-on-Wye Festival (celebrating books) got to talking about the pleasure of reading physical codices. That led me to thinking about the inevitable loss of data (if you will) when a book, any book, is digitised. I couldn't do the interdisciplinary work I do without access to a large number of digitised texts. Indeed I am exceedingly grateful for digital collections and build my own with the help of a good scanner + OCR. But then there's the sustained reading of those I regard as important to spend time with.

However such an argument runs, the fact is that digitisation is a particularly lossy form of translation. More than that, it not so much makes the scholar's life easier as quite different. I worry about the 'easier' bit because it entails loss that one quickly stops thinking about--hence the tendency to miss what only hard work will find. Given the competitiveness of the academic life esp. for junior scholars--I'd guess that in this crowd I don't have to spell it out.

On the tangible/digital, not either/or but both!

All best, Willard McCarty

On 23/05/2025 22:39, James Cortada via Members wrote:

To your second point, what if the old Irish newspapers were no longer around to check? We have 90 years of experience of librarians filming and digitizing newspapers, magazines, etc then throwing away the originals, at least in the USA. That was especially so with American librarians with newspapers beginning in the 1930s when they microfilmed them. Hate to criticize them, but they sometimes did not film all the issues of a newspaper, sometimes missed pages, or they came out blurry. Fast forward to the future if AI invents citations to stuff that no longer exists in their original. I encountered a bit of that similar result with the Google book project a few years ago. I worry about this possibility. How do we address that problem?

Meanwhile, I, an old guy who grew up pre-AI, will still look at hard copies or trusted online versions of something (e.g., a journal) and keep hard copies of anything I cite off the Internet (e.g., images) before dropping that info into an endnote, because I don't want to be accused of faking stuff. It's hard enough working with real facts and sources. I suggest others follow a similar path. Call me old fashioned, but it has not hurt my productivity so far. Thanks for posting your thoughts, Brian. Jim Cortada

On Fri, May 23, 2025 at 4:22 PM Brian Randell via Members <members@lists.sigcis.org <mailto:members@lists.sigcis.org>> wrote:

Hi:____

__ __

My colleague Brian Coghlan added two further points about the dangers AI hallucinations/fabrications pose to historians who are trying to use Generative AI tools to help their investigations:____

__ __

First, our own investigation very strongly implied the "AI error rate" was inversely proportional to the AI ____

knowledge of the query topic. So one *very easy* countermeasure would be for AI tools to calculate their ____

knowledge of the topic and inversely adapt their response to that (be more cautious, not provide citations, ____

warn users their knowledge is weak, etc). *THAT* would make me somewhat hopeful ...

Secondly, "but then I found" fake citations -- these are a *REALLY* serious issue, especially to very old ____

sources, e.g. the 100+ years old Irish Times citations I'd to check -- it took a full 2 days of my time plus an ____

archivist to check them, and he spent the previous day physically retrieving the papers from a remote ____

archive (and presumably another day returning them) -- I got this effort gratis because of the novelty of ____

our investigation at that time, but it'd be unlikely to be repeated now without funding to pay for staff effort.____

Imagine some years from now when huge numbers of these fake citations have to be checked, simply____

because one might be a previously unknown real citation. Who on earth is going to fund that? ____

__ __

Cheers____

__ __

Brian Randell____

__ __

-- ____

__ __

School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG____

EMAIL = Brian.Randell@ncl.ac.uk <mailto:Brian.Randell@ncl.ac.uk> PHONE = +44 (0)786 7805578____

URL = https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546393139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oNnyTHMuMgMucXs1HQaF7fFQ2aPrePKtCUKz%2FgOWL3I%3D&reserved=0 brianrandell.html < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546414984%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fIctrwGi4ixQuD4q3NTvxNmvkkTHWp7QVNz%2FUgsYM7w%3D&reserved=0 brianrandell.html>____

__ __

__ __

__ __

*From: *Brian Randell <brian.randell@newcastle.ac.uk <mailto:brian.randell@newcastle.ac.uk>> *Date: *Wednesday, 7 May 2025 at 14:52 *To: *Adam Hyland <achyland@uw.edu <mailto:achyland@uw.edu>> *Cc: *Sigcis <members@sigcis.org <mailto:members@sigcis.org>> *Subject: *Re: [SIGCIS-Members] Perplexity____

Hi Adam:____

____

Thanks – I hadn’t realised that URL links to Perplexity’s answers aren’t transferable.____

____

It is certainly the case that Perplexity (which is based on ChatGPT4 I believe) is much more “knowledgeable” about Percy Ludgate than ChatGPT was – not least because it has ingested subsequent additions to the Internet. ____

____

Indeed, given the huge amount of effort and money being poured into LLM research we surely should expect the latest models to be significantly improved. ____

____

And Perplexity’s answer to my question “Are the references that Perplexity provides always real ones” was very reasonable, one might even say “thoughtful”, though its cautionary comments have of course to be applied to this answer itself – a nice example of recursion. ____

____

However, the NY Times had a very interesting, and surely worrying (for the Artificial Intelligentsia, especially) piece yesterday entitled: “A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse: A new wave of “reasoning” systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.”____

____

I hope this link to it will work: ____

https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.nytimes.com%2F2025%2F05%2F05%2Ftechnology%2Fai-hallucinations-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546436545%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gEVbAtuBLymTKWaFwz9h%2Be6vYbq5q2UNlVdPUpDvRE0%3D&reserved=0 chatgpt-google.html? unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.nytimes.com%2F2025%2F05%2F05%2Ftechnology%2Fai-hallucinations-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546455331%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=vRVlqD9IhBHHxMDZHD6gukKl2tzL%2B7HvgQ1MdNMoV7U%3D&reserved=0 chatgpt-google.html? unlocked_article_code=1.E08.mKzL.qenddVF5UQVq&smid=url-share>____

____

And on the subject of “hallucinations” (or rather “fabrications”), I also strongly recommend Gary Marcus’s very recent piece:____

____

Why DO large language models hallucinate? <https:// garymarcus.substack.com/p/why-do-large-language-models-hallucinate> ____

( https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgarymarcus.substack.com%2Fp%2Fwhy-do-large-language-models-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546472543%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xPJ2YSxEUEEBw9YRCf7qod1tjpyjd52GxXOWNmzKYdU%3D&reserved=0 hallucinate < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgarymarcus.substack.com%2Fp%2Fwhy-do-large-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546489849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cyClLhEZ5tXUcNiWnOY0%2BTZUQI2OvxYAFj24rnGi3hY%3D&reserved=0 language-models-hallucinate>)____

____

Let me end by quoting from it:____

____

“Because LLMS statistically mimic the language people have used, they often /fool people /into thinking that they operate like people. But they don’t operate like people. They don’t, for example, ever /fact check/ (as humans sometimes, when well motivated, do). They /mimic the kinds of things of people say in various contexts/. And that’s essentially all they do. . .____

Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates<https://

https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-models-hallucinate-even-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546506529%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xw5N%2Fc6tdtVnuVi4D03pyaPlcthOWxyKWT02P5Ofjss%3D&reserved=0 more/>/more < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546521577%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bQjrSC%2B7xv8IYTFYlHcrdXFCDt8h9eZuoVhYJ6GqH1U%3D&reserved=0 models-hallucinate-even-more/>/than some its predecesssors <https://

https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.theleftshift.com%2Fopenai-admits-newer-models-hallucinate-even-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546534705%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZLpsZutayii5WFyyxofOWiVMASWx1O0nu4picq8gVdc%3D&reserved=0 more/>. ____

The chronic problem with creating fake citations in research papers < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.forbes.com%2Fsites%2Flarsdaniel%2F2025%2F01%2F29%2Fthe-irony-ai-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546547635%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HD1jw8wsOJghYC6LIxA6MpL%2FI7YU9HEy69mugPUys8o%3D&reserved=0 experts-testimony--collapses-over-fake-ai-citations/> and faked cases in legal briefs < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rawstory.com%2Flindell-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546560792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gyX8uT%2Fug31s5K8zkvG1VujsP%2FX8ZOnsa%2Fdz85tECU0%3D&reserved=0 dominion-2671843170/> is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is therampant problem with numerical errors in financial reports < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.vals.ai%2Fbenchmarks%2Ffinance_agent-04-22-2025&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546576681%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Jy66QrskB3aPCoO3ayT%2FICWZ4FlbqZq5M6SlRgAbyZM%3D&reserved=0

...

, documented in a recent benchmark.____

Just how bad is it? One recent study showed rates of hallucinations of between 15% and 60%< https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresearch.aimultiple.com%2Fai-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546590578%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=seRW1vLe2NuBP15YS%2F%2FmAtHGOCE%2B5838OzedZyVUnN0%3D&reserved=0 hallucination/>across various models on a benchmark of 60 questions that were /easily verifiable relative to easily found CNN source articles that were directly supplied in the exam/. Even the best performance (15% hallucination rate) is, relative to an open-book exam with sources supplied, pathetic. That same study reports that, “According to Deloitte, 77% of businesses who joined the study are concerned about AI hallucinations”.____

If I can be blunt, it is an absolute embarrassment that a technology that has collectively cost about half a trillion dollars can’t do something as basic as (reliably) check its output against wikipedia or a CNN article that is handed on a silver plattter. But LLMs still cannot - and on their own may never be able to — reliably do even things that basic.”____

Cheers____

____

Brian____

____

-- ____

____

School of Computing, Newcastle University, 1 Science Square, Newcastle upon Tyne, NE4 5TG____

EMAIL = Brian.Randell@ncl.ac.uk <mailto:Brian.Randell@ncl.ac.uk> PHONE = +44 (0)786 7805578____

URL = https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546603193%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xVmA%2FdBbDaP81326OABeW%2FbePJz4Z8aihr%2F3dSqt7zE%3D&reserved=0 brianrandell.html < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ac.uk%2Fcomputing%2Fstaff%2Fprofile%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546615899%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=QnN96y8EPhon8PY3WVdoggoWF0o3j23NTDpPYSGZlBk%3D&reserved=0 brianrandell.html>____

____

____

____

*From: *Adam Hyland <achyland@uw.edu <mailto:achyland@uw.edu>> *Date: *Tuesday, 6 May 2025 at 23:15 *To: *Brian Randell <brian.randell@newcastle.ac.uk <mailto:brian.randell@newcastle.ac.uk>> *Cc: *Sigcis <members@sigcis.org <mailto:members@sigcis.org>> *Subject: *Re: [SIGCIS-Members] Perplexity____

⚠External sender. Take care when opening links or attachments. Do not provide your login details. ____

Thanks for your return to this topic. The perplexity thread you shared is private and unfortunately the pdf generated obscures the answer with a UI element. I’ve asked the same question to Perplexity using their “research” mode which more profligately uses computing resources to give a more detailed answer. The result is here (____

https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-have-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546629150%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8QIVbhLoazvcEGTCFS3nRUmDn2neKBJFd7nq8EkVJ1o%3D&reserved=0 oxneW37nS9.bAiYvfV3QVw < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546642455%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0yHeqZbyjt5RHEE6eq%2FZoJ0J1UcyJfR9tBSJQSSQ4sA%3D&reserved=0 ludgate-s-work-have-oxneW37nS9.bAiYvfV3QVw>) and should be available to anyone with the URL.____

____

I’m familiar with perplexity because the University of Washington pays for “pro” access which makes it both partially difficult and morally dubious to proscribe its use in the classroom.____

____

One broad point is probably obvious to everyone but bears repeating: there is much more difference in capability between any of these agents today versus a year ago than there is among the agents themselves. They all show a dramatic increase in capability to “understand” queries and generate cogent responses. If someone on this thread last posed one of their exam questions to ChatGPT last year they ought to try again today, and again in another month.____

____

-Adam____

____

Adam Hyland (/he/him)/ ____

adampunk.com < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fadampunk.com%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546655285%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ow8%2FV0Y7rWMGRQhane6Pp%2FcE9Oxg55j0zNPTYgT8bBg%3D&reserved=0

...

____

UW HCDE PhD Student____

____

____

On Tue, May 6, 2025 at 8:57 AM Brian Randell via Members <members@lists.sigcis.org <mailto:members@lists.sigcis.org>> wrote:____

Hi:____

____

A while ago I and colleagues published a short critique in the Annals of the History of Computing of ChatGPT, based on its performance on some questions about Percy Ludgate.____

____

I’ve recently been trying (the free version of) Perplexity, the AI search (or more exactly question-answering) system.____

____

Perplexity itself claims:____

____

Here's what makes Perplexity different____

____

Answers that are accurate and always cited____

We continuously search the internet and identify the best sources, from academic research ____

to Reddit threads, to provide the perfect answer to any question.____

____

Citations in every response____

Every answer uses cited sources to provide a more accurate and comprehensive answer. ____

If you want to dig deeper, just click the link to the source.____

____

See its brilliant answer to the question “Did Percy Ludgate's work have any impact?”:____

____

https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546668082%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dzyBnIvPfmq65vIvQjJPwvPZHANmdBEFbwA%2FIIPaVdw%3D&reserved=0 have-2esemV.BRuCo0Q7O_4AxfA < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546681405%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oMVxUq3W6zK%2BIoz3VDH1V5sIJ8LsSD8KEo1aSvqOtm0%3D&reserved=0 __ https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.perplexity.ai%2Fsearch%2Fdid-percy-ludgate-s-work-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546694395%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nItNM3SOH4x4wTqFMVD56VVdYHMtgQiK8RCJ8GBL3i8%3D&reserved=0 have-2esemV.BRuCo0Q7O_4AxfA__;!!K-Hz7m0Vt54!gyxp0x4-

UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l7QbFBQe$>____

____

The web version limits the number of questions per day – so far the iPhone App hasn’t.____

____

I assume I’m not alone here in trying Perplexity, but I don’t recall any previous comment about it in SIGCIS.____

____

However, the Wikipedia article about it ____

https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPerplexity_AI&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546706849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ttz0ePB2g%2BxEWWIKeQcWtVWDB1FkyzqTPxuJo%2BQuZPE%3D&reserved=0 <https://

urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546719376%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0PAz3wNJkFpfXf%2BrFq5X2ndkK9Ag1geNAVatxGlAwgc%3D&reserved=0 Perplexity_AI__;!!K-Hz7m0Vt54!gyxp0x4-

UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1l_ByERqJ$>____

is a little sobering, regarding its alleged copyright violations, and failure to respect the robots.txt web-crawling standards.____

____

Cheers____

____

Brian____

_______________________________________________ This email is relayed from members at sigcis.org <http:// sigcis.org/>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546734328%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mNTHHgtSvVSBbyjJnHzo8v9L5h35RTOFbE1wt3YK8Hg%3D&reserved=0 pipermail/members-sigcis.org/__;!!K-Hz7m0Vt54!gyxp0x4- UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lxruqEdl$ < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lxruqEdl%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546747965%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f6nX7F1K3268ahsegkgnJjIzdjxAXPnNO0bhNCXmGYs%3D&reserved=0> and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lwGqTJnR%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546760503%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rYIHnjbbpNxLv98QQmLWlAQfCHs92vz1U17hSahAsSw%3D&reserved=0 < https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org__%3B!!K-Hz7m0Vt54!gyxp0x4-UgdlqkBkNCRifXnXGAv6Y1rWzqMDvgDAx9lcE0e3xttiUQD_Zgg0jayzgaPV_LCgiwp1lwGqTJnR%24&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546772708%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xwE53BautyC9Vpn6F9r8qw86O24BApLGqyGucsZqqzA%3D&reserved=0> ____

_______________________________________________ This email is relayed from members at sigcis.org <http:// sigcis.org>, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http:// lists.sigcis.org/pipermail/members-sigcis.org/ <http:// lists.sigcis.org/pipermail/members-sigcis.org/> and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546784738%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XtcDc0HkYCyhnQ3mPdaIubbp5Wo0LWrGpSQFJ5%2BKdzE%3D&reserved=0 members-sigcis.org < https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546798030%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nwG2%2BG8yCUGsx0LBjttoVj9Z8%2BwvrOd9AoMgI8j2vAw%3D&reserved=0 sigcis.org>

-- James W. Cortada Senior Research Fellow Charles Babbage Institute University of Minnesota jcortada@umn.edu <mailto:jcortada@umn.edu> 608-274-6382

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546810380%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7QZaDpifoQT10soobyZ%2FXcy3c7uvAWYc9j%2BOYHKFrx0%3D&reserved=0 and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546822843%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JfGl%2FObIRVGbcND3yPtsObIeULypic3vGckuvEjAiG4%3D&reserved=0

-- Willard McCarty, Professor emeritus, King's College London; Editor, Humanist

https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mccarty.org.uk%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546835757%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XXJSH%2F54jEpy4Z1w9jGhQe%2FYvD0shfZbVTaR8uCr%2Bho%3D&reserved=0

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Fpipermail%2Fmembers-sigcis.org%2F&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546849073%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bLukzhtSLkVWdX9sjeHXmRCsKHF8NBGBmh0cDv7x%2FjI%3D&reserved=0 and you can change your subscription options at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.sigcis.org%2Flistinfo.cgi%2Fmembers-sigcis.org&data=05%7C02%7Cdspicer%40computerhistory.org%7C38f01d8a834a4bf76b2608dd9aa1b495%7Cb6a9c12a29ee4c5f8f93f0d7a8e2db2f%7C0%7C0%7C638836741546862281%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AABudduVx2EjcqxTmhQ2LSEg%2FLaLshsxoe%2BHmG0ry38%3D&reserved=0 .

CAUTION: External email. Beware of links & attachments.

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

_______________________________________________ This email is relayed from members at sigcis.org, the email discussion list of SHOT SIGCIS. Opinions expressed here are those of the member posting and are not reviewed, edited, or endorsed by SIGCIS. The list archives are at http://lists.sigcis.org/pipermail/members-sigcis.org/ and you can change your subscription options at http://lists.sigcis.org/listinfo.cgi/members-sigcis.org

-- James W. Cortada Senior Research Fellow Charles Babbage Institute University of Minnesota jcortada@umn.edu 608-274-6382