[SIGCIS-Members] BSD (Berkeley Unix) history book

[Jeremy C. Reed]

I was encouraged last year to join this list to introduce my research 
and upcoming book about the history of the Berkeley Software 
Distributions. (My membership didn't kick in for a few months.)

I am authoring a detailed book covering the history of BSD / Berkeley 
Unix. One of its releases is known as the "Greatest Software Ever 
Written" -- selected by DARPA to be the standardized operating system 
platform for the DARPA research community, it became the widely 
distributed software release that included the TCP/IP Internet protocols 
and Internet server software to the world.

The book attempts to introduce the characters and events surrounding the 
BSD story. Some of my main goals were:

- Interview the ~74 original surviving contributors of the early BSDs 
(before 1981). (Sadly at least three early participants have already 
deceased.)

- Interview most of the contributors toward the 4.3BSD-Networking 
Release 2 and 4.4BSD-Lite2 distributions. (The replacement of 
proprietary software and key event in open source history.)

- Interview the founders and some key participants of CSRG, 386BSD, 
BSD/386, NetBSD, FreeBSD, OpenBSD, DragonFly, etc.

- Accurately and fairly share the stories for the key events in BSD 
history.

- Cover the modern BSD histories.

I have already done over 85 interviews and the book draft is currently 
around 270 pages long with over 200 citations in the bibliography (and 
over a 100 additional that need to add to the bibtex file). The book 
has 26 chapters and is around 67,000 words. I have over 100,000 words of 
notes in my subversion tex files that are commented out.

(Quickly, for my related background: journalism degree from Western 
Washington Univ.; as a small book publisher, published over ten books; 
ran two BSD news/features websites; BSD sysadmin for only around 16 
years; active participant in the BSD community, lecturing at events and 
providing marketing and project management.)

I will quickly introduce some the methods used for researching my book:

- downloaded old BSDs release tarballs (including coding up support to 
extract now extinct archive formats) and purchased CSRG archives of BSD 
releases (including some experimental releases). The BSD releases 
included documentation with lots of acknowledgements, source code with 
lots of comments, some code revision history, binaries, real-world 
configurations, including some personal files from the key developers 
left in home directories that were tar'ed up with the releases. For the 
oldest releases, I skimmed through all the source code and all the 
documentation (manual pages, howtos, master theses, etc.). For newer 
larger releases, I programmatically searched for significant details. (I 
also skimmed through other old Unix-related archives for BSD-related 
histories.) CSRG sets also included administration emails, licensing 
details, letters to sponsors, etc.

- purchased the SCCS histories and used this to find the source code 
history for various code and documentation, etc. Note that some of the 
old sets had older code not found in the SCCS histories, plus some SCCS 
histories had gaps or were lost due to filename/directory moves. The BSD 
sets were used to fill in some holes and to find older code, including 
old SCCS or RCS metadata. For newer releases, I used CVS histories.

- took notes from BSD lectures and found all known BSD history articles

- built list of all contact names (including non-developers) for the 
earliest releases up to mid 1980, and then all contributors toward the 
later BSD releases, and then significant participants toward the modern 
day BSDs. I attempted contacting maybe hundreds of these participants 
via email or some social network sites. I heard back from over a 100. 
Generally before I contact the person, I search my now-massive digital 
collection of documents and then summarize what I know to help the 
participant recollect history from long ago. I did over 80 interviews by 
email, several by phone, and some in-person (at some conferences). All 
my interview notes are stored in separate per-person files. Often the 
participants name other people I should consider. Looking now, I see I 
have at least 454 names in my detailed list. I did interviews with 
inventors of many of the open source and Unix technologies in use today.

- organized a file system with categories of different files for notes, 
massive TODO list, list of people to contact, books to purchase, etc, 
and the collection of downloaded or scanned files categorized by decades 
(plus I kept track of where the data was obtained). (I also keep a file 
of typos, errata, or other bugs I found in lectures, books, old articles 
or documentation, such as name misspellings, wrong years, etc.)

- based on history specific articles and history books plus feedback 
from key participants, I defined the key stories that must be told and 
the key participants that must be introduced in the book. (But I have 
over a hundred side stories that have consumed my time and the book!)

- setup a subversion repo to track my book writing in LaTeX. Used 
comments to integrate notes in from all the histories I found, 
timelines (94 events), and integrated interviews.

- used old Usenet / newsgroups archives to supplement stories. (Google 
Groups has been useful with its date-based searches.) I also found 
various archives available for downloads. In addition, even some 
newsgroup postings and old email archives were included in some of the 
historical BSD releases and CSRG archives.

- found copies of old magazine articles and found archives old magazines 
at libraries. In addition, some contacts scanned documents or shipped 
documents to me.  I also transcribed or took notes from several 
interviews (done by others).

- found many old master theses, university (multiple schools) technical 
reports, BBN technical reports, legal documents, old (signed) software
licenses, etc. that had much history details.

- purchased every book that I learned about that contained BSD history 
and took detailed notes. (For what it is worth, to give you a size 
example of the most expansive BSD history in print: Salus's excellent 
Quarter Century book has two chapters totaling around 18 pages; and 
McKusick's most-interesting Open Sources chapter is around 7,300 words. 
Some of my chapters for short periods of time and for only single 
releases are longer.)

I have been working on it for over five years. I try not to cover the 
side stories too much; for example I cannot cover in great detail the 
history of Unix, history of BBN's related work, history of open source 
licensing, nor DEC, PDP-11 or VAX history. (I have noticed that other 
participants on this group have written some of this related work and at 
least one document is a source in my bibliography -- Thanks Dave W.!)

The amount of stories is amazing, such as connection between
co-inventor of Unix and Berkeley, Berkeley Pascal, Virtual Memory
Extensions for Unix "vmunix", VMS vs. Unix performance, DARPA
research and founding of CSRG, TCP/IP and Internet services, support
for multiple hardware platforms from one source tree, birth of
Berkeley licensing and replacing proprietary software, the continuation
of 2BSD (PDP-11), the Internet Worm, 386BSD and patchkits, birth
of NetBSD and FreeBSD, AT&T/CSRG/BSDI/Novell legal proceedings,
OpenBSD fork, and much more.

Plus lots of interesting side stories: adventure and trek; INGRES
database; SPICE2; birth of DNS; Email services (beyond and before
SMTP); commercial Unix licensing and then open source licensing;
first multi-processor support; revision control systems, BBN and
Unix and ARPANET work; founding of Sun (and other BSD-based Unixes);
and the intersection of BSD, Lucasfilm, Pixar, CMU Mach, NeXT, BSD
again, Rhapsody/Darwin/MacOSX, etc. Lots of the stories are outside
of University of California at Berkeley.

Four other potential book ideas from my work include: history of
open source licensing (as a side project I contacted the developers
of the code containing the first "open source" style licenses
known), history of X (Andrew window system, Verex, V window system,
W window system, and then MIT wouldn't use Stanford's code so birth
of "X", etc.), technical history of SUN and Sun Microsystems, and
history of MacOSX (starting over a decade earlier, see my "intersection"
above).

I have missed several self-imposed deadlines for publishing the book. I 
have learned that every direction I take seems to bring in more 
never-ending stories (for example my recent research and interviews
introduced the history of Ultrix), so I have broke it down into a
few key things I need to cover and then ignore the many side stories
and TODOs not fully researched to save them for a "second edition"
of the book. My current work is to complete some integrations of
my interviews and finish tying in the stories; pay for proofreading
and editing for writing style; finish bibtex entries; send chapters
to my many waiting reviewers; integrate their feedback; complete
indexing; pay for cover artwork; and pay for another round of
proofreading. I will reset my goal: get the book to my printer by
end of August.

If anyone has advice or suggestions for research methods for similar 
technical computer or software history books, it may be interesting to 
continue a discussion about it.

I'd like to encourage any of you with an interest in the history of open 
source licensing, Unix, Berkeley Pascal, or BSD to contact me.

Thanks for reading!

  Jeremy C. Reed

echo 'EhZ[h ^jjf0%%h[[Zc[Z_W$d[j%Xeeai%ZW[ced#]dk#f[d]k_d%' | \
  tr            '#-~'            '\-.-{'
(This is a different book.)

p.s For fun ...

miscellaneous notes and todos:
t1:bsd-history$ wc * | grep " total$" 
   30781  140722 2035160 total

latex files:
t1:bsd-history$ wc svn-bsd-history/*tex | grep " total$" 
   33088  223360 1425282 total
t1:bsd-history$ egrep 'CITE:|\\cite{' svn-bsd-history/*tex | wc -l
    1457
t1:bsd-history$ grep '\\bibitem' generated/book.bbl  | wc -l
     211

most of my interviews:
t1:bsd-history$ wc interviews/{done,todo}/*-interview* | grep " total$"
   15948  131526  750691 total

my collection of related Usenet postings, technical reports, SCCS 
histories, old BSDs, etc:
t1:bsd-history$ du -sm archives
5306    archives
t1:bsd-history$ find -x archives -type f | wc -l
539001