I was encouraged last year to join this list to introduce my research
and upcoming book about the history of the Berkeley Software
Distributions. (My membership didn't kick in for a few months.)
I am authoring a detailed book covering the history of BSD / Berkeley
Unix. One of its releases is known as the "Greatest Software Ever
Written" -- selected by DARPA to be the standardized operating system
platform for the DARPA research community, it became the widely
distributed software release that included the TCP/IP Internet protocols
and Internet server software to the world.
The book attempts to introduce the characters and events surrounding the
BSD story. Some of my main goals were:
- Interview the ~74 original surviving contributors of the early BSDs
(before 1981). (Sadly at least three early participants have already
deceased.)
- Interview most of the contributors toward the 4.3BSD-Networking
Release 2 and 4.4BSD-Lite2 distributions. (The replacement of
proprietary software and key event in open source history.)
- Interview the founders and some key participants of CSRG, 386BSD,
BSD/386, NetBSD, FreeBSD, OpenBSD, DragonFly, etc.
- Accurately and fairly share the stories for the key events in BSD
history.
- Cover the modern BSD histories.
I have already done over 85 interviews and the book draft is currently
around 270 pages long with over 200 citations in the bibliography (and
over a 100 additional that need to add to the bibtex file). The book
has 26 chapters and is around 67,000 words. I have over 100,000 words of
notes in my subversion tex files that are commented out.
(Quickly, for my related background: journalism degree from Western
Washington Univ.; as a small book publisher, published over ten books;
ran two BSD news/features websites; BSD sysadmin for only around 16
years; active participant in the BSD community, lecturing at events and
providing marketing and project management.)
I will quickly introduce some the methods used for researching my book:
- downloaded old BSDs release tarballs (including coding up support to
extract now extinct archive formats) and purchased CSRG archives of BSD
releases (including some experimental releases). The BSD releases
included documentation with lots of acknowledgements, source code with
lots of comments, some code revision history, binaries, real-world
configurations, including some personal files from the key developers
left in home directories that were tar'ed up with the releases. For the
oldest releases, I skimmed through all the source code and all the
documentation (manual pages, howtos, master theses, etc.). For newer
larger releases, I programmatically searched for significant details. (I
also skimmed through other old Unix-related archives for BSD-related
histories.) CSRG sets also included administration emails, licensing
details, letters to sponsors, etc.
- purchased the SCCS histories and used this to find the source code
history for various code and documentation, etc. Note that some of the
old sets had older code not found in the SCCS histories, plus some SCCS
histories had gaps or were lost due to filename/directory moves. The BSD
sets were used to fill in some holes and to find older code, including
old SCCS or RCS metadata. For newer releases, I used CVS histories.
- took notes from BSD lectures and found all known BSD history articles
- built list of all contact names (including non-developers) for the
earliest releases up to mid 1980, and then all contributors toward the
later BSD releases, and then significant participants toward the modern
day BSDs. I attempted contacting maybe hundreds of these participants
via email or some social network sites. I heard back from over a 100.
Generally before I contact the person, I search my now-massive digital
collection of documents and then summarize what I know to help the
participant recollect history from long ago. I did over 80 interviews by
email, several by phone, and some in-person (at some conferences). All
my interview notes are stored in separate per-person files. Often the
participants name other people I should consider. Looking now, I see I
have at least 454 names in my detailed list. I did interviews with
inventors of many of the open source and Unix technologies in use today.
- organized a file system with categories of different files for notes,
massive TODO list, list of people to contact, books to purchase, etc,
and the collection of downloaded or scanned files categorized by decades
(plus I kept track of where the data was obtained). (I also keep a file
of typos, errata, or other bugs I found in lectures, books, old articles
or documentation, such as name misspellings, wrong years, etc.)
- based on history specific articles and history books plus feedback
from key participants, I defined the key stories that must be told and
the key participants that must be introduced in the book. (But I have
over a hundred side stories that have consumed my time and the book!)
- setup a subversion repo to track my book writing in LaTeX. Used
comments to integrate notes in from all the histories I found,
timelines (94 events), and integrated interviews.
- used old Usenet / newsgroups archives to supplement stories. (Google
Groups has been useful with its date-based searches.) I also found
various archives available for downloads. In addition, even some
newsgroup postings and old email archives were included in some of the
historical BSD releases and CSRG archives.
- found copies of old magazine articles and found archives old magazines
at libraries. In addition, some contacts scanned documents or shipped
documents to me. I also transcribed or took notes from several
interviews (done by others).
- found many old master theses, university (multiple schools) technical
reports, BBN technical reports, legal documents, old (signed) software
licenses, etc. that had much history details.
- purchased every book that I learned about that contained BSD history
and took detailed notes. (For what it is worth, to give you a size
example of the most expansive BSD history in print: Salus's excellent
Quarter Century book has two chapters totaling around 18 pages; and
McKusick's most-interesting Open Sources chapter is around 7,300 words.
Some of my chapters for short periods of time and for only single
releases are longer.)
I have been working on it for over five years. I try not to cover the
side stories too much; for example I cannot cover in great detail the
history of Unix, history of BBN's related work, history of open source
licensing, nor DEC, PDP-11 or VAX history. (I have noticed that other
participants on this group have written some of this related work and at
least one document is a source in my bibliography -- Thanks Dave W.!)
The amount of stories is amazing, such as connection between
co-inventor of Unix and Berkeley, Berkeley Pascal, Virtual Memory
Extensions for Unix "vmunix", VMS vs. Unix performance, DARPA
research and founding of CSRG, TCP/IP and Internet services, support
for multiple hardware platforms from one source tree, birth of
Berkeley licensing and replacing proprietary software, the continuation
of 2BSD (PDP-11), the Internet Worm, 386BSD and patchkits, birth
of NetBSD and FreeBSD, AT&T/CSRG/BSDI/Novell legal proceedings,
OpenBSD fork, and much more.
Plus lots of interesting side stories: adventure and trek; INGRES
database; SPICE2; birth of DNS; Email services (beyond and before
SMTP); commercial Unix licensing and then open source licensing;
first multi-processor support; revision control systems, BBN and
Unix and ARPANET work; founding of Sun (and other BSD-based Unixes);
and the intersection of BSD, Lucasfilm, Pixar, CMU Mach, NeXT, BSD
again, Rhapsody/Darwin/MacOSX, etc. Lots of the stories are outside
of University of California at Berkeley.
Four other potential book ideas from my work include: history of
open source licensing (as a side project I contacted the developers
of the code containing the first "open source" style licenses
known), history of X (Andrew window system, Verex, V window system,
W window system, and then MIT wouldn't use Stanford's code so birth
of "X", etc.), technical history of SUN and Sun Microsystems, and
history of MacOSX (starting over a decade earlier, see my "intersection"
above).
I have missed several self-imposed deadlines for publishing the book. I
have learned that every direction I take seems to bring in more
never-ending stories (for example my recent research and interviews
introduced the history of Ultrix), so I have broke it down into a
few key things I need to cover and then ignore the many side stories
and TODOs not fully researched to save them for a "second edition"
of the book. My current work is to complete some integrations of
my interviews and finish tying in the stories; pay for proofreading
and editing for writing style; finish bibtex entries; send chapters
to my many waiting reviewers; integrate their feedback; complete
indexing; pay for cover artwork; and pay for another round of
proofreading. I will reset my goal: get the book to my printer by
end of August.
If anyone has advice or suggestions for research methods for similar
technical computer or software history books, it may be interesting to
continue a discussion about it.
I'd like to encourage any of you with an interest in the history of open
source licensing, Unix, Berkeley Pascal, or BSD to contact me.
Thanks for reading!
Jeremy C. Reed
echo 'EhZ[h ^jjf0%%h[[Zc[Z_W$d[j%Xeeai%ZW[ced#]dk#f[d]k_d%' | \
tr '#-~' '\-.-{'
(This is a different book.)
p.s For fun ...
miscellaneous notes and todos:
t1:bsd-history$ wc * | grep " total$"
30781 140722 2035160 total
latex files:
t1:bsd-history$ wc svn-bsd-history/*tex | grep " total$"
33088 223360 1425282 total
t1:bsd-history$ egrep 'CITE:|\\cite{' svn-bsd-history/*tex | wc -l
1457
t1:bsd-history$ grep '\\bibitem' generated/book.bbl | wc -l
211
most of my interviews:
t1:bsd-history$ wc interviews/{done,todo}/*-interview* | grep " total$"
15948 131526 750691 total
my collection of related Usenet postings, technical reports, SCCS
histories, old BSDs, etc:
t1:bsd-history$ du -sm archives
5306 archives
t1:bsd-history$ find -x archives -type f | wc -l
539001
