Etree
Encyclopedia
etree, or electronic tree, is a music community created in the summer of 1998 for the online trading of live concerts. etree pioneered the standards for distributing lossless audio
on the net and only permits its users to distribute the music of artists that allow the free taping
and trading of their music.
First, a problem common to all curators: source material degrades over time. In particular, the magnetic audio tape used to make many live audio recordings physically decays and, as it is repeatedly played back, loses its clarity. Preserving musical source material, therefore, meant restricting access to it. As a result, archival music may have been preserved, but it was not being heard by anyone. Similarly, individuals who possessed live concert recordings were typically unable to store them appropriately (in climate controlled, fire-safe storage, for example) and/or they lacked the ability to make copies for archiving and preservation. Essentially, the musical history of 20th Century concert performances was being lost, locked up in vaults or decaying in attics and living-room bookcases.
Second, copies of analog recordings tend to degrade when copied due to the introduction of hiss or "noise" inherent to the use of magnetic tape. As a result, no two copies are identical, and each copy, or generation, sounds inferior to the generation preceding it.
Third, given the pre-Internet nature of exchanging live recordings (described below) and the fact that the right to copy many such recordings was or is quasi-legal, the provenance, or "lineage" of many recordings was poorly documented. Even today, historians and collectors find much confusion as to date, venue, setlist, etc., in early bootleg recordings. Curators and collectors searching for source material, or simply the best-sounding copy of a concert recording, were required to spend considerable time accumulating multiple copies of the (supposedly) same material, comparing recordings, following up with sources, etc. The presence of fraudsters, commercial bootleggers, and other criminals in this area did not help.
Finally, as a matter of historical preservation, the existence of a single copy of an historical object (whatever it may be) presents a significantly greater risk that the object will be destroyed, damaged or lost than if multiple copies of the object exist. Archivists, therefore, prefer to distribute copies of historical material as widely as possible, to reduce the risk that all copies are destroyed and the object be lost forever.
First, by the late 1990s, mechanisms for capturing or transferring recordings to the digital domain were well-developed. Digital, magnetic formats and media like Pulse Code Modulation (PCM) and Digital Audio Tape
(DAT), or optical media like Compact Discs (CDs), and other types of digital storage, permitted archivists to record concerts in a manner that reduced or eliminated the degradation of source material when played back or copied. Copies of such recordings could be made that were exact duplicates of the original recording, and such copies do not exhibit degradation in the way that analog audio tape does. Thus, today, digital recordings are typically made to DAT, optical disc, or to hard drives, flash memory, and other types of digital storage.
Second, the emergence of the ability to transform musical recordings to computer data files (such as .wav and .aiff files, which are containers for PCM data) permitted collectors to verify the identity of duplicate copies of a particular digital, or digitized analog, recording. This is typically done by generating a checksum of the data in a file, usually in the MD5
format, and comparing that checksum to a checksum for another file, or a known checksum of the original file. If the checksums match, the files are identical; if not, then the files are different. Such matching copies are referred to as "lossless" copies (to distinguish them from both degradable media like analog tape, and from file formats like .mp3, which remove audio information in order to reduce file size). Such copies are usually bundled with a text file including information about the recording such as date, venue, setlist, recording equipment used, etc., that reduces uncertainty and error in establishing recording provenance and comparing recording sources.
Third, distribution of lossless audio data became easier as the Internet developed. Historically, distributing copies of live music to collectors and archivists faced a bottleneck, in that collectors had to find each other and arrange to transfer copies of physical media (discs, tapes, etc.) in person or through the U.S. mail. One way of expediting distribution was to create a "tree" of people, the "seeder" of which would make copies of a "master" recording and deliver a low-generation copy to each "branch" of the tree, the members of which would then pass the low-generation master along to each "leaf" on the tree branch, thus speeding distribution greatly while minimizing generational loss (for analog material). Still, this was slow and liable to fail if a single person on the branch of the tree did not follow through.
The idea of transferring DAT-quality audio files via the Internet – i.e., an "e-tree" – was first discussed in 1996, but it was impractical at the time due to the large file sizes required to keep the quality intact. For example, a 74 minute CD holds approximately 640 MB of uncompressed PCM data, and a two-hour concert would require two CDs. Transferring a single CD worth of data over a dial-up modem takes approximately 7 days at dial-up modem speeds.
Several developments in computer technology made the fourth factor, the lossless file transfer over the Internet, possible. First, the Shorten
(SHN) file format was developed by a company called SoftSound. The Shorten process non-destructively removes extraneous data within PCM .wav files, reducing their size by approximately 45–55% while allowing the resulting SHN files to be expanded to their original form without the loss of any audio data. (The newer FLAC
format has largely replaced SHN and is now preferred.) These digital audio files, called “filesets”, are thus bit-perfect copies, identical to their original sources, and can be played on virtually any computer, converted to the appropriate format to be burnt to CD for playback on home stereo systems, or converted to other formats for use on portable music players. Second, the explosive growth of the Internet allowed many more people to set up File Transfer Protocol
(FTP) servers to distribute SHN copies of their recordings at high speed to users with broadband Internet connections. Third, mailing lists, e-mail, listservs, etc., allowed collectors and curators to locate each other and material of interest more easily.
Once these events happened, the etree community was formed by members of two highly-regarded online music-trading communities; Sugarmegs Audio and PCP (People for a Clearer Phish). Starting with 10 people, etree.org saw staggering growth rate. By February 2001, there were almost 300 independent FTP servers providing the trunk of etree.org to over 12,000 users. Tools for creating, packaging, verifying and fixing lossless filesets were developed, and included programs like mkwACT, Shorten, Shntool, and others.
BitTorrent was written with etree in mind and etree was the only bittorrent listing site linked from the official FAQ for quite some time. As BitTorrent gained in popularity, and as the availability of free, high-bandwidth FTP servers was restricted by universities and corporations, the number of etree FTP servers steadily declined, and by 2004, few remained in active service. Still, etree.org continued to grow exponentially. As of June 2010, there were nearly 400,000 registered users of db.etree.org who have contributed more than 480,000 setlists for 42,000 artists, and helped to distribute more than 90,000 lossless recordings for approximately 140 artists, to the world's cultural heritage through bt.etree.org and other like-minded online communities.
etree.org consists of four related parts: a wiki, a database, a distribution mechanism, and a community space. First, db.etree.org consists of a large database of performance data for thousands of artists – who played what, where, when, and with whom -- as well as lists of lossless filesets of “trade-friendly” artists like the Grateful Dead
, Phish
and many, many others, who permit their fans to record and copy their concert performances. Where appropriate, these resources are linked to the Internet Archive
's Live Music Archive collection, often known as the LMA or “Llama”. The advantage of a central database of lossless filesets is that collectors and historians can quickly determine what fileset they have received, and read community feedback about the accuracy, quality and other characteristics of the recording. Gone are the days of reviewing multiple cassette tapes looking for the best copy of the same recording.
The etree wiki was put in place around 2001 because the main etree.org page was no longer regularly updated. Although initially met with abundant cynicism, this new technology proved its worth, and is a thriving, successful wiki.
db.etree.org was created by Tom Anderson to keep track of shows he owned without the need of entering the setlist for each newly acquired show. He created toma.dhs.org in August 1999 and made the site multi-user in November. An etree domain was granted in January 2000. The origins of sources tracked at db.etree.org lie in handwritten lists maintained by Greg and Diana Hamilton. (The addition of MD5 checksums as an organizing mechanism came later.) These were imported into db.etree in July 2000 and a new section was created on the site that lets anyone become a volunteer to track sources for any trade friendly band.
bt.etree.org is a BitTorrent tracker
that has replaced the old network of FTP servers. bt.etree.org allows db.etree.org users to torrent filesets of trade-friendly artists for community members to download and enjoy. The copying of recordings by non-trade-friendly artists is not permitted. More than 30,000 lossless filesets have been torrented, and these recordings, each now in the possession of dozens or hundreds of people, will be preserved for future generations. Announcements of newly added torrents may be obtained by subscribing to the mailing list at mail.etree.org.
Community involvement and support are provided by http://forums.etree.org
People who directly support the etree.org community include IT and database administrators, mailing list administrators, and countless others who seed lossless filesets to bt.etree.org, or who add performance data to db.etree.org. etree.org is truly a community effort and without the support of all involved, it simply would not exist.
Lossless data compression
Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data. The term lossless is in contrast to lossy data compression, which only allows an approximation of the original data to be reconstructed, in exchange...
on the net and only permits its users to distribute the music of artists that allow the free taping
Taper (concert)
A Taper is a person who records musical events often from standing microphones in the audience for the benefit of the musical group's fanbase. Such taping was popularized in the late 1960s and early 1970s by fans of the Grateful Dead. Audio recording was allowed at shows and fans would share their...
and trading of their music.
Background
etree.org was created because collectors and curators of live music recordings historically faced four related problems:First, a problem common to all curators: source material degrades over time. In particular, the magnetic audio tape used to make many live audio recordings physically decays and, as it is repeatedly played back, loses its clarity. Preserving musical source material, therefore, meant restricting access to it. As a result, archival music may have been preserved, but it was not being heard by anyone. Similarly, individuals who possessed live concert recordings were typically unable to store them appropriately (in climate controlled, fire-safe storage, for example) and/or they lacked the ability to make copies for archiving and preservation. Essentially, the musical history of 20th Century concert performances was being lost, locked up in vaults or decaying in attics and living-room bookcases.
Second, copies of analog recordings tend to degrade when copied due to the introduction of hiss or "noise" inherent to the use of magnetic tape. As a result, no two copies are identical, and each copy, or generation, sounds inferior to the generation preceding it.
Third, given the pre-Internet nature of exchanging live recordings (described below) and the fact that the right to copy many such recordings was or is quasi-legal, the provenance, or "lineage" of many recordings was poorly documented. Even today, historians and collectors find much confusion as to date, venue, setlist, etc., in early bootleg recordings. Curators and collectors searching for source material, or simply the best-sounding copy of a concert recording, were required to spend considerable time accumulating multiple copies of the (supposedly) same material, comparing recordings, following up with sources, etc. The presence of fraudsters, commercial bootleggers, and other criminals in this area did not help.
Finally, as a matter of historical preservation, the existence of a single copy of an historical object (whatever it may be) presents a significantly greater risk that the object will be destroyed, damaged or lost than if multiple copies of the object exist. Archivists, therefore, prefer to distribute copies of historical material as widely as possible, to reduce the risk that all copies are destroyed and the object be lost forever.
Budding and Growth
etree evolved at the intersection of the technological solutions to these problems.First, by the late 1990s, mechanisms for capturing or transferring recordings to the digital domain were well-developed. Digital, magnetic formats and media like Pulse Code Modulation (PCM) and Digital Audio Tape
Digital Audio Tape
Digital Audio Tape is a signal recording and playback medium developed by Sony and introduced in 1987. In appearance it is similar to a compact audio cassette, using 4 mm magnetic tape enclosed in a protective shell, but is roughly half the size at 73 mm × 54 mm × 10.5 mm. As...
(DAT), or optical media like Compact Discs (CDs), and other types of digital storage, permitted archivists to record concerts in a manner that reduced or eliminated the degradation of source material when played back or copied. Copies of such recordings could be made that were exact duplicates of the original recording, and such copies do not exhibit degradation in the way that analog audio tape does. Thus, today, digital recordings are typically made to DAT, optical disc, or to hard drives, flash memory, and other types of digital storage.
Second, the emergence of the ability to transform musical recordings to computer data files (such as .wav and .aiff files, which are containers for PCM data) permitted collectors to verify the identity of duplicate copies of a particular digital, or digitized analog, recording. This is typically done by generating a checksum of the data in a file, usually in the MD5
MD5
The MD5 Message-Digest Algorithm is a widely used cryptographic hash function that produces a 128-bit hash value. Specified in RFC 1321, MD5 has been employed in a wide variety of security applications, and is also commonly used to check data integrity...
format, and comparing that checksum to a checksum for another file, or a known checksum of the original file. If the checksums match, the files are identical; if not, then the files are different. Such matching copies are referred to as "lossless" copies (to distinguish them from both degradable media like analog tape, and from file formats like .mp3, which remove audio information in order to reduce file size). Such copies are usually bundled with a text file including information about the recording such as date, venue, setlist, recording equipment used, etc., that reduces uncertainty and error in establishing recording provenance and comparing recording sources.
Third, distribution of lossless audio data became easier as the Internet developed. Historically, distributing copies of live music to collectors and archivists faced a bottleneck, in that collectors had to find each other and arrange to transfer copies of physical media (discs, tapes, etc.) in person or through the U.S. mail. One way of expediting distribution was to create a "tree" of people, the "seeder" of which would make copies of a "master" recording and deliver a low-generation copy to each "branch" of the tree, the members of which would then pass the low-generation master along to each "leaf" on the tree branch, thus speeding distribution greatly while minimizing generational loss (for analog material). Still, this was slow and liable to fail if a single person on the branch of the tree did not follow through.
The idea of transferring DAT-quality audio files via the Internet – i.e., an "e-tree" – was first discussed in 1996, but it was impractical at the time due to the large file sizes required to keep the quality intact. For example, a 74 minute CD holds approximately 640 MB of uncompressed PCM data, and a two-hour concert would require two CDs. Transferring a single CD worth of data over a dial-up modem takes approximately 7 days at dial-up modem speeds.
Several developments in computer technology made the fourth factor, the lossless file transfer over the Internet, possible. First, the Shorten
Shorten
Shorten is a file format used for compressing audio data. It is a form of data compression of files and is used to losslessly compress CD-quality audio files . Shorten is no longer developed and more recent lossless audio codecs such as FLAC, Monkey's Audio , TTA, and WavPack have become more...
(SHN) file format was developed by a company called SoftSound. The Shorten process non-destructively removes extraneous data within PCM .wav files, reducing their size by approximately 45–55% while allowing the resulting SHN files to be expanded to their original form without the loss of any audio data. (The newer FLAC
FLAC
FLAC is a codec which allows digital audio to be losslessly compressed such that file size is reduced without any information being lost...
format has largely replaced SHN and is now preferred.) These digital audio files, called “filesets”, are thus bit-perfect copies, identical to their original sources, and can be played on virtually any computer, converted to the appropriate format to be burnt to CD for playback on home stereo systems, or converted to other formats for use on portable music players. Second, the explosive growth of the Internet allowed many more people to set up File Transfer Protocol
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
(FTP) servers to distribute SHN copies of their recordings at high speed to users with broadband Internet connections. Third, mailing lists, e-mail, listservs, etc., allowed collectors and curators to locate each other and material of interest more easily.
Once these events happened, the etree community was formed by members of two highly-regarded online music-trading communities; Sugarmegs Audio and PCP (People for a Clearer Phish). Starting with 10 people, etree.org saw staggering growth rate. By February 2001, there were almost 300 independent FTP servers providing the trunk of etree.org to over 12,000 users. Tools for creating, packaging, verifying and fixing lossless filesets were developed, and included programs like mkwACT, Shorten, Shntool, and others.
BitTorrent was written with etree in mind and etree was the only bittorrent listing site linked from the official FAQ for quite some time. As BitTorrent gained in popularity, and as the availability of free, high-bandwidth FTP servers was restricted by universities and corporations, the number of etree FTP servers steadily declined, and by 2004, few remained in active service. Still, etree.org continued to grow exponentially. As of June 2010, there were nearly 400,000 registered users of db.etree.org who have contributed more than 480,000 setlists for 42,000 artists, and helped to distribute more than 90,000 lossless recordings for approximately 140 artists, to the world's cultural heritage through bt.etree.org and other like-minded online communities.
Community
etree.org is organized by people who love music, in collaboration with archivists and historians.etree.org consists of four related parts: a wiki, a database, a distribution mechanism, and a community space. First, db.etree.org consists of a large database of performance data for thousands of artists – who played what, where, when, and with whom -- as well as lists of lossless filesets of “trade-friendly” artists like the Grateful Dead
Grateful Dead
The Grateful Dead was an American rock band formed in 1965 in the San Francisco Bay Area. The band was known for its unique and eclectic style, which fused elements of rock, folk, bluegrass, blues, reggae, country, improvisational jazz, psychedelia, and space rock, and for live performances of long...
, Phish
Phish
Phish is an American rock band noted for its musical improvisation, extended jams, and exploration of music across genres. Formed at the University of Vermont in 1983 , the band's four members – Trey Anastasio , Mike Gordon , Jon Fishman , and Page McConnell Phish is an American rock band...
and many, many others, who permit their fans to record and copy their concert performances. Where appropriate, these resources are linked to the Internet Archive
Internet Archive
The Internet Archive is a non-profit digital library with the stated mission of "universal access to all knowledge". It offers permanent storage and access to collections of digitized materials, including websites, music, moving images, and nearly 3 million public domain books. The Internet Archive...
's Live Music Archive collection, often known as the LMA or “Llama”. The advantage of a central database of lossless filesets is that collectors and historians can quickly determine what fileset they have received, and read community feedback about the accuracy, quality and other characteristics of the recording. Gone are the days of reviewing multiple cassette tapes looking for the best copy of the same recording.
The etree wiki was put in place around 2001 because the main etree.org page was no longer regularly updated. Although initially met with abundant cynicism, this new technology proved its worth, and is a thriving, successful wiki.
db.etree.org was created by Tom Anderson to keep track of shows he owned without the need of entering the setlist for each newly acquired show. He created toma.dhs.org in August 1999 and made the site multi-user in November. An etree domain was granted in January 2000. The origins of sources tracked at db.etree.org lie in handwritten lists maintained by Greg and Diana Hamilton. (The addition of MD5 checksums as an organizing mechanism came later.) These were imported into db.etree in July 2000 and a new section was created on the site that lets anyone become a volunteer to track sources for any trade friendly band.
bt.etree.org is a BitTorrent tracker
BitTorrent tracker
A BitTorrent tracker is a server that assists in the communication between peers using the BitTorrent protocol. It is also, in the absence of extensions to the original protocol, the only major critical point, as clients are required to communicate with the tracker to initiate downloads...
that has replaced the old network of FTP servers. bt.etree.org allows db.etree.org users to torrent filesets of trade-friendly artists for community members to download and enjoy. The copying of recordings by non-trade-friendly artists is not permitted. More than 30,000 lossless filesets have been torrented, and these recordings, each now in the possession of dozens or hundreds of people, will be preserved for future generations. Announcements of newly added torrents may be obtained by subscribing to the mailing list at mail.etree.org.
Community involvement and support are provided by http://forums.etree.org
People who directly support the etree.org community include IT and database administrators, mailing list administrators, and countless others who seed lossless filesets to bt.etree.org, or who add performance data to db.etree.org. etree.org is truly a community effort and without the support of all involved, it simply would not exist.
Awards
In 2000 and 2001, etree.org won a Jambands.com Jammy Award, in the "Best Fan Web Site" category. These are the first two year's the Jammy's were held. Jambands.com stopped giving this award after 2001.The etree.org community
- wiki.etree.org – Information about etree and how to obtain the music. The information on this site supersedes etree.org
- db.etree.org – A database of live concerts and recordings
- search.etreedb.org – A search engine for the setlists at db.etree.org
- bt.etree.org – A bittorrent tracker for legally tradable artists
- mail.etree.org – Mailing list subscription and archive
- forums.etree.org – Messageboards
- – A brief history of db.etree.org
Related web sites
- www.archive.org/audio – Live Music ArchiveLive Music ArchiveThe Live Music Archive , part of the Internet Archive, is a collection of over 60,000 concert recordings in lossless audio formats. The website is known for its extensive collection of Grateful Dead recordings.-History:...
– the Llama, as it's affectionately known, approached etree.org through the siteops mailing list with a proposition to host lossless audio sets. Although essentially unchanged since production began, the Llama now hosts tens of thousands of concerts in lossless and lossy formats. - www.ibiblio.org – A public library of community projects. etree is the biggest project hosted for free by ibiblio.
- www.furthurnet.com – Another lossless fileset community with historical and continuing links to etree.org