A month or so ago I was sitting around the Christmas tree at my home in Oregon with my parents, siblings and some extended family of the Beardsworth variety (David, Helen, Michael and Kathleen), who were in visiting Bend from Eugene. Michael is an avid technologist and brought up the then-current introduction of various “E-Book” reading devices. My personal feeling at the time was that these devices were utter crap. However, I didn’t have any particularly well codified intellectual framework for feeling this way, so as is often the case with things that come out of my mouth I rationalized my views on the fly, pulling from anything and everything at the top of my head. This neurological transport is often aided by the introduction of alcohol into my bloodstream, and this case was no exception. At some point during the conversation, I said something to this effect: “E-Books create a new problem where none exists”. This statement may seem at first glance to have no real value, but the point of my comment was to suggest that the consumption of penned books doesn’t suffer from a content consumption problem and that the physical property associated with reading has inherent value. I did, and still, feel this way. I have always loved books, even when they are nothing more than a set of reference documents sitting gathering dust on my shelf. I find reading developed literature and non-fiction in digital form to be somewhat atrocious (yes, I still print out every scientific article I read), and much prefer to have the book in my hands.

“E-Books create a new problem where none exists”. Interesting. It would suggest that there is no inherent value in the digital distribution of large bodies of work, and that the introduction of such a mechanism has major flaws. Certainly it does have major flaws. Digital property rights (on which I have strong feelings but will not discuss them here in any detail), digital distribution, content protection; These are all important topics the majority of which are not yet well developed in our country or in others. So is there some historical precedent on which to judge the value of a paradigm shift from real property distribution to that of digital distribution in something as fundamentally important to our society as literature? Eureka! Music! Let’s take a look.

Since the decline of 19th century classical romanticism and the introduction of the phonograph, the consumption of music by the body public has been inexorably tied to technology. Previously to the 20th century the only method for hearing music was to do one of two things: create music yourself, or go to a concert. Both of these have obvious value even still: the music in my ears right now is being created on real guitars by real people who are doing so for their own enjoyment, and concert-going has a long and uninterrupted tradition throughout all musical disciplines. The record was king in this time of musical upheaval. It may not have been the best system, but it was a decent solution. And music being as important as it is to millennia of civilized human beings, an incredible community was created around the idea of the ‘record’ and perhaps specifically the idea of a codified set of individual works which made up an ‘album’. Incredible. And for quite some time, the album remained king.

It can I think be reasonably argued that the movement of music into the digital realm was necessary and obvious. Even though music was distributed through the sale of a physical product through years past, it was still connected to some mechanism for its translation from the physical product into something that that could be consumed. Thus the art itself was preserved and its basic understanding by the consumer was no different regardless of the mechanism of retrieval. Now of course, it is easy to copy digital content and the idea of a digital work as physical property is horribly flawed, but this leads me down a discussion path to which I will not venture in this post. For sake of discussion I will lump the CD and the digital MP3 download into the same category as they both suffer from these problems. They are both digital content.

This new retrieval mechanism for musical art did, as its predecessor (the vinyl record), have a major impact on the object of artistic desire. Previously it was personal creation and concert-going, which then transitioned into consumption of a physical record or album. Albums themselves had obvious worth: they were the vehicle by which the art was transformed from an abstract encoding into something you and I could actually hear. Perhaps this is much in the same way that the art museum is the vehicle for the consumption of priceless works of visual art. But the introduction of digital content moved the ball into a whole new regime: the album is no longer king. There is no longer a physical vehicle through which music is consumed. None at all. And so what becomes of the album? It dies. It has been dying and will continue to do so. Songs are now sound bytes and albums are now just a collection of songs (An acquaintance of a friend once said to me “You know albums are just a bunch of songs right? You don’t have to get the whole album!” Yes I do, Mr. acquaintance, and you’re an idiot). Much of the music elite in this country and others will tell you that an album is much more than just a collection of songs and sound bytes. Guess what? They’re right. For now. But as has happened previously with any invention of a new content-delivery system, the previous structure that once stood so tall and proud as the be-all-end-all of musical desire will gradually wither away until it is a faint memory for most and an elite preoccupation for few. Perhaps I fit into that elite category; Music is one of the most important influences in my life and I still store all my music in album form (though digitally). I will continue to do so.

So what does this have to do with books and “E-Books” (I still die a little inside each time I hear that phrase, so I quote it every time). Let’s examine the content consumption mechanism of books with perhaps slightly less intent that for music, simply because the subject historically is far easier to understand. In the year 1440, goldsmith Johann Gutenberg invented the printing press. And we haven’t looked back since. The dominant form of literature (and non-fiction) consumption has been and continues to be the printed word. Printing presses have become more complex and more efficient and applied to huge range of content. Newspapers, books, magazines … there isn’t a point in listing them all. Any reader will certainly understand its significance. And, like the record of the 20th century’s effect on music, over the last near-600 years consumption of the printed word is the single most significant mechanism for which thought and information has been presented to the public. The community that has sprung up around the printed word perhaps isn’t even worth mentioning in the same light as that associated with the musical album only because it is so unbelievably significant that no blog post can ever attempt to do it any justice. It transcends volumes of possible discussion. The shear voracity with which the printed word is regarded in civilized society is no foreign concept to any even remotely educated person. Even I suffer from the “endless library” problem. I love to have books just because I love having books. There is something about a physical collection of printed knowledge that is almost mysterious. I can’t explain it, but I have no problem letting it exploit me. For the purposes of this discussion though, it is only necessary to understand that the book, the printed word, is the vehicle, and you and I are the train station.

“E-Books create a new problem where none exists”. What happens when we move the printed word into the digital age (let’s forget about digital rights and digital property for the time being)? Perhaps the best known example of this in current discussion around the world is the “newspaper problem”. I put it in quotes only because I have no idea what a person educated in the field would call this (though perhaps my father John could better define the problem, as he is a very successful and well-established newspaper editor). Other examples (books to “e-books”, magazines to “e-magazines”) are equally well discussed within this framework and my take on it will be as general as possible.

The musical album used to be the purchased content. Now we buy individual songs, and it could be well argued that this system isn’t working all too well (or at least the RIAA would have you believe that — they’re may be right through the lens of the “old” distribution system). In the realm of the printed word, the vehicle of consumption was the physical product. It was the book, the magazine, the newspaper. When you bought one of these items, you weren’t buying it for a specific sentence, phrase, paragraph, chapter or article. The purchase of such a physical item was in fact the transfer your confidence that the physical vehicle had inherent value (I’m overusing the word ‘physical’ here, but with good reason). And almost 600 years of experience tells me that in fact, we were right. The physical vehicle great value which greatly exceeded the collected worth of the individual printed words. I would argue that this is in fact true of the music album. Albums, as any elite music connoisseur will tell you, have a worth that far exceeds the sum of its parts. When you buy a book or magazine or newspaper, you are not only summing its parts; You are putting your faith in the creation of that work as an intellectual object that has inherent value. What an incredible concept.

So for me to read an article (or, because of digital content restrictions read the first two paragraphs of said article) in the Wall Street Journal titled “Publisher Tests Selling by the Chapter” brings to me great distress. Like the album became song snippets and cell phone ring tones, book chapters will no longer be available as a small but integral part of a consistent body of work, but rather as small, insignificant objects to be judged on their own merit. How absurd. (Warning: sentence fragment ahead) The idea that we will enter complacently into a new era where a collected work no longer has value. Ridiculous. I have consistently resisted the destruction of the musical album, and I will be damned if I won’t do the same for the printed word.

I am quite sure that the people responsible for this brand new marketing idea are simply understanding a general trend and following suit. But at what cost? What are we willing to sacrifice? The longstanding traditions of the printed world are one of the fundamental driving forces in our society. These traditions are arguably among the most important single concepts in this history of mankind. Certainly I am not suggesting that the introduction of digital literature and the segmentation of the printed word will destroy the intellectual precedents upon which we base our society. And I would argue that the length and significance of the history of the printed word will necessarily slow its conversion into a group of only intellectual snippets. I hope I die before this transition takes full force in the same way that the album has been destroyed. But maybe tomorrow, maybe next week, maybe a month from now, it will be possible for you to buy and own rights to a digital copy of only “Leviticus” from the Hebrew Bible, Old Testament and the Torah.

What a sad day that will be.

—

A caveat: I am a huge open-access proponent and technologist. I spent the majority of my life dealing with technology in one way or another, whether that be how I think intellectual knowledge should be distributed or how to do a finite element calculation in a complex geometry where chaos in interfaces is important. This is my first attempt at a discussion of this problem, and my thoughts may not be totally clear. The major point of this post is to spur discussion. I want to talk about this with smart people, and if you think I’m horribly misguided, I want you to rip my ideas apart. I’m not kidding.

Why systems like Pandora suck
The problem with things like Pandora is that it relies on a keying system. Those of you that have gmail might make the same type of correlation to “labels” in their interface. This system is completely arbitrary in complexity and in inputs. There is an incredible amount of data out there but each individual person (input) is responsible for organizing their own sets, and no real intelligent analysis of the problem happens past that point, other than basic, basic statistics. So here’s a question. Is the key for ‘genre’ actually a significant one? How significant is it? What keys are most significant? What keys are least significant? Does my assignment of genre X and theme Y to one particular song actually differentiate it at all from any other items? How do I quantify that? This keying system is a supervised approach which absolutely sucks for this type of data.

Factor components
Factor analysis or, in the simple case principal component analysis (PCA), is generally capable of making unsupervised decisions about the significance of arbitrary factors, and creating a series of column vectors describing the most significant ones. I encourage everyone to look PCA and factor analysis up if interested.

Initial application to music classification
Screw manually organizing music into genres and ratings. I want my computer to tell me what I’m going to like without having to do anything other than analyze my current collection. If I had the processing power, I would probably assign some sort of 1D value to every 1/10th of a second second of a song based on some calculation. A perfectly reasonable example might be a frequency/amplitude analysis in the simple case, or perhaps something very fancy like a Fourier/superposition-type deconstruction. Regardless, I would end up with a column vector of values for each song. Then I’d normalize then length such that each column vector for each such is exactly the same, such that a rectangular matrix could be constructed, representing the entirety of music known to my computer. Then I’d probably do a 2D factor decomposition (although this can be done in any dimension, and often is for good reason). The eigenvectors/eigenvalues I get out of the analysis represent the most, and second most variance within the data set. This is generally most easily done with a singular value decomposition (SVD). I would bet almost anyone reading this site that most of the music you actually listen to falls within one quadrant of Factor 1 vs. Factor 2 on Cartesian plane.

Even if your musical taste is extremely eclectic, this would easily give you great differentiation of genre (probably represented in Factor 1, the factor describing the most variance in the data set) without having to actually manually key or classify anything. To those of you who know what I’m working on currently, the similarities between this and my work are, I’m sure, not lost on you.

Other systems
Now you can ask questions: Is genre a separable quantity given the vectors I have constructed from the raw data? Is any key used in the Pandora or Google model actually useful at all? If yes, great … now I can arbitrarily understand a new mp3 or email based on factors that actually have some real significance without having to do any manual classification or introduce any personal bias into the system. I can, of course, write very clever code to do this all for me. This is a general technique applicable to really any system in which you can construct column vectors describing one data point, and in which you want to differentiate between types (for instance, mass spectra). Here’s an example of a 2D plot for such data. Note that the data I used to construct this (and ran my code on) was completely arbitrary and random and therefore no separation was observed.

Google labels suck bad
So let’s extrapolate to a system that everyone is extremely familiar with: Gmail. Even though I get a lot of crap from friends for this, I really like Google. 99% percent of the time they produce products and services that are just downright impressive and well done. However, one of my least favorite concepts is this manual labeling of emails and of RSS feeds, etc. It’s a terrible system. Really terrible. So instead of manually labeling stuff, let’s analyze all your email and figure out everything we need without having to do a damn thing.

1. Take emails and normalize them.
2. Build column vectors describing something about each set of characters in the email.
3. Do the factor analysis.
4. Ask a question: are these N labels significant?
5. If so, apply.
6. Build a new set of vectors describing something different about each character set.
7. Repeat the analysis.
8. Ask another question: do we do better with this vector type?
9. You get the point …

For those of you who love your calculus I could even imagine some very fancy multidimensional function/model describing the way in which we build column vectors to describe character sets in each email, a minimization of that function giving us the best way to construct these column vectors from the data.

Now, when a new email comes in, I know everything about it based on its relationship to the vectors I’ve calculated! Sweet! There’s obviously a little more to this to get it to work right, and this does require fairly large data sets. This would be of basically no use to someone with only a few emails in their account, although there could be a global set calculated from averages of each Gmail user from which initial vectors are calculated for new/small system users.

I will kill spam with this technique
The stuff I’ve described above deals with fairly complex systems. One type of system that is absolutely not complex is spam. Spam filters work in nearly the same way as any keying mechanism in the other systems.

1. if (email_contains_word_viagra) total_score += significance_of_contains_word_viagra;
2. if (email_contains_unresolved_link) total_score += significance_of_contains_unresolved_link;
3. …
4. if (total_score >= spam_score_limit) mark_as_spam(email);

You can see this in action in any email just by looking at the X-Spam: headers (of course, there are other mechanisms that don’t work this way). Go ahead, give it a try. One of the problems with this system is that it assumes everyone gets the same type of spam. They may be right: word N might be the most significant globally, but if I don’t get any spam of that type, what good is it to me? Therefore I think it might be reasonable to address spam as a non-global issue. So, here you go Google, here’s another idea.

1. if (i_mark_something_as_spam) label_email(email, “spam”);
2. factor_analysis_for_spam(factor_results, set_of_real_email, all_current_emails_marked_as_spam);
3. int any_good = can_i_differentiate(factor_results, “spam”);
4. if (any_good) develop_filter_from_factor_analysis(factor_results, “spam”);
5. if (any_good) store_new_filter(factor_results, filter_structure);
6. remove_obsolete_filters(factor_results, filter_structure);

Without having to do anything at all, I have organically developed a new filter for a specific type of spam that I get. These aren’t run in order as their listed but they are just concepts necessary to build such a spam-vector data set.

There are a lot of good answers to good questions about this type of methodology. For instance, applications to visual pattern and facial recognition. Feel free to ask them, but for the brevity of this post, I’m leaving a lot out. It’s already far too long as it is. I’d also like to point out that I think this is the first time I’ve ever used six category “keys” to describe a post. Fitting, isn’t it?

Cheers.

1) Chromatics – Program – [Internet Download]
2) The Black Angels – Manipulation – Passover
3) Black Heart Procession – The Spell – The Spell
4) KRMTX – Palace of Divinity – Yeti 3 Compilation
5) Timesbold – Fencepost – Yeti 3 Compilation
6) Sheriff and the Ravels – Shombola – Songs the Cramps Taught Us
7) The Novas – The Crusher – Songs the Cramps Taught Us
8) The Cramps – The Crusher – Psychedelic Jungle the Cramps Gravest Hits

Any new music suggestions can be sent to rtrev -at- vdov -dot- net

1) Beard – Some Kind of Legitimate Syndrome – Surgery/Embroidery
2) Jesus and the Devil – Bring Out the Guns – Destructive Music Resists The Oncoming Light
3) Grand Ole Party – Insane – (internet download)
4) TV On The Radio – Let The Devil In – Return to Cookie Mountain
5) The Eels – Love of the Loveless – Shootenanny!
6) TV On The Radio – Wolf Like Me – Return to Cookie Mountain
7) Monsters Are Waiting – Don’t Go – Fascination

Bring on the emails… rtrev -at- vdov -dot- net

1) The Robot Ate Me – We Were Humans – Yeti 3
2) Postal Service – Recycled Air (live radio version) – Yeti 3
3) The Old Haunts – Into a New Room – Fuel on Fire
4) Phoenix – Consolation Prizes – It’s Never Been Like That
5) The Painted Saints – Company Town – Company Town
6) Monsters Are Waiting – Ha Ha – Fascination
7) Wyrd Visions – Ceremony – Half Eaten Guitar

1) Rogue Wave – Falcon Settles Me – Out of the Shadow
2) The Shins – New Slang (live) – Know Your Onion! EP
3) Animal Collective – Muffins – Live at the Empty Bottle 4/27/2005
4) Built to Spill – The Weather – Live in Portland 6/23/01
5) Neutral Milk Hotel – Two Headed Boy, Pt. 1 – Live In San Fransisco (1998)
6) Pavement – Box Elder (live) – Slanted & Enchanted: Luxe & Reduxe
7) Modest Mouse – Third Planet – Live In Portland

If you have any leads on good new music send them my way (rtrev -at- vdov -dot- net)

1) Iron and Wine – John’s Glass Eye – Yeti 3 Compilation
2) Rogue Wave (Buddy Holly cover) – Everyday – Stubbs the Zombie Rebel Without a Pulse
3) The Magickal Folk of the Faraway Tree – Being Here Has Cause Me Sorrow – Gold Leaf Branches Compilation
4) Iron and Wine – Muddy Hymnal – The Creek Drank The Cradle
6) Buddy Holly – Everyday – The Best of Buddy Holly
7) Roky Erickson – I Have Always Been Here Before – I Have Always Been Here Before: A Roky Erickson Anthology
8) Jack White – Wayfaring Soundtrack – Cold Mountain OST
9) Wooden Wand – Perch Modifier – Harem of the Sundrum and the Witness Fig

Now thats a lot of songs.

We have some music by classic indie outfits like Beat Happening who were influential in the rise of K Records, and Ween which is simply the greatest band that has ever existed.

1) Beat Happening – Sea Hunt – Music To Climb The Apple Tree By
2) The Islands – Where There Is A Will There’s a Whalebone – Return to the Sea
3) Ween – Ocean Man – The Mollusk
4) Pixies – Wave Of Mutilation – All Over The World
5) Danielson – Cast It At The Setting Sale – Ships
6) The Decemberists – The Mariner’s Revenge Song – Picaresque

1) Roky Erickson – The Creature With The Atom Brain – I Have Always Been Here Before
2) The Sons and Daughters – Monsters – The Repulsion Box
3) Electric Six – Electric Demons In Love – Fire
4) Bright Eyes (Daniel Johnston cover) – Devil Town – The Late Great Daniel Johnston: Covers
5) T. Rex – The Sea Beasts – Unicron
6) The Unicorns – Sea Ghost – Who Will Cut Our Hair When We’re Gone
7) Beck – Death Is Coming To Get You – Golden Leftovers
8) Ralph Stanley – O Death – O’ Brother Where Art Thou Soundtrack

8 Songs!? Someone stop these creatures!

1) Futureheads – Back to the Sea – News & Tributes
2) State of the Ark – Hey Kwanogoma! – The Ark
3) The Black Angels – The Sniper at the Gates of Heaven – Passover
4) The Futureheads – Worry About It Later – News & Tributes
5) Roky Erickson – True Love Cast Out All Evil – I Have Always Been Here Before: The Roky Erickson Anthology
6) State of the Ark – Clamor for Glamour – The Ark
7) The Black Angels – Bloodhounds on My Trail – Passover

That should save your soul… at least for now.

1) The Lovely Feathers – Photo Corners – Hind Hind Legs
2) The Akron/Family – Raising Sparks – Live at Treehouse Records in Minneapolis, MN
3) The Charlatans – City of the Dead – Simpatico
4) The Specials – Ghost Town – Ghost Town (single)
5) Beirut – Postcards from Italy – Gulag Orkestar

As always the Rt. Rev. Fischer always responds to email addressed to rtrev -at- vdov -dot- net

1) Man Man – White Rice, Brown Heart – The Man in a Blue Turban with a Face
2) Bird Names – Snow – Fantic Yard
3) Man Man – Monster – Man Man EP
4) Stoney – Jailbird – only available online
5) Man Man – Black Mission Goggles – Six Demon Bag
6) Wooden Wand – The Flood – The Flood

Yes, acosta looks exactly like that. I would never joke around about something like that.

1) Modest Mouse – So Much Beauty In Dirt – Everywhere and His Nasty Parlour Tricks
2) The Mountain Goats – Stars Fell On Alabama – Nine Black Poppies
3) Danielson – Five Stars And Two Thumbs Up – Ships
4) Editors – Let Your Good Heart Lead You Home – Cuttings
5) The Islands – Humans – Return To The Sea
6) White Rose Movement – Love Is A Number – Kick
7) Modest Mouse – Here It Comes – Everywhere And his Nasty Parlour Tricks

As always you can reach the Rev. for any kind of comments, suggestions, corrections, denunciations, etc. @ rtrev -at- vdov -dot- net

1) The Delgados – Russian Orthodox – Peloton
2) The Gris Gris – For The Season – For The Season
3) Kelley Stoltz – The Rabbit Hugged The Hound – Below The Branches
4) Brian Gaze – Oh Baby – Don’t Go To The Sea – Let’s Go To The Sea
5) The Mountain Goats – Neon Orange Glimmer Song – Sweden
6) The Octopus Project – Exit Counselor – One Ten Hundred Thousand Million
7) The Capricorns – End of The World Love Song – Pure Magical Love

As always you can reach the Rev. for any kind of comments, suggestions, corrections, denunciations, etc. @ rtrev -at- vdov -dot- net

1) Portugal. The Man – Chicago – Waiter: “You, Vultures”
2) Park – Stars Start Shining Bright Above Chicago – Bemusement
3) Baby Blue Sedan – Modest Mouse – Building Something Out Of Nothing
4) Colin Meloy – Jack The Ripper – Colin Meloy Sings Morrisey
5) The Hold Steady – Chicago Seemed Tired Last Night
6) Sufjan Stevens – Chicago – Illinoise
7) Neutral Milk Hotel – Where You’ll Find Me Now – On Avery Island

As always you can reach the Rev. for any kind of comments, suggestions, corrections, denunciations, etc. @ rtrev -at- vdov -dot- net

There are also contributions from The Mountain Goats specifically from the Going To series, The Flaming Lips, and not to mention the backwards speaking Fiery Furnaces who have produced a nearly accessible album that remains experimental… Bitter Tea.

That is all I have until Friday:

1) The Mountain Goats – Going To Bolivia – Sweden
2) The Mountain Goats – California Song – Sweden
3) The Fiery Furnaces – Black Hearted Boy – Bitter Tea
4) White Rose Movement – Love Is A Number – Kick
5) The Octopus Project – The Adjustor – One Ten Hundred Thousand
6) Editors – Someone Says – The Back Room (NOT *the* Editors… just Editors)
7) The Mountain Goats – Going To Georgia – Zopilote Machine
8) The Flaming Lips – Free Radicals – At War With The Mystics

8 songs!! This has to stop. As always you can reach the Rev. for any kind of comments, suggestions, corrections, denunciations, etc. @ rtrev -at- vdov -dot- net

1) The Figurines – Release Me On The Floor – Skeleton
2) Figurine – The European Beauty remixed by Phasmid – Reconfigurine
3) Figurine – Impossible – The Heartfelt
4) The Mgmt – Kids – Time To Pretend
5) The Streets – When You Wasn’t Famous – The Hardest Way To Make and Easy Living
6) Gnarls Barkley – The Boogie Monster – St. Elsewhere

As always you can reach the Rev. @ rtrev -at- vdov -dot- net

1) Tapes ‘n Tapes – Insistor – The Loon
2) The Capricorns – The Sailor – Pure Magical Love
3) Wooden and the Vanishing Voice – Dread Effigy – Gypsy Freedom
4) Wooden and the Vanishing Voice – Effigy in Dread – Earth and Turf
5) Carissa’s Wierd w/ Jenn Ghetto (The Six Parts Seven cover) – On Marriage – Lost Notes From Forgotten Songs

As always you can reach the Rev. @ rtrev -at- vdov -dot- net.