sensory substitution and sensory addition

[vasanthakokilam]

http://www.ted.com/talks/david_eagleman ... #t-1222888

This TED talk is about how the brain can interpret one sensory data ( audio ) through another sense organ and also about equipping us with extra sensory channels.

"our brain doesn't know, and it doesn't care, where it gets the data from. Whatever information comes in, it just figures out what to do with it. And this is a very efficient kind of machine. It's essentially a general purpose computing device, and it just takes in everything and figures out what it's going to do with it, and that, I think, frees up Mother Nature to tinker around with different sorts of input channels."

Though not directly related to music, this mind blowing account of sensory substitution and sensory addition triggered me to speculate about possibilities related to music. Like helping a completely deaf person hear music or can we 'listen' to music by other means than hearing (touch/braille etc.)

Given a model of such functioning of the brain it is not hard to speculate why we 'hear' music in our heads long after the music has stopped. Who knows how much such signals swirl around in our brain which results in us re-hearing it.

This may explain how repeated listening to music make someone a better musician or a rasika which seems to happen rather sub consciously. For the rasika, it is not just improvements in swara and raga identification skills etc but every time you listen to music your brain is not the same as the last time you listened to it. So you relate to the same song/artist/recording differently however small that difference may be.

Our member mahavishnu may have a lot more to stay on these matters scientifically given he is a researcher in this very field. Please correct any leaps in logic I may have made.

==
What I also got out of this talk is the reinforcement of the fact that the brain reconstructs the world that we perceive ( subjective experience ) based on such signals coming at it from all the input channels. It strictly goes by those signals. It is amazing that sitting inside a thick skull, the machinery which calls itself the brain has figured out so much about the outside world. Amazing and startling at the same time. What is startling is the interesting speculation that given our notions of reality/universe is formed by the nature of our input channels, how do even know that our Physics describes reality. What if we had a sensor for some phenomenon we are not even aware of yet, would our physics be different? In other words, can this sensory addition bring out such change in Physics itself and a different view of the world out there?

[vasanthakokilam]

Sometimes I can play a piece of melody on my flute without bringing it down to the swara level. It is not very reliable for me ( meaning I am not good at that ) but I wonder if that has something to do with such sensory substitution but operating on the output side. That is, the brain is making decisions on the finger movements to reproduce the melody without my 100% conscious involvement.Consider the violinists who shadow a vocalist so closely. Are they able to do so because of such sensory substitution?. As the speaker in the above video says, blind people do not scan each braille letter and put together words and sentences and the meaning, the brain frees the conscious mind from such low level activities. It surfaces the fully formed sentences and meaning as a whole. Is the melodic shadowing ability something similar in the reverse direction?

I also wonder the reason why even rote practicing something like music and public speaking make us better is because we are training our brain to do the lower level constitutent things automatically without our conscious involvement so we can focus on the higher level things. In that process it makes us less prone to error when we repeat it.

[kvchellappa]

That was quite interesting. Maybe musicians can wear the vest and sense the audience reaction? I wonder how it is possible to play on an instrument a melody except through the swaras.
Anyone has read this? (while looking for the book The HUman Brain, I saw this)
"This Is Your Brain on Music: The Science of a Human Obsession (English) (Paperback) by Daniel J. Levitin
Released: 2007What can music teach us about the brain? What can the brain teach us about music? And what can both teach us about ourselves?"

[ganesh_mourthy]

With this I would like to inject another interesting capability of our brain which leaves me baffling sometimes.

There is something called thinking of Music and playing your music in your mind or rather than accoustic.

But, if we think of music as only sound, it negates the whole construct of what I am trying to explain.

A simple example is I sometimes can play a raaga alaapana with utmost volume in my violin for even one hour, while my mind wanders off consumed with the several things happened to me in the office, my plans, my frustrations and the result is I have not trained my brain, or rather I could call it destructive practice which I have done several times, or rather this is what I am doing nowadays regularly.

On the contrary, sometimes, sitting in a hustle and bustle street or while running errands, paradoxically I can sing a raaga in my mind aloud and make several constructive phrases and reproduce it the next day aloud in a perfect perfect pitch. Our brain is amazing - it is a shame we only don't like to use it enough.

[mahavishnu]

VK: Just noticed this thread. Yes, David Eagleman's work is quite fascinating. Thanks for your post.

The idea is the brain is "simulating" the sensory world, even in the absence of sensory stimulation now forms the entire basis of the science of cognition! I think we might finally be close to understanding a neuro-scientific theory of imagination, which has eluded us for centuries.

What I find more interesting is that the sensory systems that we have evolved appear to have arisen from the same need for the brain to encode the energy distributions in the environment, whether they be in the optical, acoustic or mechanosensory range. The vary largely in frequency if you think about it. So the fact that one sensory system can (manage to cross-modally) do the function of the other and even have the simulation carry out in the absence of sensory information has amazing potential for the kinds of applications that you spoke of. There is a lot of sensory substitution technology that is already at work in the rehab world.

An extension of this idea is that the motor system in our brain (controlling the moving parts of the body) can also simulate these sensory signals. This has been argued to the evolutionary basis for rhythm perception and production in music. This idea originally put forth by Ani Patel and John Iversen (who is my collaborator at UCSD) is now gaining a lot of traction. I think you will appreciate browsing through this (mostly accessible) article: http://www.ncbi.nlm.nih.gov/pmc/article ... ort=reader

John and I are working on a review article on this topic now. I will post a version of it on this thread for comments when it is ready.

[mahavishnu]

Anyone has read this? (while looking for the book The HUman Brain, I saw this)
"This Is Your Brain on Music: The Science of a Human Obsession (English) (Paperback) by Daniel J. Levitin
Released: 2007"

KVC, I have read this book and I know the author quite well. I would recommend it with the following caveats 1) it is a bit dumbed-down and 2) it is anchored in examples from popular music of the west. This review from the Guardian (from around the book was released) captures is quite well: http://www.theguardian.com/books/2007/s ... ianreview9

That said, Dan is an engaging and excellent writer.

[kvchellappa]

Thanks.

[VK RAMAN]

Music lovers get attached to certain songs and or ragas based on sensory addition and substitution happens when a new one comes. It is not permanent and it is dynamic.

[kvchellappa]

Is that what David is talking about, or VK is speculating?

[vasanthakokilam]

VKR, to be sure, while your succinct observation is on the mark and such substitution does take place, the substitution that David and Mahavishnu refer to are the actual sense channels themselves. Like, 'hearing' through eyes etc., "seeing" through touch etc.

Mahavishnu: Thanks for the reference to the paper by your colleagues. As you stated, it is quite accessible to the lay audience while not losing the scientific rigor. It is quite amazing that they struck that balance so well. They could have easily gone into domain specific gobbledegook losing a lot of people in the process. I got quite a bit out of it and I look forward to reading your review article.

Their hypothesis is that the same area of the brain that is used to plan the body movements is involved in a two way communication with the auditory system for beat perception. One of their predictions is that if we are engaged in activities that require a lot of complicated body movement unrelated to the beat and hence will keep that part of the brain busy, then beat perception should suffer. Quite fascinating. This is what we do while debugging programs to gather some info on where the problem might be especially when we do not have access to the source code. 'If X is the source of the bug, then its effect should be seen in situation Y. Let us see if we see that in Y' kind of reasoning. Here we are talking about doing such things about our brain itself. And we do not have access to the source code!

I am going to try next time I am on an elliptical machine to keep a beat to somebody singing without any mridangam accompaniment or tala sounds of any kind.

[mahavishnu]

What VK Raman refers to is called an "earworm": a song that is stuck is one's head till a new one comes along. Although it is a very interesting phenomenon, it has little to do with what we call sensory subsitution. You can read more about it here:http://en.wikipedia.org/wiki/Earworm. A good (bad) example of this is "kolaveri di".

My colleague Lisa Margulis has done a lot of work on this effect. Here is a recent interview of hers from NPR's Science Friday: http://sciencefriday.com/segment/05/30/ ... heads.html

[mahavishnu]

I just saw this posted today; fits perfectly with our discussion here:
http://ed.ted.com/lessons/earworms-thos ... h-margulis