Music Visualization

[vasanthakokilam]

There has been a quiet revolution going on in Data Visualization. This is partially a side effect of touch based hand held devices being widely available. This is a broad field and there are some incredible TED lectures that talk about how to present complex data in an easy to understand manner.

But at a more modest level, look at this: http://selection.datavisualization.ch/

for a variety of creative ways of depicting data.

I am wondering if there are similar ideas for visualizing music and musical notation. We seem to be stuck in the centuries old ways of visualizing music notation through swaras arranged in thala anga boundaries or the western staff music notations. These were all developed during a time when these modern facilities were not available. I have a feeling that there may be creative ways of looking at music in a much more intuitive way.

Has there been any work that we can look at and discuss? Or we can come up with our own creative methods that are suitable for carnatic music. Imagination and creativity is the only limit as long it increases the pleasantness and usefulness for the various purposes that music visualization is typically used. Just to give you ideas, think of heat maps that you can see in the above link where data is color coded. For music, may be swaras can be color coded along with some dimensional elements to represent duration or a combination of a few such things that readily standout in an intuitive way what that means.

One low hanging fruit in all this is the pitch contour of a song. This is especially useful for a gamaka centric music like CM. The X axis is time and the Y axis the swaras but the melody is drawn as a curve exactly as it is sung. If you look at such a graph and follow the song, you can glance to the right a little bit and easily guess how the music is going to go without really knowing much about music or swaras. It is that intutive. I said it is a low hanging fruit but that does not mean it is easy but the benefits are enormous.

There is probably a lot more that can be done in this music visualization area.

[cmlover]

An excellent topic VK.
I hope our computerr guys have a look at this, especially Arun
who has already modernized CM notations.
Definitely Cm should move from outdated notations to graphic representation...

[arunk]

I have thought about it (and I would say it is still something I fancy building eventually once all the building blocks are in place - I am still at the building block stage).

The question is whether the sangatis/phrases can be understood and ingrained better by students etc. to learn something quickly and retain it better (i.e. efficiency). I guess it s quite possible if an entire gen. of students are "trained" that way - but will it be necessarily better or more efficient than existing methods? I really don't know for sure. I am more confident in wagering that most stakeholders would be skeptical than welcoming. I say this because for most practitioners, the existing methods are probably working quite well - this with really minimal help in terms of notation (i.e. compared to western scheme). Thus all this could hold interest mainly to those with an academic/research/dabbler mindset.

However, the interesting thing is people learning instruments certainly know the intricacies of the gamakas (i.e. the pitch contour) and eventually internalize them to a level that it all comes "naturally". For example (I hope I am remembering the details correctly!), at this season's lecdem at MA by Ravi Kiran, he mentioned how one flavor tODi ga, would start from r1 go to m1, but then oscillate between r2 (yes, r2) and m1. Instrumentalists probably know this as they are all executing this ga. Many vocalists (i.e. students) may not know this but would execute this perfectly when they hear their teacher demonstrate it. Heck, even dabblers/part-time-students like me can execute it but I didnt even know r2 was in play until I heard it that way! In fact, one of the attendees (one of the many practioneers/experts/legends there), there asked Ravi Kiran how that ga would be if it were to executed just between r1 and m1 (i.e. no r2). Ravi Kiran said, (paraphrasing) it would sound awful, and he demonstrated it immediately. And yes, it sounded off from the familiar todi context. The point again is that many of us probably could not "imagine" if it would sound right, but at the same time many of us knew what the right ones were (to know it wasn't right) without being conscious of their intricacies!

So does knowing the precise pitch contours while learning make it better? I would still like to think yes, but it is a belief . I have been at the receiving end, and I have also observed vocal lessons, where the teacher clearly explain and demonstrate the anuswaras of a gamaka when the student does not yet get a sangati or part of a sangati right. So that is why I would like to think a mode of teaching which has some related audio-visual help could be useful. Though, I am not yet convinced that it is a sorely needed help.

Arun

[cmlover]

You have right thought but you are still a skeptic!
IMHO it is easy to learn dance postures than musical subtlities since it is visual.
The eyes have greater comprehension power than the ears; that is evolutionary!
I agree instrumentalist have a better understanding of gamaka nuances than
vocalists. Whatever Ravikran said can be concisely illustrated in a 'graph' than in so many words.
Your gamakapp was great! Why don't you develop it further and make is user friendly?

[arunk]

You are missing my point cmlover.
1. The illustration wasn't important for rasikas to listen/discern that gamaka as a stape tODi gamaka nor for (many) to reproduce it.
2. The picture may be worth a thousand words - but in this case it is worth zero sound

Arun

[cmlover]

I see your point.
But I am thinking of the practitioners rather than the Rasikas..
Make life easy for the learner with visual cues rather than try to 'parrot' the sounds!
Also for new experiments (by dabblers

[arunk]

I included practitioners also. I am not saying it would be of no use to all - it can be of use to some unknown (but I think small) %. I just think its value is perhaps overrated.

Arun

[cmlover]

Arun
You are too negative! I can understand why...
You do things because you enjoy doing it!
Incidentally by sharing you enlighten a few others.
Some of them will be genuinely grateful and a few will ignore.
But the greatest joy comes from within you!
You will understand when you reach my age....

[vasanthakokilam]

I am thinking at much broader direction than music notation. Music Visualization can be for a whole lot of purposes including rasikas. Why they would need that and in what form the visualization is presented are all open for some brain storming.

The reason for the link I provided above is just to show an example that traditionally data is shown in line graphs but now there are 15 different ways of visualizing the data that is useful for a particular context. These do not translate directly to music but I think music visualization done in creative ways can be entertaining in itself. I think kids who are not really music students can learn to sing basic songs if the song is presented in a visual manner. I do not know exactly how but that is what I would like to brainstorm.

[cmlover]

I agree with you totally.
The technological advances permit us to present ideas which can be comprehended more easily.
In CM we are still using the outmoded 19th century notations which have to be modernized
with the available computer technology. First of all CM is analogue and the digital notations
are a handicap for understanding. Most students have a teacher who bypasses the notations.
Hence the notations simply serve as just mnemonics..

[msakella]

at this season's lecdem at MA by Ravi Kiran, he mentioned how one flavor tODi ga, would start from r1 go to m1, but then oscillate between r2 (yes, r2) and m1.

Dear brother-member, arunk, In this particular context of Gamaka of ‘Todi-ga’, though not in the context of the present thread, even though what you wrote is correct, there is another very important point to be observed which might have, accidentally, been missed by our learned brother, Chi. Ravikiran to make it clear. If this ‘Todi-ga’ has to be demonstrated either in Veena or Violin or, in fact, in any other instrument it has to be started, as you wrote, from r1 and later has to swing in-between r2 and m1. It is true only in case of singing or playing ‘s-r-g-m’ in the ascending order as, here, ga has to be started from r1 only. But, if only ‘ga’ alone has to be demonstrated, it should not be started from r2, like it is being done by many of our Vocalists and instrumentalists either in the guise of tradition or habit. But it should start from m1 only and to swing downward in-between m1 and r2. That is why, this obviously reveals that Gamaka, though many are not at all aware, should always be sung or played with a downward-glide starting from the upper-note and swinging in-between the upper and the lower notes. This also obviously reveals that entire music is mostly filled with Gamakas of downward-glides only where upward-glide is also automatically included in each of the two in-between downward-glides.

In fact, this is a very interesting matter of discussion of teaching methods in giving the exact picture of all these different oscillations of our complex music to our kids. But, how many are truly interested in holding such discussions of (useless) teaching methods keeping away all the fanatic, lunatic and conservative attitudes in truly serving our kids, is a billion dollar question. amsharma

[vasanthakokilam]

I have been mulling over a few thoughts in this music visualization aspects, specifically as applied to Indian classical music.

The general theme is, if swara level representation is analogous to atomic level representation, the higher level representation is like a molecular level representation of arbitrary size and complexity. ( like how in Organic chemistry, higher level structures are represented in a visually rich form using rings etc which instantly tells you a lot about the structure ). In some cases, higher level structures made up of lower level structures are also easy to discern in such visual representation.

When presented in such visually intuitive ( molecular level ) fashion, the following can be discerned much quickly.

1) A bunch of swaras that produce the raga bhava: I would think the overall visual shape of it would be very similar across the song. My initial sanity check of this shows it is indeed true. So, this is a physical representation of raga bhava, if I may.

2) Such a raga bhava similarity can be extended for visualizing across songs. They all will not be exactly the same shape but in abstract their shape can be discerned.

3) One next step is to normalize various songs in different kattais to a common kattai ( in the visualization space ) so things are easily seen.

4) Once we make some progress, may be there is a way to describe a song by the major "chunks" of shapes of ragas that the song uses.

5) Ragas whose raga bhava is produced by the same set of swaras but with different intonations are easy to see in such visualizations.

6) Moving from those obvious things, such higher level 'chunked' visualizations can prod some creative minds to do some fairly straightfoward 'what if' creativity. Something like 'I see this pattern produces this bhava in this raga'. What if I do the same pattern in a different raga? Does it stay with in the confines of that raga or it produces enough variations to be considered a different raga using the same swaras. Subsequent tools can actually help in such exploration and quickly get the feedback on the results ( how it sounds )

7) Next higher level thing is to actually look at the dynamics of those raga bhava chunks. How such structures are connected to each other and how they evolve in time? I am visualizing now a metaphor of how some creatures like a snail move in time. The whole body has a fungible contour which remains more or less static when they are not moving but while moving the whole shape changes according to the speed and direction of movement. This is just a metaphor for visualizing the dynamics of a song. ( as you can see, this definitely needs some work and refinement but I thought I will put it in even in this uncooked state ).

The reason why these things can actually work better for Indian ( raga ) music is, the music, by definition, is made up of such higher level melodic motifs of a given raga. So one need not necessarily go to the reductionist atomic level of swaras but can operate at the molecular level of arbitrary size and complexity.

[cmlover]

I see an analogy between the 7 swaras and the seven visible colours.
The shades of colours are the swara clusters and their specific gamakas.
In other words a kriti is like a beatiful painting with specific combinations of the colours.
The content of the paintng is the lyrics which can vary.
A vaggeyakara is a painter who uses sound instead of colours!

[vasanthakokilam]

Interesting idea, CML! It is definitely worth mulling over.

We really need 12 colors plus a variation on that color for the three octaves. But that is not a problem, one just has to be judicious so it is obvious.

What this does is, it frees up the Y dimension which can be used for another purpose like strength/amplitude/energy.
So the visualization can look like a bar graph: X axis time, Y axis stress/volume, with the color indicating the swara.
Since amplitude variation also can convey some aspect of inherent rhythmic information, that is a gain as well.

Gamakas can be represented as a combination of colors. LIke if you start at swara A and end at swara B between time t1 and t2, then it will start with the color of swara A and it will start having a tint of swara B. At mid point between t1 and t2, it will be,say, 50-50 and then at t2 it will become the pure color of swara B.

Steady karvais on a swara between t1 and t2 will look like a wide bar with height proportional to the volume with the swara color.

Oscillations will look like an interference pattern but it will convey much richer information. We will have to see.

It will take some time to get used to it but it may reveal a few things that are not apparent in other visualizations. Like how heat maps can reveal information very quickly which other types of graphs can not.

[harimau]

Try 'Soundscape' on Apple's AppStore.

Free for one day.

[VK RAMAN]

Vasanthakokilam - various ideas mentioned by you can be empherically proved if you have not already done

[vasanthakokilam]

Try 'Soundscape' on Apple's AppStore. Free for one day.

Got it. Fun stuff. Thanks.