Animation_nodes: Proposal new node : Midi

[Patochun]

Hello,
I start a code for using MIDI events to drive some animation in blender.

Like this : https://www.youtube.com/watch?v=9N9v5mHrADw

Now, I think it's time to release my work for everyone.
I think early to create a blender add-on but animation nodes seems to be the best choice

I work on a proof of concept for now. Animating a list of integer where each of them represent a note and her velocity value. So I limit in this POC for only one MIDI track.

I have few questions to start this :

• How can I debug with an IDE like VSC ?
• How create an animated list ? I manage the elements values with my code.

Animation nodes is a big beauty piece of code. I'm a python dev level 4 than AN requires dev level 15... it's a journey for me.

[Clockmender]

I did all this MIDI stuff over a year ago, including multi-track, etc. see this:

https://www.youtube.com/watch?v=QM0b1PHrigc

There are more like this on my Youtube. Never got any interest, or take up from AN devs... Talk to me if you want to know more.

[Patochun]

Hello Clockmender, what a great job !
It's not clear to me of how did you pass the MIDI evts into AN
Have you coded a new AN node ?

[Clockmender]

I coded many new nodes to read MIDI files, converted to csv format. One that "bakes" the MIDI file to a series of empties, then use lots of other standard and my AN nodes to build animations. In 2.79 I baked the MIDI events to F-Curves in the blend file, but this did not work in 2.8 and I got no help from here as to how to fix it, so for 2.8 I took that option out.

Take a look at this: https://clockmender.uk/blender/animation-nodes/midi-controls/ This will explain it all....

[OmarEmaraDev]

I am not sure what you mean by your second question, but ideally, the node should take a frame number as an input, much like the Sound node.

[Patochun]

Absolutely. The goal is to take the frame number to act with midi events.
I try to create a list output animated with these events. Just to drive a POC for now.

[Clockmender]

Do you have anything to show us yet? You will need to convert MIDI "Pulse" timings, then into seconds, then into Frame timings.... POC????

[OmarEmaraDev]

@Patochun I think discussing more details about what you are doing is of essence here. Otherwise your efforts may not align with the goals of the project. So make sure to share the elementary design and workflow of the system first.

[Patochun]

Design is not net fixed. POC = Proof Of Concept. And yes I convert MIDI time notation to seconds.
I wil take time to write a more detailled project and I post here.

[OmarEmaraDev]

@Patochun I see. Take your time then. Let me know if you need anything.

[Patochun]

To introduce the discussion I write this "MIDI node Design".
@OmarEmaraDev, Tell me by return what you think of this design. What would I need to change or dig more. I would need information on how to create a new socket.

@Clockmender, have you published your code ?
Personally I use the MIDO module for managing MIDI data.

It is a succinct document. Just to know if it's going in the right direction
AN MIDI Design.pdf

[Clockmender]

Reading the MIDI file and writing the animations to something, be that Empties, or F-Curves, should be a one-time operation, not something that is done "on-the fly" at each execution, it will kill your processor like this IMHO.

You can ue a Generic socket for now, until you know what your intended socket is supposed to do.

Yes, my code is published, see my website...

[OmarEmaraDev]

@Patochun I just read your design document, I will think about it and get back to you with feedback.

[Clockmender]

@Patochun Something else to read:

https://blenderartists.org/t/were-in-the-midi-audio-daw-real-time/1142160

[OmarEmaraDev]

@Patochun @Clockmender Here are my initial thoughts.

---

Animation Nodes is data driven by design, not event driven. So we should avoid giving users events and instead give them intuitive structured data. The structure I have in mind is as follows:

Midi File:

• Midi Track 1
• Midi Track 2
• ...

Midi Track:

• Midi Note 1
• Midi Note 2
• ...

Midi Note:

• Note Channel
• Note Number
• Note Off Time
• Note Off Velocity
• Note On Time
• Note On Velocity

I intentionally left out all other events like Polyphonic Pressure and track events because they will complicate the design at this point. But we can add them later, putting in mind that they will not be exposed as events either. For instance, Polyphonic Pressure can be represented as an interpolation or a spline.

In terms of nodes, we can provide Midi Tracks From File, Midi Track Info, and Midi Note Info nodes to access the data of Midi files. Additionally, we can provide utility nodes to filter notes by channel or note number as well as notes to evaluate the sound of a group of notes using ADSR envelops.

[Patochun]

Hi,

@OmarEmaraDev I need some time to read and rethink from your proposal. I fully understand the need to maintain a data approach. You have to animate the data a little.

I just push my POC here :
https://github.com/Patochun/animation_nodes/tree/MIDI-node/animation_nodes/nodes/sound

I am a little old developer and I learn in addition to python (1 year exp) dev environments like github, VSC, etc. By the way it's very difficult to debug with just data print
So I needed first to check that I could integrate my code in a new context
It's just a test but it works a little. There are still some bug. And he approaches my initial design.

I am very happy to be able to participate in something that will be useful to many people. At my modest level.

Patrick

[Patochun]

MTB.py is my initial code very much retouched to fit our goals.

[Clockmender]

@patochun @OmarEmaraDev you both seem to want to re-invent a wheel I solved a long time ago, so I am removing myself from this project of yours. I don't expect, nor want, to see ant if my code, systems, functions, or nodes in your project.

Perhaps, Omar, you might explain to me why you never took up my work, given all I know about MIDI and all I have done for AN. I am very disappointed in this situation, I will not be offering any more help here.

[OmarEmaraDev]

@Clockmender I am sorry you feel this way. I have always shown interest in your work, and Jacques and I always tried to help you with your interesting projects. However, if I am being honest, I never really considered your approach an optimal one, and the code quality was questionable. Your code wasn't even working correctly and you never submitted your code for review, so I didn't handle it at full capacity. Yet, I never gave up on your work, the project's thread is still open for discussion. Feel free to present your project again there, and I will be there discussing it with you again just like I am discussing this project.

You are not the first to try to implement MIDI in Animation Nodes; Leon, Steven, Jacques, and me all tried different approaches at some point and non of our trials made it in Animation Nodes, which is fine, we will just try again. Nobody is reinventing the wheel, we are just trying different approaches.

@Patochun is enthusiastic about the project and wanted to give it a go, he has some very good proofs of concept and I think the project has the potential to finally add MIDI support to Animation Nodes. I will make sure Patochun have all the support he needs to complete the project.

In conclusion, if you think your project is better, feel free to discuss it with us. Don't undermine other projects.

[Clockmender]

Don't undermine other projects.

Oh well, I must say that I have not undermined this one, merely left it.

However, your publicly ridiculing my approach and code might be perceived as you not obeying your own dictates. Let's examine that:

My approach of baking the MIDI file to F-curves was agreed with Jacques and he provided the framework for me to work with, so your criticism here should be aimed at him, not me. This I believe is in line with the way sound is baked for animation, so that approach is optimal, the same for MIDI is apparently not.

The only part of my code that does not work in 2.8 is the part provided by Jacques (reading the baked F-curves), which worked perfectly well in 2.79. Despite repeated requests for resolution here, none was forthcoming, only dismissal.

At least 9 months ago you suggested you and I had a conversation on MIDI, to which I agreed enthusiastically, I never heard from you again on the matter.

In conclusion - Live by the rules you dictate to others.

[OmarEmaraDev]

@Clockmender You stated that we are "reinventing a wheel you solved a long time ago", implying that this project should not go on and that your project should be adopted instead. To me, this sounds like you are undermining the project. But I am sure you didn't mean to, so lets just put this aside.

I never ridiculed your approach, I simply don't consider it an optimal approach, it is just my personal opinion. I apologize if I sounded ridiculing.

It is fine if Jacques agrees with your approach, but that is beside the point. You are the one working on this project and my criticism will be directed to you.

Sound baking used to work that way in Animation Nodes. I considered the approach sub-optimal and rewrote the whole sound system to be more optimal. This is exactly what I am trying to do here, except I want to do it right from the start. So your argument is counterintuitive.

It doesn't really matter which part is not working, what matters is that your code is not working, regardless of the reason. There was no "dismissal", there was me trying to help you repeatedly resolve the issues you were having. It is true that I couldn't pin point the issue in your code, but I really tried, I don't know what is expected from me here. Also, this is not the first time you brought this up, so please reconsider your position.

I don't recall ignoring a conversation with you about MIDI, if I did, please forgive me, but I really don't remember this, maybe I just missed your message. I want everybody to be happy here, and I will make amends if possible. What can I do to fix this situation?

[Patochun]

Hello all, I feel rather uncomfortable given the situation. I apologize for not having foreseen what was going to happen. I understand @clockmender that you think that here we reinvent the wheel considering your work already well advanced. On the other hand, I did not know your work before I started and I also built something to use MIDI to animate.
I'm not super comfortable with English as you can imagine (I'm French).

So I had a little trouble understanding the concept you bet on. We can put together something that marries the best of all worlds. What I have in mind:

• 1 node which takes as input the MIDI file directly. And who does not use a prior conversion to CSV.
  This node, let's call it the MIDI file or whatever, takes the MIDI file, analyzes it in terms of tempo and creates an internal object containing all the necessary for the following nodes.

• 1 node which takes as input the MIDI file directly. And who does not use a prior conversion to CSV.
  This node, let's call the MIDI file or whatever, takes the MIDI file, analyzes it in terms of tempo and creates an internal object containing all the necessary for the following nodes.

• Then we can consider several types of nodes to exploit this object:

-1 node to generate fcurves, one per notes by track?
-1 node that provides a bit more raw data, like a list with the names of the used notes and another list with their velocity value as a function of time
-1 node that will later take care of the data on the tracks themselves.

It seems to me more complicated for the facilitators to have another level of data up to the notes themselves. They would then need a very large number of nodes to gather the life of the notes to animate a track.

With a list or a fcurve you do not lose access to the note. For example, we can locate in the rhythm track the note corresponding to the bass drum to create a pulsed movement.

I keep in mind that the goal is to allow you to animate everything you want from MIDI data.
What do you think, would it be a good way to move forward together without losing anyone?

[OmarEmaraDev]

@Patochun So you are proposing we skip the structure for MIDI Notes and instead provide lists for all the MIDI Note attributes? While this might be advantageous because it present data in a contiguous form, it is disadvantageous because we lose structure and end up with events, bringing us back to the conversation about "event driven" vs "data driven" designs. What do you think?

I fail to see how using FCurves makes sense here. The only way to use FCurves in Animation Nodes is to evaluate them, so why not just take a shortcut and evaluate MIDI notes directly? Unless you are talking about writing FCurves back to objects' animation channels, which is arguably not how we do things in Animation Nodes.

@Patochun Please don't feel discouraged or pressured in anyway. My disagreement with Clockmender is not related to your project and should not affect your project in anyway.

[Patochun]

Only a "structured data" view with note-level granularity, I have trouble figuring out what all the nodes needed to animate a 15-track music with 20 notes per track would look like. Can you enlighten me on it?

[OmarEmaraDev]

This is what I have in mind:

MIDI Tracks From File -> Loop Over Tracks -> Track Info -> A List Of MIDI Notes.
MIDI Notes -> Evaluate MIDI Notes -> Floats
MIDI Notes -> Loop Over MIDI Notes -> Get start and end times

[Patochun]

OK I see the idea better. No work is done upstream, we provide raw data and all the intelligence of the treatment would be realized by the structure of the set of nodes and in particular by the "specialized nodes" for the treatment: loop, and eventually script.

A treatment consists, for example, in damping attacks and releasing notes in order to avoid a "square" signal and difficult animations.

With the unprocessed raw data, will this operation have to be done via the other nodes like script?

Moreover, even before damping anything, it should already position the notes in time. In a MIDI file it is something quite complex, which must take into account the tempo (variable in time) and the quantization of time provided by several data. I imagine that this part will have to be treated minimally by the node that opens the MIDI file and creates the track data?

For raw data, it would be time to rule on the structured data file, tracks, notes. You already proposed something above, I think it is necessary to describe the objects and the fields.

[Clockmender]

What can I do to fix this situation?

@OmarEmaraDev Well we have probably reached the low point and can only go up from here. You could show a little more respect for my age, experience and knowledge and I could refrain from churlish, remarks like explaining that the reason for deficiencies in my code were due to two factors; 1) I was learning to write nodes from a knowledge base of zero and 2) the AN API documentation had not progresses far beyond "inadequate". So let's put that aside and move on.

I have to say that I think you are hopelessly over-worked on this development and could benefit from more assistance, I tried to help, but felt my ideas were largely ignored.

As far as development is concerned, there seems to be a confusion over MIDI data. A MIDI file contains a chronological list of time-stamped events, like "Time, note On, etc." and they do not represent a continuous file of data that can be accessed on a time basis, like sound files. By that I mean that you cannot, for example, ask the file which notes are on and at what velocity, at a point in time without back-tracking the file for the previous event for that note, which might not be in place as far back as the beginning of the file, so reading the MIDI file is inherently inefficient.

The basis of my research here and hence my comment about _re-inventing the wheel_, was not to obviate further work, but to say that I had already resolved that the data must be converted as a one time operation into "continuous", rather than "event" data. I sought to do this by either writing F-curves to the blend file, or writing the same to control empties. I am not so conceited as to suggest that my methods, or code should be adopted.

It should be noted that MIDI data is good at driving sound generation, it is not so good at driving animations. Take an example; at say time = 2.2 seconds note E4 is activated for some time, then immediately re-activated in succession. If the notes are quantised perfect, the time lapse between first note off and second note on is 0 - a great problem for the animator.

The minimum time length a human can achieve is widely considered to be 1/64th second, so frame rate is also important and, indeally should be based upon the BPM and shortest note length. This causes more problems in setting the blend file to be consistent for variable BPM files.

Another problem is the animator does not want a piano key mesh, for example to "snap down" immediately - caused by a lack of damping (as Patrick calls it), or easing as I called it and also wants a definite separation between consecutive executions of the same note - hence my "note separation" value. This gives us the chance to animate a keyboard realistically from perfect quantised MIDI data.

No human could play perfect quantised sequential notes, but there is nothing to stop someone editing a MIDI file with a DAW to achieve just that, or indeed note lengths of shorter than 1/64th second. Often drum tracks consist of very short "trigger" events, rather than note lengths reflective of the sound that is produced by the drum sample.

In terms of efficiency, I have conducted experiments on a MIDI file (Scott Joplin's the Entertainer), that I seem to remember consists of 55 individual unique notes played a total of 7,500 odd times. It took my Mac about 1.5 seconds to make the F-Curves, or controls, from this data, I cannot imagine that this will work "on-the-fly" between two frames even running at 1/24th second. I have also looked at various classical pieces where the note duration and frequency would mean we have to work in 1/64ths of a second, or multiple thereof to get a realistic animation.

The system has to be able to cope with multi-track MIDI files, one of my test MIDI files has 23 tracks and lasts 11 minutes, this is impossible to interpret "on-the-fly" between frames. Then there is the problem if the user "scrubbing the timeline to say frame 1100 then expecting the system to show which notes are on at this point, from a MIDI file this can only be done by back-tracking - Pink Floyd's "Shine on you Crazy Diamond" holds the first chord for 30 seconds, from memory? that is an awful long way to back-track to find the first note-on event.

So, I should welcome any thoughts on these points I have made. I cannot and never did find a way to read _events_ data without first converting it to _continuous_ data, or to avoid the need to clean, ease and separate notes in order to get a satisfactory animation, there are quite a few examples on my YouTube, including a drum animation, to reflect the level of realism I was able to achieve.

Cheers, Clock.

[Patochun]

if you have in your data structure representing a track all the timestamped notes for on and off with a measure in seconds (float), I do not think it is impossible to find the position of the notes at a time T. The data being necessarily sorted in the timebase. Then we can use a dichotomous algo to quickly find the state of the notes that corresponds to the last change of state that may have occurred before for the moment T. Then we must go back as you indicated . There remains the problem of the note supported at time 1 while one looks for what is active at time 1100 (in frames). It's something rare, but all the same.

I think aloud as they say here.

Maybe it would be necessary to create the structure of a track a little like the way in which one optimizes the changes in video (mean compressed video codec). We limit the return by noting at regular intervals (*) the state of all the keys. It is simplified by replacing process times in the amount of data to be stored (almost free).

(*) calculated by the time granularity of the music for example.

What do you think ?
Patrick.

[Clockmender]

I'll think about what you said, I personally never found a more efficient way to handle the data other than a curve, this will always return the state of any note at any time very quickly, but let me know how this progresses. I am concerned with the concept of back tracking the file, for what could be a long way, in between frames where you may only have say 1/64th of a second to do so, along with all the other processing required for the animation.

One thing I forgot to say is that whilst you might not want to use a MIDI.csv file, they are very useful, given they are human readable, for the purposes of debugging and data confirmation.

I agree with you that time granularity is very important, but presents issues when the BPM changes...

Clock.

[Patochun]

To be more precise, I'm talking about the resulting temporal granularity. Understanding the positioning in pulsation by quarter note modulo the tempo that can change often (in one of my example files it changes at each bar).

Then have nodes for notes individually with their timeline and use loops or list nodes evolving with frame event? I think we should test both to see the feasibility in terms of ms.

[Clockmender]

Then have nodes for notes individually

Do you mean one node for each note? I have a file with 23 tracks and average of 35 uniques notes per track - that means a lot of nodes (805), unless I have misunderstood you...

[Patochun]

Yes 805 potentials nodes, that's what I understood from AN's "data" centric design as described by Omar. And this is where loop management comes in. But it also leaves other possibilities that one imagines can not be today ... choose note by note if it's make sense, do some mathematical operations that will make sense for a given animation.

I could see in depth your site and understand that you are looking to recreate existing instruments in 3D (and beautiful work by the way) but there are other ways to imagine a music-based animation. Why not smoke evolving, drops of water falling, things that vibrate without speaking of light.

The data centric versus events vision leaves open all doors to all imaginations

[Clockmender]

Yes, I imagined this also, The original 2.79 method stored an F-Curve for each note in the "Bake" node. Each Bake node held all the notes for one track and output the values from the evaluated F-curves on each frame as a float. So a list of floats was output. Example: Scott Joplin's The Entertainer - the output was a list of 55 floats, one for each unique note played - so one Bake node per track. This was fed into a loop that also had an input Object List of objects to be animated, this could be anything at all and these then responded to the floats.

In some animations I drove Shapekeys, in some I drove Armature Bones, in some I drove Material values, etc. it was completely adaptable to any situation. In the video I posted much earlier here, lights are pulsing to the drum beat for example.

However, in AN 2.1/BLender 2.8 that stopped working, presumably due to design changes in AN, Blender, or both, so I went back to Empties - one per note per track and used the animation F-curves of the Z axis to drive anything, so the empties moved in Z only from 0 (velocity 0, or note off) to 1 (velocity 127) this movement could then be used to drive anything through AN nodes.

The advantage this has over baked F-curves is that you can go in and physically change the F-curves (move keyframes) if you spot problems in the animation, so some notes too close together - just move the keyframes in Graph Editor, whereas the only way to change the baked curves was to edit the MIDI.csv file, the re-run the bake.

Personally, I would be happy with a system that stored all the notes for one track in one node, I am not convinced by one node per note, I think the node tree would be too big, un-workable and confusing.

Take this example, I know the resolution is not good:

Here, 8 tracks are driving many animations and a "sound bake" is driving others as well. There is no reason why one "bake", or "Track" node cannot drive many separate animations, just use more output loops and object feeds.

EDIT:

I am not proposing Empties as the solution, just that we keep one "feeder" node per track.

[OmarEmaraDev]

Please note that the "perceived continuity" of FCurves is only guaranteed by the FCurve evaluation function, however, the FCurve is structurally not continuous, it is just a list of spline anchors. Similarly, our MIDI Notes list will not be continuous, the continuity will be guaranteed by an evaluation function, which we will encapsulate in an Evaluate MIDI Notes node as I described above. Using our own evaluation function is, of course, very advantageous. In particular, it is not destructive, so no baking will have to be done, ever. Also, it is very flexible, we can use ADSR envelopes while evaluating the notes, a high release time can be used to visualize drums even if the note is infinitesimally short.

In addition, a data driven approach does not mean we don't have access to events. In fact, they are readily available inside MIDI Notes for low level access. The same can not be said to the event driven approach.

I see that you are worried about the performance. While I don't have numbers for you, I highly doubt it will be expensive. We we will never have to back-propagate through events, because delta times will be resolved to absolute float times of great precision at parse time. Moreover, the MIDI notes will be parsed automatically sorted, so we can use things like binary search, which is an O(log(n)) algorithm where n is the number of notes. So we shouldn't worry about performance for now.

[Patochun]

Ok Omar, I'm in with your proposal. FCurve insight is very instructive. Have you an idea of structure data and storage (and cache ?) ?

Can we use the MIDO python module or we have to recode the reading and decoding of the MIDI file ?
So it will be a good practice to update my doc and redesign the solution.

[OmarEmaraDev]

@Patochun I presented one such structure above, what do you think about it?
The MIDI tracks will be cached using an LRU cache upon parsing the file, just like we do with sounds.

We will probably write our own MIDI parser, but I think using MIDO at this stage is ok, in fact, I encourage it.

[Patochun]

We have to design what structure in output of the MIDI file node. And obviously the time signature for events inside. Actually I manage with MIDI time code and tempo change to put events in second (float). I think it's the best approach.

[OmarEmaraDev]

I think the node should return a list of MIDI Tracks. And yes, I agree with you approach, delta times should be resolved to absolute float times.

[Clockmender]

So, if I understand you correctly, you are going to parse the MIDI file to cache a set of absolute on-off times for each note? I know that is simplistic, but bear with me. Four questions please:

1) How are you going to cater for note velocity?
2) How are you going to "ease", or dampen movements of meshes, like piano keys? they should not go from off to on in 0 time.
3) How are you going to cater for consecutive quantised-perfect notes? Users will need to see that the note was played more than once, not just held on for the duration of consecutive on-off's.
4) How will users be able to "tweak" the animation, by moving notes a little, changing velocity, etc. without editing the MIDI file in a DAW.

I think you should consider these points now, rather than later... But I like the approach.

EDIT:

Time signature will allow users to set music bars, _tempo_ as a output is more important I think.

[OmarEmaraDev]

@Clockmender

1. Each MIDI Note will store the on and off velocities. A processor node, such as an Evaluate MIDI Notes node, will then use those velocities for evaluation, for instance to determine attack times.
2. Easing can be done when evaluating MIDI Notes or possibly at a lower level by the user. High level nodes can provide parameters to evaluate notes at an ADSR envelop. I mentioned this three comments above.
3. This is up to the user, a low attack and release times should handle this. And the evaluation algorithm should put this in consideration.
4. The user can tweak the animation by looping over the notes and editing them procedurally. Animation Nodes is a procedural system, manual editing has to be done on the data itself using a DAW like you said. So I personally don't consider this a limitation.

[Clockmender]

Thank you for your answers, you are clearly on top of this and I can't think of any way I can add to this project at the moment.

In my DAW nodes, I used the Aud ADSR function to affect output sounds from a generator node, so I am familiar with ADSR, I just missed your comment...

So, for this, can I assume that if a note has this (written out):

Note E4, note-on = 10.5 , note-off = 10.7, velocity = 101

Gets adjusted so the ramp-up to velocity 101 and the ramp-down to 0 takes place over say 0.05 seconds (figures are only for illustration) and would be user set as a node input?

By editing procedurally, do I understand that you will be able, with a "Separation" node, to increase the start time and/or decrease the end time, for a note procedurally? Is it possible to have a node that can do this wherever it sees two consecutive note activations that have no time gap, or would the user have to do this on a case-by-case basis. A node that could automatically look for 0, or less than a set value, time spaced consecutive activations would be nice, if you can see a way to do this.

Example (written out):

Note E4, note-on = 12.4, note-off = 12.6, velocity = 89
Note E4, note-on = 12.6, note-off = 12.8, velocity = 93

These two events have no separation. So what is in the cache (however you plan to store that) gets changed procedurally, in the node's output, to say this:

Note E4, note-on = 12.4, note-off = 12.56, velocity = 89
Note E4, note-on = 12.6, note-off = 12.8, velocity = 93

Or

Note E4, note-on = 12.4, note-off = 12.58, velocity = 89
Note E4, note-on = 12.62, note-off = 12.8, velocity = 93

And a user input for the node (Min Separation), say 0.04 to be applied whenever the separation is less than this figure automatically.

I am just trying to throw in my experience from the problems I found when I did this before.

[Patochun]

I'm working tonight on this first node and the design document and you make the proposal at the end.

A precision that must be requested on the node in addition to the path of the MIDI file: the use of channels yes / no.

In MIDI they are two different notions but which finally address the same subject.

MIDI files use a 1 to n track relationship to 1 to n channels (all combinations are possible)

1 track to 1 to n channels => use of channels to separate the instruments.

n tracks to n channels => same as before

N tracks to 1 channel => situation created by some DAW (Reason for example). The channel numbers are at 1 and do not represent the different instruments.

It is therefore necessary to add a property which indicates if one wishes to separate the instruments with respect to the numbers of channels (the most general case) or compared to the tracks without taking into account the number of channel (the rarest case)

Ideas for the name of the property? It would be a boolean.

[Patochun]

In terms of storing notes it will be more so I think:

Note E4, note-on = 12.4, velocity = 89
Note E4, note-off = 12.6, velocity = 0

Do we take this opportunity to normalize the import of note_on type writing to 0 velocity to make a note_off and turn it into real note_off event instead?

[Patochun]

@clockmender your experience, like ours too, avoids us falling into traps and thinking enough about all important aspects.

[Clockmender]

The reason I thought of having the note-on and note-off values in the record is it _may_ be easier to see if the current time is contained within the event (I know @OmarEmaraDev doesn't like this term, but I can think of no other name for it just now).

So, if on frame 100 (current time = 4.166 seconds at 24 fps) it is _may_ be easier, or more efficient for the system to know that this event is currently on at velocity 67:

Note E4, note-on = 4.0, note-off = 4.8, velocity = 67

This could even be written as (times are absolute again):

E4, 4.0, 4.8, 67

The one thing we do know is that note events in the MIDI file will be ordered note-on, note-off, note-on, note-off, etc. always in pairs.

What do you think? I am just thinking about combining note-on and note-off into one record to make things easier rather than having to look at two records to see the status of an event. I don't know if this is practical, or not, but it might help if the user "scrubs" the timeline, or jumps to say frame 102...

With your separate records for note-on and note-off, you could write it like this:

E4, 4.0, 67
E4, 4.8, 0

That would work also I think...

You know the note is on because current time is > first record and < second.

[Patochun]

It may be to avoid presuming the future. Later we can also have other events than note_on and note_off.
For example polyphonic aftertouch for notes or pitchwheel, modulation controler, sustain pedal, expression pedal, pan, volume, etc. for track.

And Internally something like that :
Note E4, note-on = 12.4, velocity = 89
Note E4, aftertouch = 12.42, velocity = 92
Note E4, aftertouch = 12.45, velocity = 95
Note E4, aftertouch = 12.48, velocity = 104
Note E4, aftertouch = 12.52, velocity = 115
Note E4, aftertouch = 12.56, velocity = 103
Note E4, aftertouch = 12.58, velocity = 95
Note E4, note-off = 12.6, velocity = 0

[Clockmender]

Good point, so that could be:

E4, 12.4, 89
E4, 12.42, 92
E4, 12.45, 95
E4, 12.48, 104
E4, 12.52, 115
E4, 12.56, 103
E4, 12.58, 95
E4, 12.6, 0

Or would you want to know that there was an aftertouch event?

[Patochun]

Almost :) Think about other MIDI msg, track msg ?

[Clockmender]

Have a 3 character code for the event type?

NOT, E4, 12.4, 89
AFT, E4, 12.6, 92
TEM, 13.6, 88
SUS, 13.9, 127
SUS, 14.2, 0
MOD, 15.6, 45
MOD, 15.8, 0
NOT, E4, 16.0, 0
TEM, 16.0, 64

Am I getting close? NOTe AFTertouch TEMpo SUStain MODulate ? Tempo is in Beats Per Minute.

[Clockmender]

You could also include track number so it becomes:

NOT, 1, E4, 12.4, 89

So format is; Message type, Track Number, Time, Velocity?

That way we could handle the entire MIDI file in one operation and filter by track & type? Maybe have a subsequent "Track" node to separate out each track for routing to an animation loop?

[Patochun]

Almost almost
I think TEMPO is useless because all msg event are now in real time. So the change of tempo is already include.

I vote for your proposal and I add something usefull in first position the canal number.

1, NOT, E4, 12.4, 89
2, NOT, E4, 12.4, 89
1, SUS, 13.9, 127
2, AFT, E4, 13.95, 98
2, AFT, E4, 14.05, 91
1, SUS, 14.2, 0
1, MOD, 15.6, 45
1, MOD, 15.8, 0
2, NOT, E4, 16.0, 0
1, NOT, E4, 16.0, 0

What about the order of fields, channel and msg time and ?
What about fixed number of fields ? I don't know.

[Clockmender]

You're right Tempo is largely useless, other than to display on the animation... Or to set a lamp intensity, etc. (lamp gets brighter as Tempo increases).

I don't mind the order, it depends on what will be the foremost filter. if we filter by channel (Track) first them set this as the first field maybe?

I also used the Time Signature to show bars in an animation and "downbeat", so outputting this might be a good idea. Maybe add/compute a pulse every bar from the Time Sig... Thinking out loud again.

Fixed field number is not important as we are mainly going to filter by the first two only Channel= and Type=.

[Patochun]

Ok for tempo. It can be keeped. I did not see the possible utility.

[Clockmender]

We can also save the track information:

For each track so they can be auto-labelled directly from the MIDI file.

MIDI file order is Channel(Track) first, so maybe that is more intuitive for programers/users.

[Patochun]

I rewind just few messages and now I said, no, the channel can be in fact by having separate objects channels. A list of objects "channel/track". Each object may contains some informations regarding "channel/track", his title, and obviously all msg MIDI for notes, etc..

[Clockmender]

Do you mean a separate node for each channel, sorry I didn't understand the last message..

[Patochun]

Socket output of MIDI File node is one track choosen by user. The default
is 0 or 1 ?
If he want manage other track then he must use loop.
Other output is the number of channel ?

Or it's all tracks in structured list of tracks ?

Everything is possible. For my quick POC I use the second choice. Omar seem
to be less confusing. Waiting to discuss all together.

[Clockmender]

Socket output of MIDI File node is one track choosen by user. The default
is 0 or 1 ?

I believe the first music track with notes in it is channel/track 2, channel/track 1 is just the info, like initial tempo, time-sig, etc.

[Patochun]

Oh I think I numbered differently starting from 0

[Clockmender]

Example:

I think that is the pulse rate in channel/track 0... I may have to look that up.

Yes it is, here is my code to get the pulse value from a csv MIDI file:

        with open(path) as f1:
            for line in f1:
                in_l = [elt.strip() for elt in line.split(',')]
                if (in_l[2] == 'Header'):
                    # Get Pulse variable.
                    pulse = int(in_l[5])
                    dataD['Pulse'] = pulse

This gives you bpm:

bpm = float( round( (60000000 / int(tempo)), 3) )

This gave me frames from MIDI timings, you can adapt this to give you seconds:

frame = round(int(in_l[1]) * (60 * fps) / (bpm * pulse),2)

int(in_l[1]) was the timing from the MIDI line, fps was blend file frames per second.

Just another thought, have you thought about the fact that some DAWs / Keyboards use middle C as C3 and some as C4? I remember I had to account for that and make it an option on my nodes, the note is just a number, not a name in the MIDI file.

[Patochun]

Thx Clock, I do that before for my other project. And I just commited changes to manage the first node.

With the use of channel :

And the use of Track instead :

[OmarEmaraDev]

@Clockmender The Separation node you are describing can be done, even manually using a loop with reassignment. So I wouldn't worry about this case.

@Patochun Correct me if I am wrong, but isn't the channel a property of a MIDI Note? Why would we do the segmentation you are describing? The node should always return a list of MIDI Tracks, the MIDI notes will store the channel as an integer ID. A node will be provided to filter notes based on channel ID to drive different animations analogous to the different instruments that would have been played on those channels.

[Patochun]

Yes it is a property of a MIDI note, and it will be the case in the vast majority of cases. But some software exports by not using this property to separate. He leaves it to the same value and uses the tracks to differentiate the instruments. This is the case of the Reason Studio Propellerhead MIDI Export (now Reason Studio). I know this issue because I work with this software.
This is not to impose another segmentation but to propose it in case the MIDI file is done in this way.
We can propose to have this boolean hidden at the start and when choosing the MIDI file to detect the configuration N tracks for 1 channel to display this boolean?

The MIDI file will be of Type 1, with one MIDI track for each track in the Reason sequencer.
Since the Reason sequencer doesn’t use MIDI channels as such, all tracks will be set to MIDI channel 1.

http://docs.propellerheads.se/reason10/wwhelp/wwhimpl/common/html/wwhelp.htm#context=EngOpManProjectPro&file=SongFileHandling.19.7.html

[Clockmender]

As a fellow Reason user, I can confirm segregation by Track is needed, Reason does not place much store in MIDI Channel...

first-time.csv.zip

A human readable version for you to look at to see how it is set out, if that helps at all. This was exported from Reason.

[OmarEmaraDev]

If Reason exports them as tracks, then why should we add this option? Are you trying to simulate Reason's method of doing things?

[Clockmender]

@OmarEmaraDev I don't understand your last comment, it's simply that we need to be able to segregate by Tracks (as Patrick has done) as well as MIDI channels... I believe there is a maximum of 32 MIDI channels, whereas Tracks is unlimited. I don't think Reason is the only DAW that does this either...

[OmarEmaraDev]

@Clockmender Segregation by channel is fine and we should provide an intuitive way to do it. However, I don't think it should be an option of that node. We can provide specialized nodes for that, like a Filter node.

[Clockmender]

I don't mind how it is done, perhaps you could agree that with Patrick, I don't want to get involved in any coding myself.

[Patochun]

@OmarEmaraDev, I think like @Clockmender that Reason is not the only DAW to export to a coded MIDI file as well => Track versus channels.
It is just impossible to separate into various instruments if the channel information is not used. On the other hand, as I show in the attached documentation, it is indicated that the tracks are used instead in this case.
We can do walk away at first, although that is not a problem in terms of code (already coded in fact), but keep in mind that it will probably be claimed soon enough.

[Patochun]

I am the type of coder / tester and I like to materialize as and when. So here quickly the state of what I prepared:

Nothing definite of course. Does it start to look like what we discussed together ?

[Clockmender]

@Patochun I like that! so, 15 tracks, you can separate for instrument and note and it's Dave Brubeck's "Take 5" I also did that in Reason some time ago, loved playing the piano bit on my TMK88.

[OmarEmaraDev]

It is just impossible to separate into various instruments if the channel information is not used.

The channel will always be stored in MIDI Notes, so they can always be separated. What I fail to understand is why you would join all of them to channels? There is probably a very good reason why notes are in a certain track. Moreover, it seems counterintuitive to return "channels" from the node, where "channels" are not structures but merely just properties. The node output is even named "Tracks".

@Patochun This doesn't look very similar to what we discussed, but I don't want to rush and judge something that isn't definite. So continue for now until you get something that is more rigorous.

[Patochun]

Just enlight me on work axes. The goal is to converge to slowly (evently) but surely to the solution.

[Clockmender]

In less I am greatly mistaken (possible), the first number is the one you want to segregate by:

2, 0, Start_track
2, 0, Title_t, "Piano Ch1"
2, 0, Instrument_name_t, "Piano Ch1"
2, 0, MIDI_port, 2
2, 1000, Note_on_c, 0, 74, 80
2, 1000, Note_on_c, 0, 86, 80
2, 7680, Note_off_c, 0, 74, 0
2, 7680, Note_off_c, 0, 86, 0
2, 7680, Note_on_c, 0, 76, 79
2, 7680, Note_on_c, 0, 88, 79
2, 15360, Note_off_c, 0, 76, 0
2, 15360, Note_off_c, 0, 88, 0

2 in this case is the first piano track.

Another example taken from the internet, not produced with Reason:

1, 8711999, Tempo, 1333333
1, 8712000, Tempo, 1500000
1, 8742719, Tempo, 1500000
1, 8742720, Tempo, 1714285
1, 8773439, Tempo, 1714285
1, 8773440, Tempo, 2000000
1, 8788799, Tempo, 2000000
1, 8788800, Tempo, 1333333
1, 8789760, Time_signature, 6, 2, 24, 8
1, 8881920, End_track
2, 0, Start_track
2, 0, Title_t, "Manual_Ch1"
2, 0, Instrument_name_t, "Manual_Ch1"
2, 0, MIDI_port, 10
2, 0, Control_c, 0, 64, 0
2, 3904320, Note_on_c, 0, 65, 64
2, 3904320, Note_on_c, 0, 81, 64
2, 3907980, Note_off_c, 0, 81, 0
2, 3908160, Note_on_c, 0, 77, 64
2, 3911640, Note_off_c, 0, 65, 0
2, 3911820, Note_off_c, 0, 77, 0
2, 3912000, Note_on_c, 0, 76, 64
2, 3915660, Note_off_c, 0, 76, 0
2, 3915840, Note_on_c, 0, 77, 64
2, 3919500, Note_off_c, 0, 77, 0
2, 3919680, Note_on_c, 0, 72, 64
2, 3923520, Note_on_c, 0, 77, 64

Exactly the same for the Manual (organ), incidentally, there are 186 tempo changes in this file, something to note for your development work. I segregated by this first number in all my work, for what that is worth.

The figure after the note_on_c is always 0 in both files BTW. I think I will excuse myself from this conversation for now...

[OmarEmaraDev]

• I was expecting the MIDI File Parser node to return a list of MIDI Tracks. Instead, I see it returning a dictionary of an implicit "Channel" structure, which is conceptually not a structure.
• I was expecting the MIDI Track Info node to take a MIDI Track and return a list of MIDI Notes. Instead, it is taking a dictionary of the implicit channel structure and return an implicit list "Messages".
• I am not sure what the MIDI Note node is doing here.
• Then you are returning events, which I though was something we agreed we are not going to do.

Also, please note that we are making this system for users who probably never heard about MIDI before and have zero knowledge of what any of its details mean. So just because you or Alan understand what is going on, that doesn't mean your average user will.

[Patochun]

Hello all,
Ok @OmarEmaraDev, I have full time today and tomorrow, so we can build the fundations.
First we need to understand ourselves much better. I'm doing my best, I think, I'll try to have a less impaired vision from my MIDI experience. Write a little about the specifications, it's always a good working basis

I was expecting the MIDI File Parser node to return a list of MIDI Tracks. Instead, I see it returning a dictionary of an implicit "Channel" structure, which is conceptually not a structure.

Node : MIDI File Parser
Prop : none => Note 1
Input : Path of the file
Output : List of MIDI track => Question 1

I was expecting the MIDI Track Info node to take a MIDI Track and return a list of MIDI Notes. Instead, it is taking a dictionary of the implicit channel structure and return an implicit list "Messages".

Node : MIDI Track Info
Prop : Index track => Question 2
Input : List of MIDI track
Output : List of MIDI Notes => Question 3

I am not sure what the MIDI Note node is doing here.

Me either in the context described above. So forget it!

Then` you are returning events, which I though was something we agreed we are not going to do.

This is in my opinion relative to our misunderstanding of what is a note and a note played approached widely in question 3. Essentially a MIDI file is an event file. I think that we have blocked on a question of semantics. It does not pose any problem to clarify this point even more if there is need

Also, please note that we are making this system for users who probably never heard about MIDI before and have zero knowledge of what any of its details mean. So just because you or Alan understand what is going on, that doesn't mean your average user will.

Apart from the fact that I think that people who have never heard of MIDI will be reluctant to go for animations from that, I fully agree to make this experience the simplest and most efficient possible. Of course. I put my experience at the service of others, it is also the meaning of free software and I am attached to it.

I do not know how much time you have for this project. But I hope we can move forward strongly these next two days.

====================================================

Note 1 = Ok, if we have separate tracks but unordered channel numbers for MIDI messages note_on and note_off, discrimination is done by separate tracks. It's just that if afterwards in the process we aggregate by channel numbers, in this case we will mix notes for the piano with notes for the drums, etc ... in any case for MIDI files created by Reason, I have not looked for all DAWs, but it seems to be a bit common.

Question 1 = Under it forms the list of tracks. A list of track objects (which I thought I did), a list containing exactly what ? and in what form ? (python or AN speaking). Do not forget that I only have a little more than a year of experience for python (even if I am a computer scientist for 28 years) and not even a month for AN. Do not hesitate to be more precise and directive, it will help me.

Question 2 = Index track is an integer. it would be nice to display the name of the track corresponding to the chosen index to help the user, what do you think ?

Question 3 = Same question as for the list of MIDI tracks, what's in it, in what form?
To be more precise, what is a note in this context? I imagine that we are talking about a note played? which is very similar to a MIDI event noted in the MIDI file by the concept of MIDI message.

In the first structuring design given by you:

Midi File:

Midi Track 1
Midi Track 2
...
Midi Track:

Midi Note 1
Midi Note 2
...
Midi Note:

Note Channel
Note Number
Note Off Time
Note Off Velocity
Note On Time
Note On Velocity

What is called MIDI note 1 or 2 is not an "object" conceptually notes but an "object" note that describes the event of a note played. What then comes down to:

A "Note Number" note played on the "Note Channel" at the moment "Note On Time" with the velocity "Note On Velocity" and stopped playing at the moment "Note Off Time". (We can discuss keeping the velocity value for note_off which must always be 0 and strongly resembles a constant)

Do we agree, now, that what you call a Note is actually a note played, so an event in time?

[OmarEmaraDev]

• Note 1: Ok, I see what you mean better now. Some questions about Reason's export structure though. Does reason export 16 tracks for each of the 16 channels? Does that mean it always exports a MIDI Format 2? Will it ever export more than or less than 16 tracks?
• Question 1: Putting AN aside for now, it should return a flat python list that contains objects of type MIDI Track. From the screenshot you posted, it seems the output is, in fact, a dictionary with integer keys and "Channel" entries.
• Question 2: The node shouldn't have an index property. The node should take a MIDI Track as an input. The user can get this MIDI Track from the list of MIDI Tracks using the Get List Element node, which is the node that has the index property. The user can then identify the track name from there.
• Question 3: The node should return a flat python list that contains objects of type Midi Note. I will reiterate what I mean by a MIDI Note below.

What is a MIDI Note in this context?

A MIDI Note is a structures that encapsulates two consecutive MIDI events on the same channel and number, a Note On and a Note Off events. Conceptually, it is a description of a sound that was played.

I didn't know the Note Off velocity was always zero, in that case, we can omit it as you suggested. Your description of the MIDI Note seems to align with mine, so I think we are on agreement.

I guess I am fine with calling it an event in time as long as we make it clear that it is not a trigger event, but rather, a gate event.

[Clockmender]

I don’t know if this helps but any DAW will export the same number of tracks as there are tracks in the DAW’s sequencer, it could be 1 or any number. It is not fixed, the first one I posted has only 1, my largest has 23, but this should not be seen as a maximum.

[OmarEmaraDev]

@Clockmender I realize that, but my question is, will Reason export a MIDI file with any number of tracks? From what I understood from you is that it will put all events of the same channel into a track. But what if there were multiple tracks, will the 0-15 tracks be the channels of the first track, the 15-31 tracks be of the channels of the second track and so on?

[Clockmender]

It will export any number of tracks, each record (line in the midi file with note-on, etc.) starts with the track number, MIDI channel is irrelevant and only occurs in the track header. So all notes in one track will start with the track number, not the channel number. It that a better description?

Many DAW’s refer to this as midi-port not midi channel, if you look at the last bit of midi I posted you can see this: port was 10, track was 2.

Perversely many tracks could share the same midi-port as can be seen by the discrepancy in Patrick’s two lists.

[Clockmender]

Also perversely is the fact that all tempo changes are in track 1, the file header, not chronologically in the midi messages, as these are only chronological per track, and the tracks are then in index order.

[OmarEmaraDev]

My head hurt :grinning:, I will have to do some more research on the topic.

[Patochun]

Thank you for your comments and approval. I will work in this way. I have already started as usal :)

For Reason, in fact the MIDI file type is 1 and he use one track per instrument. But leave all MIDI channel to 0 everywhere.

[Patochun]

To be more precise, I don't have a MIDI with more than 15 tracks. So I don't know how Reason deal with more. I can made some tests.

[Patochun]

After coding that I don't think the

Node : MIDI Track Info
Prop : Index track => Question 2
Input : List of MIDI track
Output : List of MIDI Notes => Question 3

is usefull anymore. Because the use of AN _Get List Element node_ do the job, what do you think?

I also put time for note_on then note_off in this order

[Patochun]

Will we be able to go to the next step?

It is necessary at a given moment to recalculate the time in seconds to transform it into a frame via the current fps.

[OmarEmaraDev]

@Patochun We do not allow lists of lists in Animation Nodes. This only works right now because you are using Generic lists. So, no, I think the MIDI Track Info node is essential.

Not entirely sure what you mean, but the times should be fps agnostic.

[Patochun]

@OmarEmaraDev We agree that the output of "MIDI File Parser" is a flat python list that contains objects of type MIDI Track... but I did not understand that this object could not be a list itself. So what should it be?

About times will be fps agnostic, I do not have your view on the continuation and how one will treat all these data to make an animation. I would need to know more. It could avoid bad roads.

[OmarEmaraDev]

The MIDI Track object should be an instance of a MIDITrack class that contains a list of MIDI Notes. See the Sound class to see how this looks in practice. Similarly, a MIDI Note will be an instance of a MIDINote class that contains the info we have as discusses. See the SoundSequence class to see how this looks in practice.

We can't use lists because Animation Nodes rely mostly on typed lists, that is, all elements of the same list should have the same type, which is not the case here.

What kind of animation do you have in mind?

[Patochun]

ok, I understand better for the lists, I discover the constraint to have elements of the same type. And so every time that will not be the case the need to use an object, I'm right?

I will go to study more deeper the Sounds nodes.

For the first test animation, I was thinking of something very simple, a bar graph. Or each bar represents a note. A group of bars represents a track. And these bars evolve over the course of music.

[OmarEmaraDev]

And so every time that will not be the case the need to use an object, I'm right?

Not sure I understand what you mean. Can you elaborate?

[Patochun]

In the AN "World" I cannot use a list containt elements of differents types as you said. The other sentence is a consequence of that. nothing more.

[Clockmender]

As an animator, I would just like to see a node that outputs via float socket, a list of floats, one for each note played, at every frame, with a second socket telling me what note those floats are, i.e first is e3, second is g2s etc. A major node lets me select a midi file and outputs a set of tracks for the first one I mentioned to access these via a standard get element node. How you get there; je m'en fous!

EDIT:

I have some routines that translate MIDI note numbers in to actual note names, for both cases of where middle C is and also some routines to translate notes into frequencies if these are any use to you, just let me know.

[Patochun]

I commit this :

[OmarEmaraDev]

@Patochun Looks nice. Now try to create sockets for those data. Use the existing socket classes as a reference.

[Patochun]

Yes, it is the next step. I'm in.
A question : from output of MIDI File Parser, it is a list of object And in the input of MIDI Track info is only on object track. The question is, we cannot link these two nodes without get lis element ? I'm right ?

[Patochun]

Ok I see with each socket come list socket

[OmarEmaraDev]

Yes, we can't link them directly.

[Patochun]

With sockets :

I confess that I did not really understand how sockets worked. I took a model and adapted. But not really understood.

@OmarEmaraDev maybe we should already do a code review, so as not to diverge from the standards and correct as soon as possible what should be.

I have a bug on the 2 MIDI Track Info and MIDI Note Info nodes. It does not work when there is nothing connected in input. I can not understand why for now.

How do you see next?

[OmarEmaraDev]

@Patochun Nice progress! I think we should postpone code review for now until we get some practical examples first.

Regarding the MIDI Note Info node, why are the Time On and the Time Off outputs of type Integer, should't they be floats? Also, since we know the maximum value of the Velocity output, can we normalize it and make it a float?

Next, I want you to create two nodes:

• Filter MIDI Notes: This node primarily takes a list of MIDI Notes and outputs only a subset of the input MIDI Notes. The node only outputs the MIDI Notes that are on a certain channel or/and have a certain note number. The channel and/or the note number should be taken as integer inputs. The node have two boolean options, Filter By Channel and Filter By Number, the inputs are created dynamically based on those options (i.e. The Channel input should only be available if Filter By Channel is enabled, and the same goes for the note number).
• Evaluate MIDI Notes: The node takes a list of MIDI Notes and a float input representing the time in seconds. The node outputs a float that is either 1 or 0. The output float is 1 if a note is currently playing (Regardless of the channel and number) and 0 otherwise. For now, we will skip validation, so assume the input is always valid.

[Patochun]

Yes, Float. Well seen

Ok for the next nodes. I'm on my way.

[Patochun]

Regarding the MIDI Note Info node, why are the Time On and the Time Off outputs of type Integer, should't they be floats? Also, since we know the maximum value of the Velocity output, can we normalize it and make it a float?

You mean Velocity between 0.0 to 1.0 ? (I assume is yes)

[OmarEmaraDev]

@Patochun Yes.

[Patochun]

Evaluate MIDI Notes: The node takes a list of MIDI Notes and a float input representing the time in seconds. The node outputs a float that is either 1 or 0. The output float is 1 if a note is currently playing (Regardless of the channel and number) and 0 otherwise. For now, we will skip validation, so assume the input is always valid.

The output float = 1 if a note from a set of notes is currently playing and else 0.
I don't see exactly why this node?
Why not a boolean instead a float?

[OmarEmaraDev]

0 and 1 are just a simplification for now. Eventually, we will incorporate ADSR envelops in the node, allowing the output to be a float ranging between 0 and 1.

[Patochun]

Ok I understand this part. But I don't undestand exactly the utility of this node !

[Patochun]

One new node :

I have a bug, when I click on the boolean to have the new input input value, I experiment this :
TypeError: cannot unpack non-iterable NoneType object

import bpy
from bpy.props import *
from ... sockets.info import isBase, toBaseDataType
from ... events import propertyChanged
from ... base_types import AnimationNode, ListTypeSelectorSocket

class MidiNoteFilterNode(bpy.types.Node, AnimationNode):
    bl_idname = "an_MidiNoteFilterNode"
    bl_label = "MIDI Note Filter"
    bl_width_default = 180

    usefilterByChannel: BoolProperty(name = "Channel Filter", default = False,
        update = AnimationNode.refresh)
    usefilterByNote: BoolProperty(name = "Note Filter", default = False,
        update = AnimationNode.refresh)

    def create(self):
        self.newInput("MIDINoteList", "Notes", "notesIn")
        self.newOutput("MIDINoteList", "Notes", "notesOut")
        if self.usefilterByChannel:
            self.newInput("Integer", "Channel", "channel")
        if self.usefilterByNote:
            self.newInput("Integer", "Note", "number")

    def draw(self, layout):
        layout.prop(self, "usefilterByChannel")
        layout.prop(self, "usefilterByNote")

    def getExecutionFunctionName(self):
        if self.usefilterByChannel:
            if self.usefilterByNote:
                return "execute_FilterByChannelAndNote"
            else:
                return "execute_FilterByChannel"
        else:
            if self.usefilterByNote:
                return "execute_FilterByNote"
            else:
                return "execute_NoFilter"

    def execute_FilterByChannelAndNote(self, notesIn, channel, number):
        notesTmp = notesIn
        notesTmp = [note for note in notesTmp if note.channel == channel]
        notesTmp = [note for note in notesTmp if note.number == number]
        return notesTmp

    def execute_FilterByChannel(self, notesIn, channel):
        notesTmp = [note for note in notesIn if note.channel == channel]
        return notesTmp

    def execute_FilterByNote(self, notesIn, number):
        # print([note for note in notesIn if note.note == number])
        notesTmp = [note for note in notesIn if note.number == number]
        return notesTmp

    def execute_NoFilter(self, notesIn):
        return notesIn`

[OmarEmaraDev]

Where/When exactly do you get that error?

The Evaluate MIDI Notes node will be the most commonly used node here. If you want to drive a certain object by the notes of a certain channel and number, you first filter the track notes you have by that channel and note number then evaluate the output of the notes to get a float that can be used to control the target object.

[Patochun]

When note choosen doesn't exist in list of notes and same for channel.

[OmarEmaraDev]

Not sure, maybe use different variables in execute_FilterByChannelAndNote.

[Patochun]

Here, actual situation :

I work on few minor bugs.

[Clockmender]

Before you get too far down the track @Patochun could you consider the need to translate midi note numbers to names, I use this look up:

noteList = [
    'c0','cs0','d0','ds0','e0','f0','fs0','g0','gs0','a0','as0','b0',
    'c1','cs1','d1','ds1','e1','f1','fs1','g1','gs1','a1','as1','b1',
    'c2','cs2','d2','ds2','e2','f2','fs2','g2','gs2','a2','as2','b2',
    'c3','cs3','d3','ds3','e3','f3','fs3','g3','gs3','a3','as3','b3',
    'c4','cs4','d4','ds4','e4','f4','fs4','g4','gs4','a4','as4','b4',
    'c5','cs5','d5','ds5','e5','f5','fs5','g5','gs5','a5','as5','b5',
    'c6','cs6','d6','ds6','e6','f6','fs6','g6','gs6','a6','as6','b6',
    'c7','cs7','d7','ds7','e7','f7','fs7','g7','gs7','a7','as7','b7',
    'c8','cs8','d8','ds8','e8','f8','fs8','g8','gs8','a8','as8','b8',
    'c9','cs9','d9','ds9','e9','f9','fs9','g9']

The note number is then the index for a search, shifted 12 for different middle C .

Which enables me to cater for middle C being either C3, or C4 - I found that putting # to mean a sharp note caused all sorts of problems, also there was no need to cater for flats since E-flat is also D-sharp.

Good progress by the way Sir!

[OmarEmaraDev]

I think it is time we do a practical example. Can you try to make a piano or something with the current node tree? A simple loop for every note number should do the job.

[Clockmender]

Here is a sample keyboard if it is of any use:

88keys.dae.zip

[Patochun]

@Clockmender have you a MIDI file because we can't use a DAE file. Can you try to provide us a MIDI file relatively simple with just few track and not so long.
@OmarEmaraDev go for the first practical example. I must look deeper on how loop work

[Patochun]

@Clockmender I don't know exactly place where you want have a converstion from number of note to name ? maybe MIDI File Note node ?

[Clockmender]

have you a MIDI file because we can't use a DAE file

It's a 88 key keyboard mesh set, import it into Blender... It looks like this, ignore the empties, they are not included:

One object for each note. Do you need a MIDI file?

[Patochun]

I'm not aware with loop design. I need a booster from expert
An example? with few explanations

[OmarEmaraDev]

There are a few examples in the documentation on loops. Have you checked that page?

[Patochun]

Yes, but examples relies to math operations. I search for object, instances and object output
I go to google it

[Patochun]

I start to understand the loop concept into AN:

I show you later the result... It's seem to work !!!

[Clockmender]

I think you will have some issues with your node tree, there should not be anything connected to the Loop, other than through the Invoke Sub-programme, here is an example:

You can see I am feeding all the MIDI data into the Invoke and letting the Loop drive the animated objects. This is 2.97, but the same rules still apply. Basically you feed the Loop with parameters and, normally only 1 iterator, but that can change. The iterator is the objects you want to animate and the parameters are what values are used to animate.

This tree has two MIDI channels feeding the same Loop, there's no need for separate Loops if they do the same thing.

Unless @OmarEmaraDev wants to make some other observations...

[Patochun]

Thank you, I will look at this a little later. I started a video calculation for the first animation.

[OmarEmaraDev]

@Patochun While the node tree probably work, it is not very optimal and will probably run very slow. This is due to the fact that you do a lot of things inside the loop, try to pass anything that doesn't change as a parameter of the loop.

Otherwise, great job! Looking forward to see the video.

[Patochun]

So here is a first test with no pretensions other than to validate what was done

https://docouatzat.files.wordpress.com/2019/10/an-midi-01.mp4

And the nodes set used :

I understand slowly but strongly :)

[OmarEmaraDev]

Very nice @Patochun !

I am starting to think that the Filter node should be removed and its functionality should be added to the Evaluate node itself. This would allow us to vectorize the Evaluate node and thus make the node tree much simpler.

So here is what I want you to try. Remove the Filter node and transfer its functionality to the Evaluate node, meaning the evaluate node should do the filtering. Then, convert the Note Number input and the output of the Evaluate node into vectorized sockets.

I am not sure if you are familiar with vectorized sockets, check the documentation on the topic. Then check any of the vectorized nodes to identify how the code should be written.

[Patochun]

To be sure of this proposal and how it works:

If 1 input note => 1 value for the note played 0 or 1 ()
If a list of notes in input => 1 list of values for the notes played 0 or 1 (*)

(*) If a single entry note, then no filtering possible, remove all that is used for filtering

(**) If an input note list, then the output value list represents only the filtered notes (channel / note) if filtering is active.

Is there also the list of filtered notes as output?

[OmarEmaraDev]

The input will remain a list of MIDI Notes, always. The Note Number input will be the input that will be vectorized, where we will be evaluating multiple note numbers at the same time.

• Evaluate MIDI Notes node:
  
  
  
  • Inputs:
  
  
  • MIDI Notes. (A list of MIDI Notes)
  
  
  • Time.
  
  
  • Note Number(s). (A vectorized socket, can either be an integer or a list of integers)
  
  
  • Channel Number.
  
  
  • Outputs:
  
  
  • Value(s). (A float when the note number is a single integer and a list of floats when the note numbers are a list)
  
  
  • Functions:
  
  
  • If the note number is an integer:
    
    The MIDI Notes are first filtered based on the note and channel numbers. Then the list is evaluated, just like before.
  
  
  • If the note numbers is a list of integers:
    
    For each of the note numbers in the list, do the same as above. Returning a list of values.
  
  

[Patochun]

it's clear now

[Patochun]

Not yet finished but on the way. I must learn a little about comprehension lists nested

[Patochun]

Functions:
If the note numbers is a list of integers:
For each of the note numbers in the list, do the same as above. Returning a list of values.

@OmarEmaraDev I am interested on a specific point:
if a note in the list of integer input simply does not exist, should it be in the output list with 0.0 or not?

Actually in the other case, if the single note in input doesn't exist in list of notes, then the output is 0.0 and not None.

[Patochun]

In progress

I think I'll have to flatten the result by note number

[OmarEmaraDev]

@Patochun If you mean that the input number is not between 0 and 127 and thus does not exist, then yes, I think it should be included in the output with a value of zero.

It is unclear to me why the output list is like this. Maybe you misunderstood my point. In this case, the output should be a list of 3 floats, the first float is the output when the note number is 55, the second float is the output when the note number is 58, and the third float is the output when the note number is 60.

[Patochun]

@OmarEmaraDev do not be mistaken, it's still under construction.
I understood what is asked. The picture show a debug step.

No, I am asking what is done for a note number provided in the input list that will not exist in the list of runway notes.

Since you are filtering by note number, a value between 0 and 127 may simply not exist in the track's list of notes. In this case, but I think I have the answer, you have to put 0.0 for the note in the output list and not to put any value at all.

It is believed that the value 0.0 potentially indicates two things:
1 - The note exists but is not played at the given moment
2 - The note does not exist

[OmarEmaraDev]

The existence and non existence of a certain note in a certain track seems to be a useless information in the context of note evaluation. So I think returning zero to indicate either states is the best thing to do, otherwise, we lose correspondence between inputs and outputs.

[Patochun]

It was also my understanding

[Patochun]

Et voilà

[OmarEmaraDev]

Very nice Patrick!

I think the next thing to do is to implement ADSR envelops, starting with Attack and Release times.

The Evaluate node should have 4 extra inputs:

• Attack Time.
• Release Time.
• Attack Interpolation.
• Release Interpolation.

The values during the attack time will be evaluated at the attack interpolation and the same goes for release. This allow us to have non linear attacks and releases.

Also, if you don't mind, can you recreate the practical example using the new node without any loops. You can use the output of the Integer Range node to get the note numbers. I would like to see the node tree.

[Patochun]

The video here :

https://docouatzat.files.wordpress.com/2019/10/an-midi-02.mp4

[Patochun]

@OmarEmaraDev we need to think how to use the velocity data.

[OmarEmaraDev]

Yes, a multiple of the velocity can be used as an attack time, so we should implement attack and release time now.

[Patochun]

The Evaluate node should have 4 extra inputs:

Attack Time.
Release Time.
Attack Interpolation.
Release Interpolation.
The values during the attack time will be evaluated at the attack interpolation and the same goes for release. This allow us to have non linear attacks and releases.

@OmarEmaraDev After few days off, I come back into the project.

Can you be more specific about these new entries.
I assume they are floats and what about their default values ?

And be more precise about math behind attack, interpolate...

Velocity mean the strength you apply when the note is playing, nothing to do with attack time.

[OmarEmaraDev]

Attack and Release times are Float inputs. Lets not worry about the default values yet. The Attack and Release interpolations are, well, Interpolation inputs.

Lets say a note starts at time 5 and ends at time 10. If the attack time is zero, the output will be 1 as soon as the input time becomes in the range of the duration of the note, which is the behavior we have now. If the attack time is 2, the output will gradually increase from zero until it reaches its maximum amount (one) after 2 seconds from the start of the note.

Check the Wikipedia article on the topic:
https://en.wikipedia.org/wiki/Envelope_(music)

Hardware ADSR envelops are mostly linear. We want to grant the user more control, so we allow the use of interpolations to define the rate of changes. Practically, this is as easy as evaluating the linear value at the input interpolation.

The same goes for release, except release happen after the note is finished, so it will be more complicated to implement.

[Patochun]

Hello, I started work for the new version of the node. @OmarEmaraDev could you give me some explanations or a node model that integrates the management of interpolations?
There is something to code or is it already taken into account in AN

[OmarEmaraDev]

An interpolation is just a function that takes a single float between zero and one and returns a float. So, given the input interpolation testInterpolation, to evaluate the interpolation at 0.5, just do testInterpolation(0.5). Check the evaluate interpolation node for example.

[Patochun]

Ok for interpolation. I think the note is fully on at normal time and the attack time take place before. What do you mean ?

[OmarEmaraDev]

No, the attack time should be after the note starts (So from 5 to 7 when the attack is 2). The release should be after, just like you did. The note velocity has nothing to do with the evaluate node, the user may choose to use the velocity as an attack time, which the user can get using the Note Info Node. And the "full" state is always one.

[Patochun]

First look

when I could I would do a video
I still have to complete and clean the code and improve it can be a little bit

[OmarEmaraDev]

Very nice Patrick! Eventually, the attack and release times should be virtual lists. Are you familiar with virtual lists? If you aren't, I will elaborate on them later not to complicate things for now.

[Patochun]

I'm not familiar with virtual lists. You know I discover many things with this project. The only thing I have is the skill to code, I hope. But what do you mean, Attack and release time dependant to note number ?

I come back to the story of velocity. Velocity does not indicate anything in terms of speed of attack for a note. The velocity tells us only that with force we press the key of the piano, we blow in the trumpet, finally you see what I'm talking about.

[OmarEmaraDev]

Essentially, we want to be able to assign a different attack and release times to different notes. To do so, we vectorize the attack and release inputs, allowing the user to input float lists as well. Virtual lists are Animation Nodes data structures that makes lists more dynamic. In particular, if the provided list is too short, it is resized, and if it is empty, it is filled with a default values. So for instance, the user may provide a list of two attacks only, in which case, the notes will alternate between them.

I see your point about the velocity. That makes sense. But it is good to provide the user with such flexibility nevertheless.

[Patochun]

I'm going to work on it, but before I'm going to create another video to see already what we can do in the current state.

[Patochun]

New video published here : url

[OmarEmaraDev]

This look awesome! Great job Patrick. :+1:

[Patochun]

Big picture of AN related to video. 3 groups of nodes, one for each track.

[Patochun]

I commited all change in my branch :
https://github.com/Patochun/animation_nodes/tree/MIDI-node

[Patochun]

I commited all change in my branch :
https://github.com/Patochun/animation_nodes/tree/MIDI-node

[OmarEmaraDev]

I wounder if we should remove the Filter By Channel and Filter By Note options. I think they should always be enabled, otherwise the node doesn't really make sense. The evaluated MIDI notes should be in a valid state, that is, they shouldn't overlap. So not filtering them by channel and note number would potentially make them overlap. What do you think?

[Patochun]

Effectively not filtering them is equivalent to accepting overlaps, ie notes played on several channels simultaneously. it does not really make sense in terms of music. The filter option should be removed per channel and made mandatory.

But I think we should let the filter by note option. The node would answer the question does something happen at some point? without wishing to exploit the note played. For example, turning on / off a lamp on all the objects / notes of this channel. But we can do it also by summing the output values, so in fact no, we can also remove it.

[OmarEmaraDev]

Ok, so I think we should remove both options and make them mandatory. I think we can start code review soon, but after we handle couple of points.

• The MIDI Note Info node should be vectorized. It should be able to accept a list of notes and outputs lists.
• What happens when a note starts playing while the release of another note is still in effect? Do we immediately drop to the level of the playing note? Or do we take the maximum of both levels? What do you think? May be @Clockmender might have an opinion on this?
• How do we handle bad data? A user might generate a list of notes that overlap, do we just invalidate the data? Do we try to fix it? Should we always check the integrity of the data? Do we expect the notes to be sorted? What do you think?

[Patochun]

The MIDI Note Info node should be vectorized. It should be able to accept a list of notes and outputs lists.

This node brings only velocity as information.
Velocity is very important to give even more expression to the animations.
But keeping this node will only complicate things to the user
My idea is to use this value as coefficent in the node "Evaluate MIDI note" (with the choice to use it or not). Coefficient means here that a velocity of 127 corresponds to 1.0 and that this figure is to mitigate following the attack and the release of course.
what do you think?

What happens when a note starts playing while the release of another note is still in effect? Do we immediately drop to the level of the playing note? Or do we take the maximum of both levels? What do you think? May be @Clockmender might have an opinion on this?

currently in this case the attack phase is evaluated first and never the release phase (elif) so the note is "replayed" respecting time and attack interpolation.

@OmarEmaraDev you say few weeks ago :

Not entirely sure what you mean, but the times should be fps agnostic.

nevertheless we can only animate actions that last at least the time of an image. At 25 fps, an animation can take less than 1/25 second.

We will see something between 2 successive notes perfectly quantified with the attack phase. I think we see it in the last video.

How do we handle bad data? A user might generate a list of notes that overlap, do we just invalidate the data? Do we try to fix it? Should we always check the integrity of the data? Do we expect the notes to be sorted? What do you think?

The user can mix cabbage and carrots, or multiply the same carrots several times ... we will never be in his head. I think we have to let it go, without trying to check integrity / reprocess / correct / filter. The maximum penalty will be an unsatisfactory animation for the user.

[OmarEmaraDev]

The MIDI Note Info node doesn't just return the attack velocity, it also returns the attack and release times, the channel number, and the note number. The Evaluate MIDI Notes node is not the only way to create animations from MIDI. Experienced users may want to have lower level control over MIDI notes, which we grant using the MIDI Note Info node. For instance, one might use the attack time as the start input of an Animate Vector node, with the duration equal to the duration of the note, creating a parametric animation of some kind. Similarly, the channel and note numbers can be used as means of spatially positioning object during animations. So, in my opinion, the node is essential and shouldn't be removed. Moreover, I think it is acceptable to put the responsibility of implementing the velocity coefficient on the user, but I am not strongly against adding an option to the evaluate node. We will do some workflow testing later.

nevertheless we can only animate actions that last at least the time of an image. At 25 fps, an animation can take less than 1/25 second.

Not really, frames in Blender are floats not integers, they becomes fractional in some cases.

Regarding input validation, alright. I think we should let it go, at least for now.

[Patochun]

For instance, one might use the attack time as the start input of an Animate Vector node, with the duration equal to the duration of the note, creating a parametric animation of some kind. Similarly, the channel and note numbers can be used as means of spatially positioning object during animations

Ok, I understand the low level

Moreover, I think it is acceptable to put the responsibility of implementing the velocity coefficient on the user, but I am not strongly against adding an option to the evaluate node. We will do some workflow testing later.

Ok, I will also do a video proof of concept on the subject to validate the workflow

Not really, frames in Blender are floats not integers, they becomes fractional in some cases.

It is a mathematical vision (your world I think :) ) that if it is valid in the internal management can not really be translated that if we have time to display an event in the time allotted by the fps.

[OmarEmaraDev]

Actually, we should change the Time input to be a Frame input. There will also be a Scene input, the fps of the scene will be used to compute the absolute time in seconds. Take a look at the Sound Spectrum node.

[Clockmender]

Hi chaps, I am on holiday in Chile just now and on my way to Antartica after that. I will be home December 1. However, on the point you raised Omar I worked on the assumption that I interrupted a note playing just before it was due to play again. So if e4 was due to terminate at 12.5 seconds, but the midi file called it to start at 12.48 I would stop it at say 12.47. The system must of course be able to play multiple notes concurrently even if they have different end times.

Hope all this makes sense, I will look from time to time here as and when I get internet connection!

[OmarEmaraDev]

@Clockmender I see. Thanks for the feedback! I hope you have a great time on your holiday! :smiley:

[Patochun]

Vectorization done

[OmarEmaraDev]

@Patochun Great! I think we should start code review now, so if you don't have anything else left to do, clean the code and submit a pull request.

It should be noted that we will also write our own minimal MIDI Parser, will you do that? If you don't want to do it, I think we can do it.

[Patochun]

I made the pull for first code review

I have to think about the minimal MIDI Parser. I can certainly do it but I think I would not be the most effective.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.