NoteZcore · Note Extraction from Piano Recordings

Yamaha CF3S, my favorite instrument

NoteZcore was intended to decode the audio recording of the American-Fantasy played by Wibi Soerjadi in the Carnegy Hall into sheet music. Currently, the score is not publically available and NoteZcore is not able to backtrace the notes.

This is another project in the category "Challenge Of Reverse Engineering". Though the results at the present are not entirely satisfying, the program works fine for pieces like the Goldberg Variations and is quite in a state for comfortable extentions, if one understands the physics behind the nascency of piano sound, and is willing to encorporate non-linear effects in the algorithm.

The strategy and results as exposed below were also presented at the "Studentische Vortragsreihe 2003" at the Mathematics faculty of the TU-Darmstadt.

SourceCode (MATLAB 6)	notezcore.zip	12 kB
MIDI Compilation of extracted sound	notezcore_midi.zip	3 kB

Notezcore was my first research project of considerable size. Over the course of the following years, I realized that ...

Even if you do learn to speak correct English,
whom are you going to speak it to?
Clarence Darrow

Let's follow the programs strategy at a short example, an excerpt of a Goldberg Variation by Johann Sebastian Bach:

My program divides into several modules. It beginns with the detection of time points (events). In between two neigboured events, the sound doesn't "change" - notes are neither hit nor released in the meantime. That is important for the spectral analysis and is done by filtering high frequencies for characteristic sound formings.

Next comes the measure graduation. Coping with (small) speed changes the program tries to adapt the pitch according to accelerations. In (very) fast pieces the resolution for event extraction has to be high to fulfill the measure partitioning correctly.

After that follows a global spectrum estimation, frequencies from 27.5 Hz - 17000 Hz are tested.

Another filter deletes the pure obertones. A first sight at potential notes is possible.

All notes which are indicated above a certain threshold are used to approximate the given spectrum in the 1-norm (at present the best way I found). For approximation I use the sound of notes I recorded from a grand piano.

The notes are exported in the GUIDO format and displayed by GUIDO NoteViewer. You can download the midi file to listen to the result here here. The runtime of the program on a normal computer is about the length of the piece itself. For people who know about the Goldberg Variations the compilation of generated midi files might be of interest (see top of page).

Please note, that you have to provide library tones for execution. If you mail me, I can send you my recordings.

Parameter settings

It follows the initiation file. Here one sees the parameter-set required for a piece.

global N S X eX E P M G cost smh

N=struct(...
'sd','C:\eigene~1\matlab\notes\',... %directory
'sf',44100,...      %resolution
'sk','G',...        %key
'sm',[4 4;1 4;1 4;1 8],... %measure;dominant;pretone;smallest
'sn','b1',...       %file name
'st',[0 Inf],...    %time sequence [in seconds]
'sp',0,...          %play sound: 0|1
'sv',[1 1 1],...    %view windows of analysis: 0|1
'le','pf1',...      %scale energy []|pf1|smo
'lf',3,...          %load filter 1attck|2smooth
'll',1,...          %load lib 0|1
'ln','p1',...       %libname
'lo',6,...          %obertone
'lw',3700,...       %width
'ly',100,...        %layer begin
'mm',[1],...        % multiples
'mp',1,...          %initial pitch
'mu',[],...         %ticks of smallest unit
'ab',20:75,...      %key band (must be connected)
'af','id',...       %frequencies id|lo|hi|vector [Hz]
'ag',[1 12],...     %frequencies min:(1):max+(2)
'al',1,...          %logarithmic shift
'an',0,...          %norm: 0 filter|1|2(|Inf not yet!)
'ao',40,...         %filter with tone
'ap',[],...         %filter
'as',05E-2,...      %noteZ threshold
'at',10E-2,...      %event threshold
'aw',2000); tic;    %tic lengthjnbv1_1

jnload;
jnevent;
jnmeasur;
jnlysis;
jnguido;

fprintf('\ntime elaps: %3.1f sec\n',toc);

And for almost 10 years we have pitted
this unbelievable and unprecedented capability against:
A bunch of dudes in bed sheets and flip-flops
Daniel L. Davis