music/demos/transcription.html

<!-- Copyright 2018 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================-->
<html>
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1, user-scalable=yes">
  <link rel="stylesheet" href="./style.css">
  <title>Transcription</title>
  <style>
    input[type="file"] {
      width: 0;
      height: 0;
      opacity: 0;
      cursor: pointer;
      display: none;
    }
  </style>
</head>
<body>
  <h1>Onsets and Frames</h1>
  <p><code><a href="https://g.co/magenta/onsets-frames">Onsets and Frames</a></code>
    is a deep neural network that transcribes audio of piano performances to
    <code>NoteSequences</code>/<code>MIDI</code>.
  </p>

  <h2>From a local audio file</h2>
  <p>You can use your own piano music file (i.e. actual audio, not midi) for transcription:</p>
  <section>
    <label class="button">
      Load audio file
      <input type="file" id="fileInput">
    </label>
    <br>
    <p><audio id="filePlayer" controls hidden></audio></p>
    <p><b>Actual Transcription:</b> <code id="file-results" class="file"></code></p>
    <p><b>It Took:</b> <code id="file-time" class="file"></code></p>
    <p><b>Total Leaked Memory:</b> <code id="file-leaked-memory"></code></b></p>
  </section>

  <h2>From a microphone recording</h2>
  <p>You can record piano audio from a microphone and transcribe it. If you're recording
    something other than piano (like your voice), this will be transcribed but it
    will probably be fairly noisy and incorrect.
  </p>
  <section>
    <button id="recordBtn">Record</button>
    <br>
    <p><audio id="recorderPlayer" controls hidden></audio></p>
    <p><b>Actual Transcription:</b> <code id="recorder-results" class="recorder"></code></p>
    <p><b>It Took:</b> <code id="recorder-time" class="recorder"></code></p>
    <p><b>Total Leaked Memory:</b> <code id="recorder-leaked-memory"></code></b></p>
  </section>

  <h2>From a test audio (250 frames / 8 seconds)</h2>
  <p>We verify the model can transcribe a short sequence of piano audio, first computing its
    <a href="https://librosa.github.io/librosa/generated/librosa.feature.melspectrogram.html" target="_blank"></a>mel spectogram</a>.
  </p>
  <section>
    <p><b>Original Audio</b><br><audio controls><source type="audio/wav" src="https://storage.googleapis.com/magentadata/js/checkpoints/transcription/onsets_frames_htk0/MAPS_MUS-mz_331_3_ENSTDkCl-250frames.wav"></audio></p>
    <p><b>Expected Transcription:</b> <code id="expected-audio-ns"></code></p>

    <button id="audioBtn">Start Audio Transcription</button>
    <p><b>Actual Transcription:</b> <code id="audio-results" class="audio"></code></p>
    <p><b>Match:</b> <code id="audio-match" class="audio"></code></p>
    <p><b>It Took:</b> <code id="audio-time" class="audio"></code></p>
    <p><b>Total Leaked Memory:</b> <code id="audio-leaked-memory"></code></b></p>
  </section>


  <h2>From <a href="https://librosa.github.io/librosa/generated/librosa.feature.melspectrogram.html" target="_blank">Mel Spectrogram</a> (250 frames / 8 seconds)</h2>
  <p>Below we verify that the model properly transcribes a mel spectrogram to match the output
    of the Python TensorFlow implementation. We found it is necessary to process the convolution
    in batches for longer inputs to avoid a GPU timeout enforced in Chrome, and we verify that
    the transcription works properly with different batch sizes that exercise various edge cases.
  </p>
  <section>
    <p><b>Original Audio</b><br><audio controls><source type="audio/wav" src="https://storage.googleapis.com/magentadata/js/checkpoints/transcription/onsets_frames_htk0/MAPS_MUS-mz_331_3_ENSTDkCl-250frames.wav"></audio></p>
    <p><b>Expected Transcription:</b> <code id="expected-ns"></code></p>
    <button id="melBtn">Start Batched Transcription</button>
  </section>

  <h3>Chunk Length 250 / Batch Size 1</h3>
  <section>
    <p><b>Actual Transcription:</b> <code id="250-results" class="batch"></code></p>
    <p><b>Match:</b> <code id="250-match" class="batch"></code></p>
    <p><b>It Took:</b> <code id="250-time" class="batch"></code></p>
  </section>

  <h3>Chunk Length 150 / Batch Size 2</h3>
  <section>
    <p><b>Actual Transcription:</b> <code id="150-results" class="batch"></code></p>
    <p><b>Match:</b> <code id="150-match" class="batch"></code></p>
    <p><b>It Took:</b> <code id="150-time" class="batch"></code></p>
  </section>

  <h3>Chunk Length 80 / Batch Size 4</h3>
  <section>
    <p><b>Actual Transcription:</b> <code id="80-results" class="batch"></code></p>
    <p><b>Match:</b> <code id="80-match" class="batch"></code></p>
    <p><b>It Took:</b> <code id="80-time" class="batch"></code></p>
  </section>

  <h3>Chunk Length 62 / Batch Size 4</h3>
  <section>
    <p><b>Actual Transcription:</b> <code id="62-results" class="batch"></code></p>
    <p><b>Match:</b> <code id="62-match" class="batch"></code></p>
    <p><b>It Took:</b> <code id="62-time" class="batch"></code></p>
  </section>

  <h3>Chunk Length 50 / Batch Size 5</h3>
  <section>
    <p><b>Actual Transcription:</b> <code id="50-results" class="batch"></code></p>
    <p><b>Match:</b> <code id="50-match" class="batch"></code></p>
    <p><b>It Took:</b> <code id="50-time" class="batch"></code></p>
  </section>

  <section>
    <p><b>Total Leaked Memory:</b> <code id="leaked-memory"></code></b></p>
  </section>

  <script src="transcription_bundle.js"></script>
</body>
</html>