web-audio-api 🚀 - (AudioBufferSourceNodePlaybackRate): AudioBufferSourceNode.playbackRate not strictly defined

Original comment by Ehsan Akhgari [:ehsan] on W3C Bugzilla. Thu, 01 Aug 2013 17:24:34 GMT

I think it probably makes sense to specify a 0 playbackRate to produce silence, and for negative values to play the buffer backwards, that is, from duration to offset (or from loopEnd to loopStart).

Also, note that there is another way that this node can perform resampling, that is, when there is a doppler shift applied to it in the face of a PannerNode. I think it makes sense to specify what needs to happen based on the multiplication of these two ratios.

Another point which was brought up on today's call was handling of values larger than one, but I think that is probably non-controversial by specifying that the final computed sampling rate ratio should be multiplied by the sampling rate of the buffer for the AudioBufferSourceNode in order to determine the target sampling rate that the resampler should use.

olivierthereaux on 11 Sep 2013

Original comment by Olivier Thereaux on W3C Bugzilla. Fri, 02 Aug 2013 09:12:14 GMT

Per our meeting on 2013-08-01 (http://www.w3.org/2013/08/01-audio-minutes.html), this is a call for volunteers to suggest a patch to the web audio API spec to define the expected behaviour when setting negative values for AudioBufferSourceNode.playbackRate.

olivierthereaux on 11 Sep 2013

Original comment by Ehsan Akhgari [:ehsan] on W3C Bugzilla. Fri, 02 Aug 2013 14:25:44 GMT

(In reply to comment #2)

Per our meeting on 2013-08-01
(http://www.w3.org/2013/08/01-audio-minutes.html), this is a call for
volunteers to suggest a patch to the web audio API spec to define the
expected behaviour when setting negative values for
AudioBufferSourceNode.playbackRate.

Hmm, this is what I was hoping to do in comment 1. :-) Do we absolutely need a patch here? I think it probably makes sense to have the basic discussion first and then move to the exact prose when everybody is on the same boat about what we want. Do you agree?

olivierthereaux on 11 Sep 2013

Original comment by Joe Berkovitz / NF on W3C Bugzilla. Fri, 02 Aug 2013 15:21:38 GMT

I was thinking about volunteering a patch but reached the same conclusion as Ehsan: we need to have a basic discussion first.

The basic outline of my proposal is different from Ehsan's but similar in spirit (I think):

define an "effective sampling rate" that is the product of the playbackRate AudioParam and the base sampling rate of the underlying AudioBuffer.

(NB I would not include the effect of downstream resampling-like effects such as doppler shifts as I think this may lead to confusion over the behavior of graphs with branched routing. It seems harder for developers to predict what will happen.)

At each sample frame, determine a buffer playback quantum that advances a notional "playback cursor" associated with the buffer. The quantum is in fractional sample frames and is (effective sampling rate) / (context sampling rate).
Prior to starting playback the cursor is initialized to the starting offset of the buffer. When playing, the cursor advances by the playback quantum; a positive quantum moves the cursor forwards while a negative quantum moves the cursor backwards. The data window between the previous and newly updated cursor position is used to interpolate the value of the next sample frame. If the cursor moved forwards, sample frames in the window are considered in forward order; if backwards, sample frames in the window are considered in reverse order. Such interpolation may take advantage of the contents of the previously calcualted data window for the buffer [I'm not going to attempt a description of how interpolation actually works here].
Within the loopStart/loopEnd region, the cursor can only advance by cycling through the loop forwards or backwards. The data window for resampled rendering includes the contents of these cycles.
If prior to the loop region, the cursor cannot move backwards past offset 0 of the buffer -- i.e. the left edge of the data window is clamped to offset 0.
If the cursor moves forwards past the end of the buffer, playback ends (I am sure the spec has better language describing this transition).
If the cursor does not move during the rendering of a frame because the effective sampling rate is 0, then the data window starts at ends at a single point within the buffer, so the output is the single [interpolated] value at thet buffer offset. This implies that a zero playback rate will produce a DC output with the value at the current notional position of the cursor.

olivierthereaux on 11 Sep 2013

Original comment by Ehsan Akhgari [:ehsan] on W3C Bugzilla. Fri, 02 Aug 2013 19:18:05 GMT

Doesn't this assume a linear interpolating resampler? The resampler that we use in Gecko is much more complicated (and of higher quality as a result) than that! (It's the libspeex resampler.)

If we're going to make it possible for implementations to compete on the resampler quality, assuming the resampling algorithm seems like a mistake.

olivierthereaux on 11 Sep 2013

Original comment by Joe Berkovitz / NF on W3C Bugzilla. Fri, 02 Aug 2013 19:34:03 GMT

@Ehsan: I had no intention of assuming any particular algorithm (and tried to call this out -- sorry if it was unclear). Of course linear interpolation is not a preferred choice. I suppose that a literal interpretation of my proposal could suggest linear interpolation, but that was not the intention.

The proposal specifies a sequence of {data window, effective sampling rate} pairs with fractional sample-offset boundaries that form the input to an arbitrary interpolation algorithm. How the interpolator makes use of this sequence is not a concern. A nonlinear interpolator can work with as much of the sequence as it likes, processing arbitrarily large batches of data points at a time.

Of course in practice an implementor would probably not accumulate such a sequence and apply an interpolation algorithm to it, this is an idealized behavior for specification purposes.

If this approach turns out to be too naîve I welcome an improved recasting of it. I think the important aspect of it has to do with the way that playback progress through the buffer is affected by a time-varying playback rate, and I found an idealized cursor the easiest way to express this progress.

olivierthereaux on 11 Sep 2013

Original comment by Chris Wilson on W3C Bugzilla. Mon, 05 Aug 2013 01:49:06 GMT

+1 to Joe's general idea - I also do NOT agree that playbackRate < 0 should change where the cursor starts; other than that, I think we're all on the same page.

olivierthereaux on 11 Sep 2013

Original comment by Joe Berkovitz / NF on W3C Bugzilla. Mon, 05 Aug 2013 17:51:36 GMT

Just to amplify Chris's comment: apart from my attempt to tease out a more detailed spec of playbackRate, the main behavioral difference in my proposal from Ehsan's is that a negative playbackRate does not cause playback to start at a different point than it would have otherwise. playbackRate determines the time derivative of a "playback path" through the buffer, but not the origin of that path, which remains the buffer offset as specified in the start() call (which defaults to 0).

If we want the ability to start playing a buffer from the end, I think there's a clearer and more explicit way to do that: attach that interpretation to a negative "offset" parameter passed to AudioBufferSourceNode.start(). I don't feel strongly that we need that feature but I do think we should avoid overloading the meaning of playbackRate w/r/t start offsets.

olivierthereaux on 11 Sep 2013

Original comment by Ehsan Akhgari [:ehsan] on W3C Bugzilla. Thu, 08 Aug 2013 03:21:06 GMT

I think I was unclear about what I meant, sorry about that. In the first paragraph of comment 1, I meant to describe the cursor jump boundaries, not that the playback should _start_ at `duration'. In other words, I meant to propose exactly the same thing as Joe described better in terms of the cursor concept. In light of comment 6, I believe we're mostly proposing the same thing (with my proposal intentionally not talking about the details of the resampling, and with Joe's proposal doing a much better job describing the cursor concept, etc.)

olivierthereaux on 11 Sep 2013

1) I think playbackRate of 0 (presuming the buffer is playing) should result in the output of the current sample, not "silence" (which implies zero). Otherwise, ramping to zero and then back to a very small value will cause a click.

2) I think negative rates should be treated as zero. Playing backwards complicates the model.

cwilso on 24 Oct 2014

It came up a few times at the Web Audio Conference that page authors were going to crazy lengths to get negative playbackRate AudioBufferSourceNodes working. Whether or not "playing backwards complicates the model", this is something the API should provide.

jernoble on 27 Jan 2015

It _could_ be specified, and I'm fine doing so if the vendors on the WG would sign up to implement it. If so, I'd suggest this as a rough draft of rules:

If looping=false, then playing backward ceases playback when it reaches the start point. So if playbackRate is negative, buffer.start(0,0) will immediately cease playback. If the offset is >0, I think playback ceases when the start offset is reached. (Clearly, the interesting scenarios here are when the playbackRate is set to negative after proceeding forward for some time.) Or, conversely (and maybe this is more interesting), playback proceeds backward past the starting offset to the beginning of the buffer. Note that duration will need to be slightly redefined to account for negative playbackRate. (I think it just doesn't apply when playbackRate<0.

If looping is true, then playing backward will work differently if it's in the "lead-in" portion (i.e. it's before the loopStart) or the looping portion when playbackRate is set to negative. I'd suggest if it's in the lead-in portion it proceeds backward until it hits the starting offset (or buffer begin, see above), then stops (despite it being "looping"). If it's in the looping portion, it should proceed to the loop begin, then wrap to loop end and keep going (in reverse).

Marked as "Needs WG Review" so we'll discuss.

cwilso on 29 Jan 2015

:+1:

notthetup on 29 Jan 2015

"It could be specified, and I'm fine doing so if the vendors on the WG would sign up to implement it."

See: https://bugs.webkit.org/show_bug.cgi?id=140955

jernoble on 2 Feb 2015

:+1: :+1:

notthetup on 2 Feb 2015

I agree this is useful, and I agree with cwilso that we need to discuss the exact behavior.

padenot on 2 Feb 2015

There is also an issue with the fact that the nature of playbackRate's interpolation is not fully specified. This is especially important when working with speeds substantially slower than 1 .

joeberkovitz on 1 Apr 2015

Resolution: spec negative playbackRate as have being exactly mirrored behaviour from the positive playbackRate behaviour.

padenot on 7 Apr 2016

🎉1

I'm going to do this by speccing the algorithm used to compute a block of audio with an AudioBufferSourceNode so that we can kill all ambiguities in one shot.

padenot on 30 Sep 2016

👍1

I've started drafting this. For now, this only handles positive sample-rate, and is not super elegant. I'd be interested in feedback.

To convert a value to sample-frame from seconds is to multiply it by the
sample-rate at its nominal sample-rate, and to round it to the nearest integer.

Let readIndex be the value of the offset attribute of the start method,
converted to sample-frame time in the sample-rate of the AudioContext.
Let startOffset be the value of the attribute when converted to
sample-frame time in the sample-rate of the AudioContext.
Let stopPoint be the value of the when parameter of the stop method
converted to sample-frame time in the sample-rate of the AudioContext, or
+Infinity if stop has not been called.
Let duration be the value of the duration attribute of the AudioBuffer,
converted to sample-frame time, in the sample-rate of the AudioContext.
Let currentTime be the value of AudioContext.currentTime converted to
sample-frame time at the sample-rate of the AudioContext.
Let loopStart be the value of the loopStart attribute converted to
sample-frame time at the sample-rate of the AudioContext.
Let loopEnd be the value of the loopStart attribute converted to
sample-frame time at the sample-rate of the AudioContext.

Writing silence to a sequence s from index start to index end means
setting the elements between s[start] to s[end] to 0.0.

Rendering a block of audio for an AudioBufferSourceNode means executing the
following steps:

Let s be a sequence of 128 float.
Let writeIndex be 0.
Let input rate be the sample-rate of the AudioBuffer.
Let output rate be the sample-rate of the AudioContext divided by the
value of computedPlaybackRate at currentTime.
Let resampling ratio be input rate divided by output rate.
While writeIndex is not 128
1. If currentTime is less than startOffset:
  1. Write silence to s from index writeIndex to index startOffset - currentTime.
  2. Increment writeIndex by startOffset - currentTime.
2. If looping is True, let outputFrames be the minimum of count,
  
  stopPoint - currentTime and loopEnd - currentTime.
3. Else, let outputFrames be the minimum of count, stopPoint - currentTime, and duration.
4. Resample a portion of the input buffer from input rate to output rate
  
  to produce outputFrames frames, and copy them to s starting at
  
  writeIndex.
5. Increment readIndex by the number of frames consumed by the resampler.
6. If looping is True and readIndex is equal to loopEnd, set readIndex
  
  to loopStart.
7. Increment writeIndex by the number of frames produced by the resampler.
8. If currentTime + writeIndex is equal to stopPoint:
  1. Set an ended flag to true.
  2. Write silence to s from index writeIndex to index endOffset.
If ended is True:
1. Post a task to fire a simple event named "ended" at the
  
  AudioBufferSourceNode, and remove the self-reference.

This algorithm is intentionaly vague on the resampling process, in terms of
input and output frame count, as different resampling techniques have different
characteristics.

padenot on 3 Oct 2016

Thanks @padenot! Here are some of the content issues and questions I see. I could try to take a crack at addressing them but thought it would be better to surface these questions first.

"converted to sample-frame time in the sample-rate of the AudioContext" is repeated many times, we should factor this out of the definitions.
some of the variables defined up front are not AudioContext sample-rate times, but time _intervals_. Others are positions within the buffer (like loopStart or loopEnd and so it's not clear how these are to be converted into times)
In these definitions it is not clear that rounding to the nearest integer is always correct. for instance with respect to loop start/end points that we have now said could lie at fractional sample frame positions. As another example, a looped buffer in sample rate A might be looping in a context at sample rate B, and we wouldn't want glitches on every loop.
count is never defined in the render algorithm.
In step 6/i, it seems as though one could wind up writing past index == 127
In 6.ii, stopPoint - currentTime could be negative since we could be past the stop point
In 6.ii, loopEnd - currentTime is not a number of frames because loopEnd is not a context time, but a position within the AudioBuffer (see my comment above).
6/vi, looping should be loop

joeberkovitz on 3 Oct 2016

Also, recall https://github.com/WebAudio/web-audio-api/issues/915#issuecomment-248930220 where we decided that we should actually do sub-sample accurate sampling. Although that was for the oscillator, I think we need to do the same for a buffer source because I think the same issues will show up.

rtoy on 8 Nov 2016

👍1

I think the algorithm would be simpler if we just first described the case where the buffer rate matched the context rate. When the rates don't match, we can say that the same algorithm applies if the buffer behaved as if it were first resampled and used in this algorithm.

I think you also need a step 8 for the case where ended is not true so that you go back to step 2. Also probably want to reorder the initial steps so that step 2 is the step just before step 6. All of these steps are just initializations that only need to happen once.

I also don't think we need to make this process 128 frames at a time. To describe the algorithm, we don't really need the finite-sized s buffer. We can assume unlimited length and writeIndex keeps track of what we're doing, and it can increment forever.

rtoy on 8 Nov 2016

Another iteration on this, with comments addressed and negative playbackRate support. Note the loop, depending on if we're looping and reached the end point of the loop, we'll loop multiple times, same if the loop is very small. I tried to perform sub-sample accurate start, but it's probably not correct for the loop end, I need to double check.

To convert a value to sample-frame from seconds is to multiply it by the
sample-rate at its nominal sample-rate.

Sample-frame time is a fractional time value in frames, in the sample-rate of
the AudioContext. It MUST be rounded to access the samples themselves. This is
used to implement sub-sampling start point, stop point and loop points.

This algorithm is described for a playbackRate of 1.0, and when the
sample-rate of the AudioBuffer is the same as the sample-rate of the
AudioContext, i.e. when no resampling is necessary. If this is not the case,
execute those steps before producing a block of audio:

Let input rate be the sample-rate of the AudioBuffer.
Let output rate be the sample-rate of the AudioContext divided by the
value of computedPlaybackRate at currentTime.
Let resampling ratio be input rate divided by output rate.
If resampling ratio is negative, reverse the data of the AudioBuffer and
let resampling ratio be -resampling ratio.
If resampling ratio is not 1.0, resample the AudioBuffer data, and use
this resampled audio data as the source data.
Else, the source data is the raw data of the AudioBuffer.
Let readIndex be the value of the offset attribute of the start method,
converted to sample-frame time.

Writing silence to a sequence s from index start to index end means
setting the elements between s[start] to s[end] to 0.0.

Rendering a block of audio for an AudioBufferSourceNode means executing the
following steps:

Let source be the buffer containing the source data, to be indexed by
readIndex.
Let startOffset be the value of the attribute when converted to
sample-frame time.
Let stopPoint be the value of the when parameter of the stop method
converted to sample-frame time, or +Infinity if stop has not been called.
Let duration be the value of the duration attribute of the AudioBuffer,
converted to sample-frame time.
Let currentTime be the value of AudioContext.currentTime converted to
sample-frame time.
Let loopStart be the value of the loopStart attribute converted to
sample-frame time.
Let loopEnd be the value of the loopStart attribute converted to
sample-frame time.
Let s be a sequence of 128 float.
Let writeIndex be 0.
While writeIndex is not 128
1. If currentTime is less than startOffset:
  1. Write silence to s from index writeIndex to the minimum of
    
    startOffset - currentTime and 128.
2. Increment writeIndex by startOffset - currentTime.
3. Jump to the beginning of this loop.
4. Let count be 128 - writeIndex.
5. If loop is True, let outputFrames be the minimum of count,
  
  stopPoint - currentTime and startOffset + loopEnd - currentTime.
6. Else, let outputFrames be the minimum of count, stopPoint - currentTime, and startOffset + duration.
7. If readIndex is not an integer, perform sub-sample interpolation:
  1. Let left be readIndex - ceil(readIndex) and right be 1 - left.
  2. Set s[writeIndex] to source[readIndex] * left.
  3. Increment writeIndex by 1. If writeIndex is 128, jump to the
    
    beginning of this loop.
  4. Set s[writeIndex] to source[readIndex] * right.
  5. Let readIndex be ceil(readIndex). If readIndex is greater than
    
    loopEnd and loop is True, substract loopEnd - loopStart from
    
    readIndex and jump to the beginning of this loop.
8. Copy outputFrames frames of audio from the source data starting at index
  
  readIndex to s starting at index writeIndex.
9. Increment readIndex by outputFrames.
10. Increment writeIndex by outputFrames.
11. If loop is True and readIndex is greater or equal to loopEnd,
  
  substract loopEnd - loopStart to readIndex.
12. If currentTime + writeIndex is greater or equal to stopPoint or
  
  startOffset + duration:
  1. Set an ended flag to true.
  2. Write silence to s from index writeIndex to index endOffset.
If ended is True:
1. Post a task to fire a simple event named "ended" at the
  
  AudioBufferSourceNode, and remove the self-reference.

padenot on 17 Nov 2016

If resampling ratio is not 1.0, resample the AudioBuffer data, and use this resampled audio data as the source data.

Specify that the resampling results in new audio data whose sample rate is now output rate.

Let readIndex be the value of the offset attribute of the start method,
converted to sample-frame time.

The phrase "sample-frame time" is ambiguous with at least 3 different rates in play. In this case I believe offset would be specified by the caller as time units that assume the buffer's own sample rate, i.e. input rate. But in other cases, the phrase clearly refers to AudioContext rate. I think it would be clearer always to say something explicit like, "let readIndex be offset multiplied by input rate". This issue comes up a bunch of times in the algorithm.

ii. Increment writeIndex by startOffset - currentTime.

I think this needs to increment writeIndex by the lesser of the given expression, or 128 (the same amount as the silence that was written in te previous step).

vi. Else, let outputFrames be the minimum of count, stopPoint - currentTime, and startOffset + duration.

The last expression I think should be startOffset + duration - currentTime.

vii (sub sample interpolation)

I see a few different issues here although perhaps I've missed something important:

I don't see any actual interpolation between successive samples yielding a result corresponding to s[i]*left + s[i+1]*right. It looks like you are setting either a left-weighted or right-weighted value into the output sample frame, rather than a sum of both.
left would be readIndex - floor(readIndex) wouldn't it? subtracting the ceiling value will yield a negative interpolation weight for the left value.
step e. "Let readIndex be ceil(readIndex)" doesn't seem right to me - if readIndex is fractional, surely it should remain fractional but be incremented by 1 as it advances through a fractionally-shifted region of the buffer. To avoid glitches, interpolation must occur throughout a shifted region, not just at the endpoints of that region.

joeberkovitz on 17 Nov 2016

Let me propose a somewhat different expression of the algorithm. I think it's a bit simpler, but a bit harder to visualize. It would be best if I drew some simple diagrams to show what happens at the buffer start, the loop end and the loop start points. Anyway, without further ado:

Let

t = context.currentTime
dt = 1 / context.sampleRate

tb = ABSN buffer index
dtb = playbackRate
ts = start time for ABSN
tls, tle = loop start and end time

b = ABSN buffer, resampled if necessary to the context sample rate
such that b[0] still represents the very first value.

If ABSN is started with start(when), set ts = when, duration = length
of buffer in seconds, and toff = 0.

If ABSN is started with start(when, offset, duration), set ts = when,
toff = offset and duration = duration if given or infinity.

Let bufferSample(k) be a function that interpolates values from the
ABSN buffer such that if k is an integer, b[k] is the result. If k is
not an integer, compute an interpolated value using b[n] and b[n+1]
where n = floor(k). Interpolation method is not specified and more
samples of the buffer are allowed to be used.

Let output(x) be a function outputs the value x as the next output
sample of the ABSN.

t = 0;
while (1) {
  if (t <= ts < t+dt) {
    // Buffer is starting. Compute an offset into the buffer based on
    // when the current sample is being taken.
    tb = (t + dt - ts + toff) * sampleRate

    total = 0;
    while (1) {
      if (loop == false)
        if (total > duration) {
          // We've reached the end of the buffer and/or duration, so
          // stop
          break;
        }
      } else {
        // We're looping
        if (total > duration) {
          // We've output duration seconds of audio. Time to stop.
          break;
        }
        if (tle < tb) {
          // We're trying to output the first sample PAST the end of
          // the loop.  Rewind the buffer pointer back to the loop
          // start.
          tb = ceil(tls / sampleRate) * sampleRate;
        }
      }

      output(bufferSample(tb))

      tb = tb + dtb;
      // Wrap tb if tb goes past the end of the buffer.  This wraps tb
      // back near the beginning of the buffer or near the start of
      // the loop point.
      tb = wrapPointer(tb);

      t = t + dt;
      total = total + dt;
    }
  }

  output(0);
  t = t + dt;
}

rtoy on 18 Nov 2016

@rtoy I agree that this way of expressing the algorithm is easier to understand, although it would need some preamble explaining that it ignores rendering quanta and instead describes an idealized sequence of sample frames generated by an ABSN as if nothing else were going on.

Also, this algorithm can deal with computed playback rate more gracefully. The previous version appears to treat computedPlaybackRate as a constant that can be eliminated via resampling; this version only resamples to compensate for the AudioBuffer's own sample rate, and treates computedPlaybackRate as truly dynamic (since it is k-rate).

I did identify a couple of problems with the algorithm and also felt the control structures and variable names made it a bit more obscure, so I've taken the liberty of trying to do another iteration below -- but it's really just a restatement of @rtoy's version.

I also attempted to incorporate explicit logic for the wrapping of loops as I believe this needs to be spelled out and not left to the UA.

Here we go:

while (1) {
  if (currentTime <= start && start < currentTime + dt) {
    // Buffer is starting. Compute an offset into the buffer based on
    // when the current sample is being taken.
    sampleIndex = (currentTime + dt - start + offset) * sampleRate

    total = 0;
    while (1) {
      if (total > duration) {
        // We've reached the end of the buffer and/or duration, so
        // stop
        break;
      }

      if (loop && sampleIndex > loopEndIndex) {
        // We're trying to output the first sample PAST the end of
        // the loop.  Rewind the buffer pointer back to the loop
        // start.
        sampleIndex = loopStartIndex;

        // @rtoy: Didn't understand rationale for the following
        // sampleIndex = ceil(loopStart / sampleRate) * sampleRate;
      }

      output(bufferSample(sampleIndex))

      sampleIndex += dtb * computedPlaybackRate;  // @rtoy did not see any reference to playback rate 

      // Wrap tb if tb goes past the end of the buffer.  This wraps tb
      // back near the beginning of the buffer or near the start of
      // the loop point.
      while (loop && sampleIndex > loopEndIndex) {
        sampleIndex -= loopEndIndex - loopStartIndex;
      }

      currentTime += dt;
      total += dt;
    }
  }

  output(0);
  currentTime += dt;
}

joeberkovitz on 21 Nov 2016

Thanks for cleaning up the code. I was lazy about typing things out, so this looks much better. I think this algorithm works fine if computedPlaybackRate is non-negative. If it's negative, we'll have to do something about the various tests because they'll have to change direction or maybe swap loop start and loop end. (I assume that's how loop points would work for negative playbackRate.)

You don't describe how loopStartIndex and loopEndIndex are computed from loopStart and loopEnd. I think loopStartIndex is probably ceil(loopStart / sampleRate) * sampleRate and loopEndIndex is similar but uses floor instead of ceil. But maybe loopStartIndex = loopStart * sampleRate.

If loopStart * sampleRate lies between two sample points, what should we output? I guess I was thinking we should output the buffer sample just past that. But maybe it should be the interpolated value between the sample points?

rtoy on 28 Nov 2016

If it's negative, we'll have to do something about the various tests because they'll have to change direction or maybe swap loop start and loop end. (I assume that's how loop points would work for negative playbackRate.)

Correct, probably that while loop should work in the opposite sense if computedPlaybackRate is negative.

You don't describe how loopStartIndex and loopEndIndex are computed from loopStart and loopEnd. I think loopStartIndex is probably ceil(loopStart / sampleRate) * sampleRate and loopEndIndex is similar but uses floor instead of ceil. But maybe loopStartIndex = loopStart * sampleRate.

Two points:

We can't use ceil() or floor() because we've agreed that loop points are not quantized. These are "indices" only in the sense that they are expressed in units of sample frames, but they can have a nonzero fractional part. So loopStart(End)Index = loopStart(End) * buffer sample rate.

We need to note in the algorithm that sampleRate has to be the sample rate of the buffer, not the sample rate of the context or the computed playback rate.

If loopStart * sampleRate lies between two sample points, what should we output? I guess I was thinking we should output the buffer sample just past that. But maybe it should be the interpolated value between the sample points?

Since loopStart is allowed to lie between frames, on any sort of looping back to a start point we should just assign loopStartIndex (which can be fractional as noted above) to sampleIndex and not try to force to the frame to the left or to the right.

A similar argument applies to the loop endpoint in a negative playback rate situation.

joeberkovitz on 28 Nov 2016

On Mon, Nov 28, 2016 at 10:53 AM, Joe Berkovitz notifications@github.com
wrote:

If it's negative, we'll have to do something about the various tests
because they'll have to change direction or maybe swap loop start and loop
end. (I assume that's how loop points would work for negative playbackRate.)

Correct, probably that while loop should work in the opposite sense if
computedPlaybackRate is negative.

You don't describe how loopStartIndex and loopEndIndex are computed from
loopStart and loopEnd. I think loopStartIndex is probably ceil(loopStart /
sampleRate) * sampleRate and loopEndIndex is similar but uses floor instead
of ceil. But maybe loopStartIndex = loopStart * sampleRate.

Two points:

-

We can't use ceil() or floor() because we've agreed that loop points
are not quantized. These are "indices" only in the sense that they are
expressed in units of sample frames, but they cannot be quantized. So
loopStart(End)Index = loopStart(End) * buffer sample rate.

I think I've confused myself. I need to draw a picture and study it to
figure out what we really want.

>

-
-

We need to note in the algorithm that sampleRate has to be the sample
rate of the buffer, not the sample rate of the context or the computed
playback rate.

This is also confusing because we have two sampleRates: one is for the
audio context and one is for the buffer. I'd really prefer to state the
algorithm as if the buffer was already resampled to the context rate. The
implementation can then do whatever optimizations it wants, including doing
the resampling during playback (via linear interpolation). Or actually
doing the resampling first with a high quality resampler.

I think of resampling as really orthogonal to specifying how loops and
playbackRate works.

>

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-263358902,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPPlCDOEAXqSyQ7JWgCE-1KXJi-Klks5rCyMagaJpZM4A_H6-
.

rtoy on 28 Nov 2016

I'd really prefer to state the algorithm as if the buffer was already resampled to the context rate.

Yes, that makes sense to me -- my real point was that computed playback rate (i.e. modulating the rate via the playbackRate AudioParam) should have no influence on loop point location. Loop points are a stable way of referencing the buffer content.

joeberkovitz on 29 Nov 2016

As discussed on yesterday's call I'm going to take a crack at refining this further.

joeberkovitz on 2 Dec 2016

After quite a bit of thinking and much careful checking, I offer the following algorithm which makes heavy use of all the work done so far. I believe it works fine for all these requirements:

forward and backward playback
subsample-accurate start, stop, offset, loopStart, loopEnd
well defined and intuitive behavior with respect to loop wrapping at any playback rate
arbitrary resampling algorithm
arbitrary interpolation algorithm

Probably the key new feature of this approach is the approach to loops, in which a looped buffer is considered to be an infinite sequence of half-open ranges [0..loopStart), [loopStart..loopEnd), [loopStart+loopLength..loopEnd+loopLength), .... This greatly clarifies the way that interpolation behaves in the neighborhood of loop boundaries, even when these do not lie at exact sample indices; the comments explain in detail.

I've adopted the idiom of an AudioWorkletProcessor to describe the algorithm because this helped me understand it and check it. It is thus a hybrid of @padenot's original block-rendering-oriented approach and @rtoy's approach. However, we could reduce this to a narrative description if desired.

// Description of the algorithm employed by
// an AudioBufferSourceNode to render successive
// blocks of audio. The description is framed as if
// the node was implemented using an AudioWorkletProcessor.
//
// It is designed for clarity only, not efficiency.

class AudioBufferSourceProcessor extends AudioWorkletProcessor {
  static get parameterDescriptors() {
    return [{
      name: 'detune',
      defaultValue: 0
    }, {
      name: 'playbackRate',
      defaultValue: 1
    }];
  }

  // Initialize processing
  constructor(options) {
    this.readTime = 0;
    this.buffer = this.originalBuffer = options.buffer;
    this.loop = options.loop;
    this.loopStart = options.loopStart;
    this.loopEnd = options.loopEnd;
  }

  // To be called when start() is applied to the node
  function start(when, offset, duration) {
    this.start = when;
    this.offset = 0;
    this.stop = Infinity;
    this.bufferTime = -1;

    if (arguments.length > 1) {
      this.offset = offset;
    }
    if (arguments.length > 2) {
      this.stop = this.when + duration;
    }
  }

  // To be called when stop() is applied to the node
  function stop (when) {
    this.stop = when;
  }

  function process(inputs, outputs, parameters) {
    // Set up working variables
    let currentTime = contextInfo.currentTime;  // current context time of next output frame
    let dt = 1 / contextInfo.sampleRate;  // context time per sample
    let index = 0;  // index of next frame to be output
    let computedPlaybackRate = parameters.playbackRate[0] * Math.pow(2, parameters.detune[0] / 1200);

    // Optionally resample buffer to reduce need for interpolation
    optimizeBuffer(computedPlaybackRate);

    // Render next block
    while (index < outputs[0][0].length) {
      if (currentTime < this.start || currentTime >= this.stop) {
        for (let channel = 0; channel < buffer.numberOfChannels; channel++) {
          outputs[0][channel][index] = 0;
        }
      }
      else {
        // set up bufferTime the first time we have a signal to put out
        if (this.bufferTime < 0) {
          this.bufferTime = offset + (currentTime - this.start) * computedPlaybackRate;
        }
        for (let channel = 0; channel < buffer.numberOfChannels; channel++) {
          outputs[0][channel][index] = bufferSample(this.bufferTime, channel);
        }
        this.bufferTime += dt * computedPlaybackRate;  // advance read time of buffer, independent of loop points
      }

      currentTime += dt;   // advance output time
      index += 1;  // advance output index
    }

    // Consider the node to be active if we have not reached stop time yet.
    return currentTime < this.stop;
  }

  // Returns a channel's signal value at the time offset "effectiveTime" in
  // the buffer. This function takes care of enforcing silence before and after
  // the buffer contents, and also folds the body of any loop.
  function bufferSample(effectiveTime, channel) {
    // Convert time to a playhead position with fractional part
    let bufferPos = effectiveTime * this.buffer.sampleRate;

    // Now get an interpolated value.
    var leftFrame, rightFrame, interval;

    if (this.loop) {
      let loopStartPos = this.loopStart * this.buffer.sampleRate;  // playhead position for loop start
      let loopEndPos = this.loopEnd * this.buffer.sampleRate;  // playhead position for loop end
      let loopLength = loopEndPos - loopStartPos; // loop length in frame units

      // Determine the first and last exact sample frame positions that lie
      // within the body of the loop, which is the half-open range
      // [loopStartPos, loopEndPos) -- that is, loopEndPos is exclusive.
      let loopFirstFrame = Math.ceil(loopStartPos);
      let loopLastFrame = Math.ceil(loopEndPos - 1);   // N => N-1; (N + epsilon) => N

      // At any loop wrap point, the mapping between a playhead position in units of frames
      // and fractional sample indices looks like this (where N is a nonnegative loop iteration count)
      //
      //   playhead position                        sample index            interpolated?
      //
      //   loopLastFrame + N*loopLength             loopLastFrame           no
      //   loopEndPos + N*loopLength - epsilon      loopEndPos - epsilon    yes
      //   loopStartPos + (N+1)*loopLength          loopStartPos            yes
      //   loopFirstFrame + (N+1)*loopLength        loopFirstFrame          no
      //
      // In any region covered by this range of values, we are potentially interpolating
      // between a "left sequence" of frames ending in loopLastFrame, and a "right sequence"
      // of frames beginning with loopFirstFrame.

      // Fold the loop to bring the value of N to zero, by requiring that that
      // the playhead poition not exceed (loopFirstFrame + loopLength).
      while (bufferPos >= loopFirstFrame + loopLength) {
        bufferPos -= loopFrames;
      }

      if (bufferPos >= loopLastFrame) {
        // If after folding the playhead is after the last exact frame in the loop,
        // then we'll be interpolating at the wrap boundary between a sequence of frames
        // ending in loopLastFrame, and a (wrapped) sequence of frames beginning with
        // loopStartFrame.

        // The time interval between left and right frames may be less than 1
        // frame in this case, because of fractional loop points.

        leftFrame = loopLastFrame;
        rightFrame = loopFirstFrame;
        interval = (loopFirstFrame + loopFrames) - loopLastFrame;
      }
      else {
        leftFrame = Math.floor(bufferPos);
        rightFrame = leftFrame + 1;
        interval = 1;
      }
    }
    else {
      leftFrame = Math.floor(bufferPos);  // get the exact frame index before the time of interest
      rightFrame = leftFrame + 1; // and the frame after that
      interval = 1; // interval in sample frames between left and right
    }

    return interpolateValue(bufferPos, channel, leftFrame, rightFrame, interval);
  }

  // Return an interpolated value for the playhead position bufferPos,
  // working from a sequence of exact frames at:
  //   leftFrame-M ... leftFrame, rightFrame ... rightFrame+N
  // that map onto playhead positions:
  //   leftFrame-M ... leftFrame, leftFrame + interval ... leftFrame + interval + N
  // with the constraint that positions outside the buffer content may not be included.
  function interpolateValue(bufferPos, channel, leftFrame, rightFrame, interval) {
    // The UA may employ any desired algorithm.
    // It may also elect to use bufferPos as an exact index and not interpolate,
    // if the difference between bufferPos and leftFrame is sufficiently small.

    /*
      This sample implementation uses linear interpolation between two adjacent frames.

      let weight = (bufferFrame - leftFrame) / interval;
      let leftValue = leftFrame >= 0 ? this.buffer.getChannelData(channel)[leftFrame] : 0;
      var rightValue = rightFrame < buffer.length ? this.buffer.getChannelData(channel)[rightFrame] : 0;
      return leftValue + (weight * rightValue);
    */
  }

  function optimizeBuffer(playbackRate) {
    // This function may resample the buffer contents, entirely or partially,
    // as often as desired for reasons of quality or computational efficiency.
    // The results of the resampling, if carried out, are available to the
    // bufferSample() method via this.buffer.

    /*
      Example implementation that ensures buffer is at optimal rate for the first-rendered block:

      let bestSampleRate = contextInfo.sampleRate / playbackRate;
      if (!this.resampled && this.buffer.sampleRate != bestSampleRate ) {
        this.buffer = cubicResample(this.originalBuffer, bestSampleRate);
        this.resampled = true;
      }
    */
  }
}

joeberkovitz on 3 Dec 2016

I thought it would be useful to abstract a narrative definition that's a bit more rigorous. Here it is.

Definitions

Let start correspond to the when parameter to start() if supplied, else 0.
Let offset correspond to the offset parameter to start() if supplied, else 0.
Let stop correspond to the sum of the when and duration parameters to start() if supplied, or the when parameter to stop(), otherwise _Infinity_.
Let buffer be the AudioBuffer employed by the node.
Let loop, loopStart and loopEnd be the loop-related attributes of the node, with the loop body clamped to the range [0, buffer.length].
A _playhead position_ for buffer is any quantity representing an unquantized time offset in seconds, relative to the time coordinate of the first sample frame in the buffer. Playback rate and AudioContext sample rate are not relevant: this offset is expressed purely in terms of the buffer's audio content.
Let _framePosition(index)_ be a many-valued function yielding a set of one or more _playhead positions_ for the sample frame within the buffer at _index_. These represent the idealized times at which the frame would be rendered, given a start time of 0, a playback rate of 1 and an infinite sample rate. The function is as follows:
1. Let bufferTime be index / buffer.sampleRate.
2. If loop is false, the result is bufferTime.
3. If loop is true,
  1. If bufferTime < loopStart, the result is bufferTime.
  2. If bufferTime >= loopStart and bufferTime < loopEnd, the function has multiple results given by bufferTime + (count * (loopEnd - loopStart)), where count takes on non-negative integer values.
  3. If bufferTime >= loopEnd, the position is not defined.
Let _frameValue(index)_ correspond to the vector of actual signal values in buffer at the given index, one component per channel.
Let the _playback frame sequence_ for buffer be the set of all tuples [_framePosition(index)_, _index_, _frameValue(index)_], ordered by increasing values of _framePosition(index)_. This sequence describes a signal whose values are known only at discrete, monotonically increasing playhead positions that correspond exactly to specific samples in the buffer.

For an unlooped buffer, this sequence is finite (frame values are omitted here, since they are not relevant):

framePosition(index) | index
----- | --------------------
0 | 0
1 / buffer.sampleRate | 1
2 / buffer.sampleRate | 2
... | ...
(length - 1) / buffer.sampleRate | (length - 1)

For a looped buffer, this sequence is infinite. Let loopStartFrame be ceil(loopStart * buffer.sampleRate) (the first exact frame index within the loop body), and loopEndFrame be ceil(loopEnd * buffer.sampleRate - 1) (the last exact frame index within the loop body). The sequence is as follows:

framePosition(index) | index
----- | --------------------
0 | 0
1 / buffer.sampleRate | 1
2 / buffer.sampleRate | 2
... | ...
loopStartFrame/buffer.sampleRate | loopStartFrame
... | ...
loopEndFrame/buffer.sampleRate | loopEndFrame
loopStartFrame/buffer.sampleRate + (loopEnd - loopStart) | loopStartFrame
... | ...
loopEndFrame/buffer.sampleRate + (loopEnd - loopStart) | loopEndFrame
loopStartFrame/buffer.sampleRate + 2*(loopEnd - loopStart) | loopStartFrame
... | ...

Interpolation and Buffer Optimization

Let the function _interpolateFrame(pos)_ yield a vector which estimates the channel values at the given playhead position of _pos_, which need not map onto a exact index in the buffer. The interpolation method MUST obey all of the following constraints:
1. The method relies ONLY on the relationship between playhead positions and signal values supplied by the playback frame sequence. (Typically, only a limited range of tuples in the neighborhood of _pos_ are considered.)
2. For _pos_ equal to _framePosition(index)_, the result is exactly _frameValue(index)_.
3. For _pos_ < 0, the result is a zero vector (silence).
4. For _pos_ >= buffer.length / buffer.sampleRate where loop is false, the result is a zero vector (silence).

Note that this definition ignores loop points, since these are already embodied in the definition of the playback frame sequence.

Let the operation _optimize the buffer_ be any operation that alters both the buffer contents and sample rate, while attempting to minimize changes to the value of _interpolateFrame(pos)_. The nature of the operation is up to the UA. Examples of such operations might include upsampling, downsampling, applying a subsample offset, or loop unrolling.

Initialization

Let bufferTime be the _playhead position_ within buffer of the next output sample frame. Assign it the initial value -1 to indicate that the position has not yet been determined.
Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

Let currentTime be the current time of the AudioContext.
Let dt be 1 / (_context sample rate_).
Let index be 0.
Let computedPlaybackRate be _playbackRate_ * pow(2, _detune_ / 1200).
Optimize the buffer during rendering, if desired.
While index is less than the length of the audio block to be rendered:
1. If currentTime < start or currentTime >= stop, emit silence for the output frame at index.
2. Else,
  1. If bufferTime < 0, set bufferTime to offset + (currentTime - start) * computedPlaybackRate.
  2. Emit the result of interpolateFrame(bufferTime) as the output frame at index.
  3. Increase bufferTime by dt * computedPlaybackRate.
3. Increase index by 1.
4. Increase currentTime by dt.
If currentTime < stop, consider playback to have ended.

joeberkovitz on 4 Dec 2016

I have not yet read through everything, but the code looks really nice and is quite clear. Now for a few comments.

Not sure we want to define this in terms of AudioWorklet. We haven't done that anywhere else and we don't yet have any practical experience with a working AudioWorklet implementation.
I wonder if optimizeBuffer(computedPlaybackRate) is really needed. That seems to be an implementation optimization that isn't really needed for the algorithm.

I think this is wrong:

this.bufferTime = offset + (currentTime - this.start) * computedPlaybackRate;

If computedPlaybackRate were, say, a zillion, and currentTime > this.start, the bufferTime would be huge so the first output would be very far along the actual buffer. I think instead of computedPlaybackRate we want just dt.

Having bufferSample keep track of looping seems a bit complicated. I did like your original idea of the buffer index being an time value. Then you can say I want the value of the buffer at time t, and that would implicitly take into account the sample rate of the buffer and allow the implementation to do some appropriate resampling and interpolation as needed.

So, I guess what I'm saying is let the main loop maintain the bufferIndex time value, including looping and computedPlaybackRate, and let bufferSample(t) produce the appropriate buffer value at time t.

rtoy on 6 Dec 2016

Not sure we want to define this in terms of AudioWorklet. We haven't done that anywhere else and we don't yet have any practical experience with a working AudioWorklet implementation.

I had some misgivings about that too and perhaps the narrative description would be better (did you read it yet?) Or we could just keep the JS description and move it away from AudioWorklet.

I wonder if optimizeBuffer(computedPlaybackRate) is really needed. That seems to be an implementation optimization that isn't really needed for the algorithm.

Agreed, and this is a point that is perhaps clearer in the narrative version. I do think it's important to state that optimization (resampling, etc) at the UA's discretion is _allowed_.

I think this is wrong: this.bufferTime = offset + (currentTime - this.start) * computedPlaybackRate; If computedPlaybackRate were, say, a zillion, and currentTime > this.start, the bufferTime would be huge so the first output would be very far along the actual buffer. I think instead of computedPlaybackRate we want just dt.

I still believe this expression is correct -- if computedPlaybackRate is a crazy-large value and currentTime > start, then bufferTime could well be past the end of the buffer. Why is that wrong? The bufferSample() function (or equivalent language in the narrative version) takes care of checking the buffer limits and enforcing silent output in this case.

Having bufferSample keep track of looping seems a bit complicated. I did like your original idea of the buffer index being an time value. Then you can say I want the value of the buffer at time t, and that would implicitly take into account the sample rate of the buffer and allow the implementation to do some appropriate resampling and interpolation as needed.

I agree - this is the approach adopted in the narrative version of the algorithm. I think it is cleaner.

Perhaps you could go over the narrative piece and see how you find it. I think it makes the approach clearer than the code and it avoids unnecessarily prescriptions about how things work.

joeberkovitz on 6 Dec 2016

On Tue, Dec 6, 2016 at 10:28 AM, Joe Berkovitz notifications@github.com
wrote:

Not sure we want to define this in terms of AudioWorklet. We haven't done
that anywhere else and we don't yet have any practical experience with a
working AudioWorklet implementation.

I had some misgivings about that too and perhaps the narrative description
would be better (did you read it yet?) Or we could just keep the JS
description and move it away from AudioWorklet.

I've skimmed the narrative, but not studied it in detail.

I wonder if optimizeBuffer(computedPlaybackRate) is really needed. That
seems to be an implementation optimization that isn't really needed for the
algorithm.

Agreed, and this is a point that is perhaps clearer in the narrative
version. I do think it's important to state that optimization (resampling,
etc) at the UA's discretion is allowed.

I think this is wrong: this.bufferTime = offset + (currentTime -
this.start) * computedPlaybackRate; If computedPlaybackRate were, say, a
zillion, and currentTime > this.start, the bufferTime would be huge so the
first output would be very far along the actual buffer. I think instead of
computedPlaybackRate we want just dt.

I still believe this expression is correct -- if computedPlaybackRate is a
crazy-large value and currentTime > start, then bufferTime could well be
past the end of the buffer. Why is that wrong? The bufferSample()
function (or equivalent language in the narrative version) takes care of
checking the buffer limits and enforcing silent output in this case.

Let's say bufferTime ends up being near the end of the buffer (using your
computation). Why should the very first output from the buffer come from
the end of the buffer? I would think the very first output should be very
near the beginning. The second output sample would then be near the end,
which makes sense to me.

Having bufferSample keep track of looping seems a bit complicated. I did
like your original idea of the buffer index being an time value. Then you
can say I want the value of the buffer at time t, and that would implicitly
take into account the sample rate of the buffer and allow the
implementation to do some appropriate resampling and interpolation as
needed.

I agree - this is the approach adopted in the narrative version of the
algorithm. I think it is cleaner.

Perhaps you could go over the narrative piece and see how you find it. I
think it makes the approach clearer than the code and it avoids
unnecessarily prescriptions about how things work.

I'm going to read over that very soon....

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-265230892,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPMqt_AHYwcPDOz5ReGBe3IZaWgD5ks5rFak4gaJpZM4A_H6-
.

--
Ray

rtoy on 6 Dec 2016

Let's say bufferTime ends up being near the end of the buffer (using your
computation). Why should the very first output from the buffer come from
the end of the buffer? I would think the very first output should be very
near the beginning. The second output sample would then be near the end,
which makes sense to me.

The answer has to do with sample-accurate start times as required by #915. If start occurs at an exact sample frame, then currentTime will equal start on the first-rendered frame of output. Hence (currentTime - start) * computedPlaybackRate is going to be zero, as you expect.

joeberkovitz on 6 Dec 2016

On Tue, Dec 6, 2016 at 11:40 AM, Joe Berkovitz notifications@github.com
wrote:

Let's say bufferTime ends up being near the end of the buffer (using your
computation). Why should the very first output from the buffer come from
the end of the buffer? I would think the very first output should be very
near the beginning. The second output sample would then be near the end,
which makes sense to me.

The answer has to do with sample-accurate start times as required by #915
https://github.com/WebAudio/web-audio-api/issues/915. If start occurs
at an exact sample frame, then currentTime will equal start on the
first-rendered frame of output. Hence (currentTime - start) *
computedPlaybackRate is going to be zero, as you expect.

Sorry, I still don't understand. If the start time isn't on a sample
boundary, currentTime - start is non-zero as expected. Why would I want
the first output sample to come from very near the beginning of the
buffer? If computedPlaybackRate were huge, the offset will also be huge
and probably not produce a sample near the beginning of the buffer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-265251060,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPKeWGkin7q4tJF0rllDvSH6Y_JUeks5rFbotgaJpZM4A_H6-
.

--
Ray

rtoy on 6 Dec 2016

Some initial comments. I like this narrative, but I think the code is
easier to follow.

On Sun, Dec 4, 2016 at 10:28 AM, Joe Berkovitz notifications@github.com
wrote:

I thought it would be useful to abstract a narrative definition that's a
bit more rigorous. Here it is.
Definitions

1.

Let start correspond to the when parameter to start() if supplied,
else 0.

I think it's better to say "is equal to" instead of "correspond to", here
and below.

>

1.
2.

Let offset correspond to the offset parameter to start() if supplied,
else 0.
3.

Let stop correspond to the sum of the when and duration parameters to
start() if supplied, or the when parameter to stop(), otherwise
Infinity.

This needs to be defined a bit better, but isn't really key to the
overall algorithm. It's not clear which takes precedence (sum or stop), if
there is any.

>

1.
2.

Let buffer be the AudioBuffer employed by the node.
3.

Let loop, loopStart and loopEnd be the loop-related attributes of the
node, with the loop body clamped to the range [0, buffer.length].

As mentioned about the code, I really like your idea of just forgetting
about buffer indices and treat everything in terms of the buffer time. I
think the buffer time (playback head) is well-defined and is independent of
the sample rates.

>

1.
2.

Define the term playhead position for buffer as an unquantized time
offset in seconds, relative to the time coordinate of the first sample
frame in the buffer. Playback rate and AudioContext sample rate are
not relevant: this offset is expressed purely in terms of the buffer's
contents and its own sample rate.
3.

Let framePosition(index) be a many-valued function yielding a set of
one or more playhead positions for the sample frame within the
buffer at index. These represent the idealized times at which the
frame would be rendered, given a start time of 0, a playback rate of 1 and
an infinite context sample rate. The function is as follows:

Let bufferTime be index / buffer.sampleRate.

If loop is false, the result is bufferTime.

If loop is true,

If bufferTime < loopStart, the result is bufferTime.

If bufferTime >= loopStart and bufferTime < loopEnd, the

sample index maps onto multiple results given by bufferTime +

(count * (loopEnd - loopStart)), where count is any non-negative

loop iteration count.

If bufferTime >= loopEnd, the position is not defined.

4.

Let frameValue(index) correspond to the vector of actual signal
values in buffer at the given index, one component per channel.
5.

Let the playback frame sequence for buffer be the set of all tuples [
framePosition(index), index, frameValue(index)], ordered by
increasing values of framePosition(index).

For an unlooped buffer, this sequence is finite (frame values are omitted
here, since they are not relevant):
framePosition(index) index
0 0
1 / buffer.sampleRate 1
2 / buffer.sampleRate 2
... ...
(length - 1) / buffer.sampleRate (length - 1)

For a looped buffer, this sequence is infinite. Let loopStartFrame be ceil(loopStart

buffer.sampleRate) (the first exact frame index within the loop body),
and loopEndFrame be ceil(loopEnd * buffer.sampleRate - 1) (the last exact
frame index within the loop body). The sequence is as follows:
framePosition(index) index
0 0
1 / buffer.sampleRate 1
2 / buffer.sampleRate 2
... ...
loopStartFrame/buffer.sampleRate loopStartFrame
... ...
loopEndFrame/buffer.sampleRate loopEndFrame
loopStartFrame/buffer.sampleRate + (loopEnd - loopStart) loopStartFrame
... ...
loopEndFrame/buffer.sampleRate + (loopEnd - loopStart) loopEndFrame
loopStartFrame/buffer.sampleRate + 2*(loopEnd - loopStart) loopStartFrame
... ... Interpolation and Buffer Optimization

1.

Let the function interpolateFrame(pos) yield a vector which
estimates the channel values at the given playhead position of pos,
which need not map onto a exact frame position in the playback sequence.
The interpolation method MUST obey all of the following constraints:

It relies ONLY on the relationship between playhead positions and
channel values supplied by the playback frame sequence.

For pos equal to framePosition(index), the result is exactly

frameValue(index).

For pos < 0, the result is a zero vector (silence).

For pos >= buffer.length / buffer.sampleRate where loop is

false, the result is a zero vector (silence).

2.

Let the operation optimize the buffer be any operation that alters
both the buffer contents and sample rate, while attempting to minimize
changes to the value of interpolateFrame(pos). The nature of the
operation is up to the UA. Examples of such operations might include
upsampling, downsampling, applying a subsample offset, or loop unrolling.

Initialization

1.

Let bufferTime be the playhead position within buffer of the next
output sample frame. Assign it the initial value -1 to indicate that the
position has not yet been determined.
2.

Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

1.

Let currentTime be the current time of the AudioContext.
2.

Let dt be 1 / (context sample rate).
3.

Let index be 0.
4.

Let computedPlaybackRate be playbackRate * pow(2, detune / 1200).
5.

Optimize the buffer during rendering, if desired.
6.

While index is less than the length of the audio block to be rendered:

If currentTime < start or currentTime >= stop, emit silence for the
output frame at index.

Else,

If bufferTime < 0, set bufferTime to offset + (currentTime -

start) * computedPlaybackRate.

Emit the result of interpolateFrame(bufferTime) as the output

frame at index.

Increase bufferTime by dt * computedPlaybackRate.

Increase index by 1.

Increase currentTime by dt.

7.

If currentTime < stop, consider playback to have ended.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-264721064,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPD0aBqqbWxXx0pfBr33_jjgAbtI4ks5rEwZNgaJpZM4A_H6-
.

--
Ray

rtoy on 7 Dec 2016

If the start time isn't on a sample boundary, currentTime - start is non-zero as expected. Why would I want the first output sample to come from very near the beginning of the buffer? If computedPlaybackRate were huge, the offset will also be huge and probably not produce a sample near the beginning of the buffer.

Note that if the playback rate is 1, then this means the first output sample is interpolated, to estimate the buffer value at delta * computedPlaybackRate which of course in this case is delta. That's your sample-accurate start behavior right there. If start = 0.99 and currentTime = 1, we want to estimate the buffer value at a time offset of 0.01.

Now assume that the playback rate is greater than 1 (and maybe a huge number). Given that at the first sample, currentTime is some quantity delta from the current buffer, don't we have to respect that delta and multiply it by the playback rate? If start = 0.99 and currentTime = 1 and computedPlaybackRate = 2, we need to estimate the buffer value at an offset of 0.02.

joeberkovitz on 8 Dec 2016

Sorry for being so dense. You are, of course, exactly right.

So, back to the original discussion. I like the code version quite a bit, but I'm not sure I want it described in terms of an AudioWorklet. The text version is also quite nice. I think either would work.

I do think the main algorithm should keep track of the buffer playback head (as a float time value) for looping, and let a bufferSample method (mostly unspecified except for what it produces) produce the correct sample based on the playback head value.

rtoy on 8 Dec 2016

Relying on the code version is OK w me - but I think we do need to say how interpolation at loop points works, and that requires a bit of narrative footwork inside the body of something like bufferSample(), presumably using comments. If we don't address what "given points" we are interpolating between, this opens the door to many possible interpretations of how subsample-accurate loop points work.

I'll think about how we could do this.

In the meantime, what do other implementors in the group think of these contrasting approaches?

joeberkovitz on 8 Dec 2016

The more I think about this, the more I think that the narrative version does a better job of describing the trickiest part of the deal, which is the way that sub-sample start, stop and loop points behave. If we moved to a code version, we'd have to somehow put this narrative into comments in order to constrain the way that UA's implement certain operations like interpolation. And if we are using code to embody a specification, it feels inappropriate to rely on comments for a key part of that specification.

joeberkovitz on 12 Dec 2016

Here is my latest revision to this algorithm based on the feedback I've received. It is the same in substance but I think it's more condensed and easier to understand.

Definitions

Let start correspond to the when parameter to start() if supplied, else 0.
Let offset correspond to the offset parameter to start() if supplied, else 0.
Let stop correspond to the sum of the when and duration parameters to start() if supplied, or the when parameter to stop(), otherwise _Infinity_.
Let buffer be the AudioBuffer employed by the node.
Let loop, loopStart and loopEnd be the loop-related attributes of the node, with the loop body clamped to the range [0, buffer.length].
Let loopStartFrame be ceil(loopStart * buffer.sampleRate) (the first exact frame index within the loop body), and loopEndFrame be ceil(loopEnd * buffer.sampleRate - 1) (the last exact frame index within the loop body).
A _playhead position_ for buffer is any quantity representing an unquantized time offset in seconds, relative to the time coordinate of the first sample frame in the buffer. Playback rate and AudioContext sample rate are not relevant: this offset is expressed purely in terms of the buffer's audio content and sample rate.
Let the function playbackSignal(position) be the _playback signal function_ for buffer, which is function that maps from a playhead position to a set of output signal values, one for each output channel. This function is only specified at a set of discrete playhead positions which correspond exactly to specific sample frames in the buffer. At all other positions, its value is determined by a UA-supplied algorithm that performs interpolation based on these well-defined values.

For an unlooped buffer, the specified values of this function are as follows (note that channels are ignored for purposes of clarity):

position | signal value
-------- | ------------
0 | channelData[0]
1 / sampleRate | channelData[1]
2 / sampleRate | channelData[2]
... | ...
(length - 1) / sampleRate | channelData[length - 1]

For a looped buffer, this sequence is infinite:

position | signal value
-------- | ------------
0 | channelData[0]
1 / sampleRate | channelData[1]
2 / sampleRate | channelData[2]
... | ...
loopStartFrame/sampleRate | channelData[loopStartFrame]
... | ...
loopEndFrame/sampleRate | channelData[loopEndFrame]
loopStartFrame/sampleRate + (loopEnd - loopStart) | channelData[loopStartFrame]
... | ...
loopEndFrame/sampleRate + (loopEnd - loopStart) | channelData[loopEndFrame]
loopStartFrame/sampleRate + (2 * (loopEnd - loopStart)) | channelData[loopStartFrame]
... | ...
loopEndFrame/sampleRate + (2 * (loopEnd - loopStart)) | channelData[loopEndFrame]
loopStartFrame/sampleRate + (3 * (loopEnd - loopStart)) | channelData[loopStartFrame]
... | ...

Buffer Optimization

Let the operation _optimize the buffer_ be any operation that alters both the buffer contents and sample rate in a way that increases the efficiency or quality of rendering, while minimizing changes to the playbackSignal() function. The nature of this operation is up to the UA. Examples of such operations might include upsampling, downsampling, applying a subsample offset, or loop unrolling.

Initialization

Let bufferTime be the _playhead position_ within buffer of the next output sample frame. Assign it the initial value -1 to indicate that the position has not yet been determined.
Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

Let currentTime be the current time of the AudioContext.
Let dt be 1 / (_context sample rate_).
Let index be 0.
Let computedPlaybackRate be _playbackRate_ * pow(2, _detune_ / 1200).
Optimize the buffer during rendering, if desired.
While index is less than the length of the audio block to be rendered:
1. If currentTime < start or currentTime >= stop, emit silence for the output frame at index.
2. Else,
  1. If bufferTime < 0, set bufferTime to offset + (currentTime - start) * computedPlaybackRate.
  2. Emit the result of playbackSignal(bufferTime) as the output frame at index.
  3. Increase bufferTime by dt * computedPlaybackRate.
3. Increase index by 1.
4. Increase currentTime by dt.
If currentTime < stop, consider playback to have ended.

joeberkovitz on 5 Jan 2017

On Thu, Jan 5, 2017 at 5:56 AM, Joe Berkovitz notifications@github.com
wrote:

Here is my latest revision to this algorithm based on the feedback I've
received. It is the same in substance but I think it's more condensed and
easier to understand.
Definitions

1.

Let start correspond to the when parameter to start() if supplied,
else 0.
2.

Let offset correspond to the offset parameter to start() if supplied,
else 0.
3.

Let stop correspond to the sum of the when and duration parameters to
start() if supplied, or the when parameter to stop(), otherwise
Infinity.
4.

Let buffer be the AudioBuffer employed by the node.
5.

Let loop, loopStart and loopEnd be the loop-related attributes of the
node, with the loop body clamped to the range [0, buffer.length].
6.

Let loopStartFrame be ceil(loopStart * buffer.sampleRate) (the first
exact frame index within the loop body), and loopEndFrame be ceil(loopEnd

buffer.sampleRate - 1) (the last exact frame index within the loop
body).

I think loopEndFrame should be floor(loopEnd*buffer.sampleRate)

>

1.
2.

A playhead position for buffer is any quantity representing an
unquantized time offset in seconds, relative to the time coordinate of the
first sample frame in the buffer. Playback rate and AudioContext
sample rate are not relevant: this offset is expressed purely in terms of
the buffer's audio content and sample rate.
3.

Let the function playbackSignal(position) be the playback signal
function for buffer, which is function that maps from a playhead
position to a set of output signal values, one for each output channel.
This function is only specified at a set of discrete playhead positions
which correspond exactly to specific sample frames in the buffer. At all
other positions, its value is determined by a UA-supplied algorithm that
performs interpolation based on these well-defined values.

For an unlooped buffer, the specified values of this function are as
follows (note that channels are ignored for purposes of clarity):
position signal value
0 channelData[0]
1 / sampleRate channelData[1]
2 / sampleRate channelData[2]
... ...
(length - 1) / sampleRate channelData[length - 1]

If this is going to be part of the spec, you need to say this is only
true if playbackRate = 1. Also for sub-sample start, I'm not quite sure
this is correct either.

For a looped buffer, this sequence is infinite:
position signal value
0 channelData[0]
1 / sampleRate channelData[1]
2 / sampleRate channelData[2]
... ...
loopStartFrame/sampleRate channelData[loopStartFrame]
... ...
loopEndFrame/sampleRate channelData[loopEndFrame]
loopStartFrame/sampleRate + (loopEnd - loopStart)
channelData[loopStartFrame]
... ...
loopEndFrame/sampleRate + (loopEnd - loopStart) channelData[loopEndFrame]
loopStartFrame/sampleRate + (2 * (loopEnd - loopStart))
channelData[loopStartFrame]
... ...
loopEndFrame/sampleRate + (2 * (loopEnd - loopStart))
channelData[loopEndFrame]
loopStartFrame/sampleRate + (3 * (loopEnd - loopStart))
channelData[loopStartFrame]
... ...

I'm not sure this is correct for sub-sample looping. I need to think about
this a bit more.

Buffer Optimization

Let the operation optimize the buffer be any operation that
alters both the buffer contents and sample rate in a way that increases the
efficiency or quality of rendering, while minimizing changes to the
playbackSignal() function. The nature of this operation is up to the
UA. Examples of such operations might include upsampling, downsampling,
applying a subsample offset, or loop unrolling.

Initialization

1.

Let bufferTime be the playhead position within buffer of the next
output sample frame. Assign it the initial value -1 to indicate that the
position has not yet been determined.
2.

Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

1.

Let currentTime be the current time of the AudioContext.
2.

Let dt be 1 / (context sample rate).
3.

Let index be 0.
4.

Let computedPlaybackRate be playbackRate * pow(2, detune / 1200).
5.

Optimize the buffer during rendering, if desired.

I don't think we need to say anything about optimizing. I think this is
all implied by the playbackSignal function. It can do anything it wants so
long as it provides the correct value. We probably do want to give some
constraints on what playbackSignal does when the buffer rate and context
rate are the same and the the start time is on a sample boundary. In that
case playbackSignal should produce exactly the samples in the buffer.

bufferTime not initialized.

1.
2.

While index is less than the length of the audio block to be rendered:

If currentTime < start or currentTime >= stop, emit silence for the
output frame at index.

Else,

If bufferTime < 0, set bufferTime to offset + (currentTime -

start) * computedPlaybackRate.

Emit the result of playbackSignal(bufferTime) as the output

frame at index.

Increase bufferTime by dt * computedPlaybackRate.

Increase index by 1.

Increase currentTime by dt.

3.

If currentTime < stop, consider playback to have ended.

I think you meant currentTime >= stop.

This also seems to be missing the looping points. Assuming playbackRate

0, we can say something like

if loop == true && bufferTime > loopEnd then
bufferTime = loopStart

I think by doing this, we get sub-sample looping. We don't really need
loopStartFrame or loopEndFrame.

I still need to write an example of sub-sample accurate start for ABSN like
I did for Oscillators. Probably need one for ConstantSource when there are
automations; it doesn't matter otherwise for a constant source.

--
Ray

rtoy on 6 Jan 2017

I think loopEndFrame should be floor(loopEnd*buffer.sampleRate)

Hmmm... I still don't think so. _Recall that loopEnd is exclusive of the loop body_.

Let's pretend that sampleRate is 1000 Hz, for clarity, and that we have a 10-sample buffer (of duration 0.010 seconds). Take loopStart as 0 (although this is not relevant).

Take loopEnd as 0.010. The loop clearly includes the entire buffer (indices 0 through 0). floor(loopEnd*buffer.sampleRate) will be floor(10) or 10, which can't be the right answer: there is no frame with index 10. ceil(loopEnd*sampleRate - 1) will be ceil(9) which is 9.

Or take loopEnd as 0.0095. The loop still includes buffer indices 0 through 9, although its duration is less than 0.010. floor(loopEnd*buffer.sampleRate) will be floor(9.5) or 9. ceil(loopEnd*sampleRate - 1) will be ceil(8.5) which is also 9.

If this is going to be part of the spec, you need to say this is only
true if playbackRate = 1. Also for sub-sample start, I'm not quite sure
this is correct either.

I have already defined the concept of "playback position" as independent of playback rate. The accounting for computed playback rate occurs at the end of this mini-spec, and happens prior to interpolating this function.

I don't think we need to say anything about optimizing. I think this is
all implied by the playbackSignal function. It can do anything it wants so
long as it provides the correct value.

The optimization does have an effect on where frames fall within the buffer, so I think it's important to say that it can happen at various prescribed points, because this can result in interpolation results differing during the course of playback.

We probably do want to give some
constraints on what playbackSignal does when the buffer rate and context
rate are the same and the the start time is on a sample boundary. In that
case playbackSignal should produce exactly the samples in the buffer.

I agree that we need to say this, although I think it already follows from the definitions given.

If currentTime < stop, consider playback to have ended.
I think you meant currentTime >= stop.

Yes I did! Thanks.

This also seems to be missing the looping points. Assuming playbackRate
0, we can say something like
if loop == true && bufferTime > loopEnd then
bufferTime = loopStart

I think by doing this, we get sub-sample looping. We don't really need
loopStartFrame or loopEndFrame.

I disagree. This approach of "wrapping around" the buffer time that you suggested, caused all kinds of spec problems in the past. This requires a lot of gymnastics for playback rates that can go zero or negative, and recall that first iteration of the loop is preceded by the "prefix" between bufferTime == 0 and loopStart, which makes running time backwards non-trivial if you just keep wrapping bufferTime around. The current approach takes care of all that, because bufferTime always increases or decreases continuously as per the computed playback rate without jumps.

Also, we still very much need the definition of loop start/end frames in order to prescribe what data points are being interpolated by playbackSignal(). Without these definitions, it becomes very difficult to say what the buffer's contents actually _mean_ to the interpolation algorithm.

At least, that's my current 2-cent opinion :-)

joeberkovitz on 9 Jan 2017

Appreciate all of your comments and especially your effort on this.

I think I've confused myself many times, so can we start over with a simple
example so we can agree on what should happen.

Let's assume a context and source with a sample rate of 1 Hz to keep things
simple. Let the source have a buffer that has 10 samples, with values 0,
1, 2,...,9.

Let loopStart = 3.25, loopEnd = 7.5 (arbitrarily chosen).

For this case, source.start(0) and currentTime = 0. Let out[t] be the
output value at time t.

Then out[0]=0, out[1]=1,out[2]=2, and so on up to out[7] = 7. Since
loopEnd is 7.5, out[8] can't be 8 because that would be past the end of the
loop. To get the output for time 8, we need to go back to loopStart.

The question here is what the value of out[8] should be. Should out[8] =
buffer[loopStart]? Since loopStart is not on a sampling boundary, do we
interpolate (somehow) and say that out[8] = buffer[3.5] = 3.5? If so, the
output would then be buffer[3.5], buffer[4.5], buffer[6.5], buffer[7.5] and
loop back to 3.5 again. But what do we actually output buffer[7.5] since
loopEnd = 7.5?

Or should out[8] = buffer[4] = 4? Then we continue with buffer[5],
buffer[6], buffer[7] and then go back to loopStart since 8 > loopEnd.

I think if we can answer these questions we'll have an appropriate
algorithm that we should be able to extend easily to arbitrary source start
time and playback rate.

On Mon, Jan 9, 2017 at 9:30 AM, Joe Berkovitz notifications@github.com
wrote:

I think loopEndFrame should be floor(loopEnd*buffer.sampleRate)

Hmmm... I still don't think so. Recall that loopEnd is exclusive of the
loop body.

Let's pretend that sampleRate is 1000 Hz, for clarity, and that we have a
10-sample buffer (of duration 0.010 seconds). Take loopStart as 0
(although this is not relevant).

Take loopEnd as 0.010. The loop clearly includes the entire buffer
(indices 0 through 0). floor(loopEndbuffer.sampleRate) will be floor(10)
or 10, which can't be the right answer: there is no frame with index 10. ceil(loopEndsampleRate

1) will be ceil(9) which is 9.

Or take loopEnd as 0.0095. The loop still includes buffer indices 0
through 9, although its duration is less than 0.010. floor(loopEndbuffer.
sampleRate) will be floor(9.5) or 9. ceil(loopEndsampleRate - 1) will
be ceil(8.5) which is also 9.

If this is going to be part of the spec, you need to say this is only
true if playbackRate = 1. Also for sub-sample start, I'm not quite sure
this is correct either.

I have already defined the concept of "playback position" as independent
of playback rate. The accounting for computed playback rate occurs at the
end of this mini-spec, and happens prior to interpolating this function.

I don't think we need to say anything about optimizing. I think this is
all implied by the playbackSignal function. It can do anything it wants so
long as it provides the correct value.

The optimization does have an effect on where frames fall within the
buffer, so I think it's important to say that it can happen at various
prescribed points, because this can result in interpolation results
differing during the course of playback.

We probably do want to give some
constraints on what playbackSignal does when the buffer rate and context
rate are the same and the the start time is on a sample boundary. In that
case playbackSignal should produce exactly the samples in the buffer.

I agree that we need to say this, although I think it already follows from
the definitions given.

If currentTime < stop, consider playback to have ended.

I think you meant currentTime >= stop.

Yes I did! Thanks.

This also seems to be missing the looping points. Assuming playbackRate
0, we can say something like
if loop == true && bufferTime > loopEnd then
bufferTime = loopStart

I think by doing this, we get sub-sample looping. We don't really need
loopStartFrame or loopEndFrame.

I disagree. This approach of "wrapping around" the buffer time that you
suggested, caused all kinds of spec problems in the past. This requires a
lot of gymnastics for playback rates that can go zero or negative, and
recall that first iteration of the loop is preceded by the "prefix" between
bufferTime == 0 and loopStart, which makes running time backwards
non-trivial if you just keep wrapping bufferTime around. The current
approach takes care of all that, because bufferTime always increases
monotonically.

Also, we still very much need the definition of loop start/end frames in
order to prescribe what data points are being interpolated by
playbackSignal(). Without these definitions, it becomes very difficult to
say what the buffer's contents actually mean to the interpolation
algorithm.

At least, that's my current 2-cent opinion :-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271348935,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPGyAcVMw4G_vQqSiIDvW7tGOeDDNks5rQm61gaJpZM4A_H6-
.

--
Ray

rtoy on 9 Jan 2017

Let's assume a context and source with a sample rate of 1 Hz to keep things
simple. Let the source have a buffer that has 10 samples, with values 0,
1, 2,...,9.

Let loopStart = 3.25, loopEnd = 7.5 (arbitrarily chosen).

I so love examples!!! Please read on...

For this case, source.start(0) and currentTime = 0. Let out[t] be the
output value at time t.

Then out[0]=0, out[1]=1,out[2]=2, and so on up to out[7] = 7. Since
loopEnd is 7.5, out[8] can't be 8 because that would be past the end of the
loop. To get the output for time 8, we need to go back to loopStart.

The question here is what the value of out[8] should be. Should out[8] =
buffer[loopStart]? Since loopStart is not on a sampling boundary, do we
interpolate (somehow) and say that out[8] = buffer[3.5] = 3.5? If so, the
output would then be buffer[3.5], buffer[4.5], buffer[6.5], buffer[7.5] and
loop back to 3.5 again. But what do we actually output buffer[7.5] since
loopEnd = 7.5?

Or should out[8] = buffer[4] = 4? Then we continue with buffer[5],
buffer[6], buffer[7] and then go back to loopStart since 8 > loopEnd.

I think if we can answer these questions we'll have an appropriate
algorithm that we should be able to extend easily to arbitrary source start
time and playback rate.

This is a great question and I believe that the above spec language actually does answer it. In fact this is the exact variety of question that led me to the proposed approach. Let me walk through the answer in detail.

First, let me use this example to fill out the values of the table I speced, that maps from playbackPosition to signalValue -- the second table, which handles the case of loops. I will leave out all of the sampleRate divisors since sampleRate is 1 in this world. Note that in this case the signal value _is_ the frame index in our imaginary world (i.e. the signal value at channelData[N] is N).

Let loopStartFrame = floor(loopStart) = ceil(3.25) = 4
Let loopEndFrame = ceil(loopEnd-1) = ceil(7.5 - 1) = 7
Note: loopEnd - loopStart = 7.5 - 3.25 = 4.25

position expression | actual position | signal value
--------------------| --------------- | ------------
0 | 0 | 0
1 | 1 | 1
2 | 2 | 2
3 | 3 | 3
loopStartFrame | 4 | 4
... | ... | ...
loopEndFrame | 7 | 7
loopStartFrame + (loopEnd - loopStart) | 8.25 | 4
... | ... | ...
loopEndFrame + (loopEnd - loopStart) | 11.25 | 7
loopStartFrame + (2 * (loopEnd - loopStart)) | 12.5 | 4
... | ... | ...

The position value of 8.25 might seem surprising you, but here's the rationale (which is encoded in the formula loopStartFrame + loopEnd - loopStart): the end of the loop is at pos=7.5 (0.5 _after_ the last loop sample at pos=7), and it wraps back to the start of the loop at 3.25 (0.75 _before_ the first loop sample at pos=4). So after [7] we have a time interval of 0.5 to the loop wraparound point, and then another time interval of 0.75 to [4], the first sample in the loop body. That takes us to pos=8.25.

So... what is the value of out[8]? (or, playbackSignal(8) in spec language)?

Let's assume that the UA is using linear interpolation. We have two adjacent data points in the neighborhood of 8 that give the signal value for exact sample frame positions: out[7] = 7, and out[8.25] = 4. The answer is therefore 7 + ((4 - 7) * (8 - 7) / (8.25 - 7)), which comes to 4.6. Makes sense: 8 is almost (but not quite) at the position which would yield exactly 4.

Doing the exact same interpolation at the positions of 7 and 8.25 gives the expected signal values of 7 and 4 respectively.

joeberkovitz on 9 Jan 2017

@rtoy By the way I noticed later that you used two different values in your example: it begins with loopStart = 3.25, but then later you use the value loopStart = 3.5.

Let's look at this other case of loopStart = 3.5, loopEnd = 7.5 because it's also instructive (and maybe it's what you meant all along). In this case, the loop spans an exact number of samples and so it yields a simpler result.

All of the formulas I gave before apply, but the results are different. The table looks like this:

Let loopStartFrame = floor(loopStart) = ceil(3.5) = 4
Let loopEndFrame = ceil(loopEnd-1) = ceil(7.5 - 1) = 7
Note: loopEnd - loopStart = 7.5 - 3.5 = 4

position expression | actual position | signal value
--------------------| --------------- | ------------
0 | 0 | 0
1 | 1 | 1
2 | 2 | 2
3 | 3 | 3
loopStartFrame | 4 | 4
... | ... | ...
loopEndFrame | 7 | 7
loopStartFrame + (loopEnd - loopStart) | 8 | 4
... | ... | ...
loopEndFrame + (loopEnd - loopStart) | 11 | 7
loopStartFrame + (2 * (loopEnd - loopStart)) | 12 | 4
... | ... | ...

So the value of out[8] is simply 4. No interpolation required.

In fact, this result is what you get whenever 3 < loopStart <= 4 and loopEnd = loopStart + 4. Which makes sense: when the loop encompasses an exact number of sample frames, it doesn't matter exactly where it starts and ends so long as the same sample frames are included.

joeberkovitz on 9 Jan 2017

On Mon, Jan 9, 2017 at 11:44 AM, Joe Berkovitz notifications@github.com
wrote:

Let's assume a context and source with a sample rate of 1 Hz to keep things
simple. Let the source have a buffer that has 10 samples, with values 0,
1, 2,...,9.

Let loopStart = 3.25, loopEnd = 7.5 (arbitrarily chosen).

I so love examples!!! Please read on...

For this case, source.start(0) and currentTime = 0. Let out[t] be the
output value at time t.

Then out[0]=0, out[1]=1,out[2]=2, and so on up to out[7] = 7. Since
loopEnd is 7.5, out[8] can't be 8 because that would be past the end of the
loop. To get the output for time 8, we need to go back to loopStart.

The question here is what the value of out[8] should be. Should out[8] =
buffer[loopStart]? Since loopStart is not on a sampling boundary, do we
interpolate (somehow) and say that out[8] = buffer[3.5] = 3.5? If so, the
output would then be buffer[3.5], buffer[4.5], buffer[6.5], buffer[7.5] and
loop back to 3.5 again. But what do we actually output buffer[7.5] since
loopEnd = 7.5?

Or should out[8] = buffer[4] = 4? Then we continue with buffer[5],
buffer[6], buffer[7] and then go back to loopStart since 8 > loopEnd.

I think if we can answer these questions we'll have an appropriate
algorithm that we should be able to extend easily to arbitrary source start
time and playback rate.

This is a great question and I believe that the above spec language
actually does answer it. In fact this is the exact variety of question that
led me to the proposed approach. Let me walk through the answer in detail.

First, let me use this example to fill out the values of the table I
speced, that maps from playbackPosition to signalValue -- the second table,
which handles the case of loops. I will leave out all of the sampleRate
divisors since sampleRate is 1 in this world. Note that in this case the
signal value is the frame index in our imaginary world (i.e. the signal
value at channelData[N] is N).

Let loopStartFrame = floor(loopStart) = ceil(3.25) = 4
Let loopEndFrame = ceil(loopEnd-1) = ceil(7.5 - 1) = 7
Note: loopEnd - loopStart = 7.5 - 3.25 = 4.25
position expression actual position signal value
0 0 0
1 1 1
2 2 2
3 3 3
loopStartFrame 4 4
... ... ...
loopEndFrame 7 7
loopStartFrame + (loopEnd - loopStart) 8.25 4
... ... ...
loopEndFrame + (loopEnd - loopStart) 11.25 7
loopStartFrame + (2 * (loopEnd - loopStart)) 12.5 4
... ... ...

The position value of 8.25 might seem surprising you, but here's the
rationale (which is encoded in the formula loopStartFrame + loopEnd -
loopStart): the end of the loop is at pos=7.5 (0.5 after the last loop
sample at pos=7), and it wraps back to the start of the loop at 3.25 (0.75
before the first loop sample at pos=4). So after [7] we have a time
interval of 0.5 to the loop wraparound point, and then another time
interval of 0.75 to [4], the first sample in the loop body. That takes us
to pos=8.25.

I understand the math, but I don't understand why the value should be
8.25. if 8.25 is the actual position at which to sample the buffer, then
why is the output value 4 instead of something closer to 8.25? Are you
implicitly wrapping the position back to loopStartFrame? That would make
sense to me.

I can also understand that since the last position is 7, we need to go a
time step of 1 to get the next value. This would mean 0.5 to get to 7.5
for loopEnd, and then we need to go another 0.5, but this time from
loopStart. That would put us at position 3.75 (3.25 + 0.5). I can then
see we want to output the value of the buffer at time 3.75, which would be
3.75 (assuming linear interpolation).

I just don't know how you get to 8.25 that also produces a value of 4. The
buffer has 10 samples after all for times 0 to 9.

Note that I have not looked at what Chrome does.

So... what is the value of out[8]? (or, playbackSignal(8) in spec
language)?

Let's assume that the UA is using linear interpolation. We have two
adjacent data points in the neighborhood of 8 that give the signal value
for exact sample frame positions: out[7] = 7, and out[8.25] = 4. The answer
is therefore 7 + ((4 - 7) * (8 - 7) / (8.25 - 7)), which comes to 4.6.
Makes sense: 8 is almost (but not quite) at the position which would yield
exactly 4.

Doing the exact same interpolation at the positions of 7 and 8.25 gives
the expected signal values of 7 and 4 respectively.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271387217,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPPqbxieZZTmOYI0PoYPVCkZbSEALks5rQo4ugaJpZM4A_H6-
.

--
Ray

rtoy on 10 Jan 2017

Here's the derivation. And yes, wrapping is involved. Playback positions are not indices into the buffer; they are time offsets from the first frame of the buffer. So wrapping has to be applied.

At playback position 8.25, we are 0.75 past the loop end point of 7.5. Because of loop wrapping, this is supposed to play back a signal value that is 0.75 past the loop _start_ point of 3.25 (i.e., 4). Thus, 8.25 corresponds to a wrapped playback position of 4, at which there is an exact, non-interpolated sample frame of 4.

At playback position 8 (your original question), the position has not progressed all the way to 8.25. Hence the result of 4.6, which is 80% of the way between the interpolated values 7 (last frame in the loop, at position 7) and 4 (first frame in the loop, at position 8.25).

In this proposed model of loop wrapping, loop wraparound points only interpolate using sample frames that lie inside the loop body -- in other words, the frames 3 and 8 are not in the picture. The rationale if they were, we would get glitches at the wraparound points due to abruptly switching from an interpolation function between frames 7 and 8 to an interpolation function between 3 and 4. That is why loopStartFrame and loopEndFrame are central to this approach.

Imagine a slow playback rate in this example (say, 0.01). We'll see this behavior at a wraparound point:

pos | value
--- | -----
7 | 7
7.01 | 6.9625
... | ...
7.49 | 5.8375
7.5 | 5.8
7.51 | 5.7625
... | ...
8.24 | 4.0375
8.25 | 4

This output has a continuous linear ramp as it progresses through the wraparound point.

If we did not carefully define start and end frames, an implementation might make use of sample frames 3 and 8 to interpolate at wraparound points. That could lead to glitchy output like this:

pos | value
--- | -----
7 | 7
7.01 | 7.01
... | ...
7.49 | 7.49
7.5 | 3.25 (sudden discontinuity at loop boundary)
7.51 | 3.26
... | ...
8.24 | 3.99
8.25 | 4

Of course, this example is made somewhat artificial by the fact that the buffer contains a linear ramp function which is being looped. In reality the loop points would probably connect values that were very close to each other.

joeberkovitz on 10 Jan 2017

On Tue, Jan 10, 2017 at 6:25 AM, Joe Berkovitz notifications@github.com
wrote:

Here's the derivation. And yes, wrapping is involved. Playback positions
are not indices into the buffer; they are time offsets from the first frame
of the buffer. So wrapping has to be applied.

At playback position 8.25, we are 0.75 past the loop end point of 7.5.
Because of loop wrapping, this is supposed to play back a signal value that
is 0.75 past the loop start point of 3.25 (i.e., 4). Thus, 8.25
corresponds to a wrapped playback position of 4, at which there is an
exact, non-interpolated sample frame of 4.

At playback position 8 (your original question), the position has not
progressed all the way to 8.25. Hence the result of 4.6, which is 80% of
the way between the interpolated values 7 (last frame in the loop, at
position 7) and 4 (first frame in the loop, at position 8.25).

In this proposed model of loop wrapping, loop wraparound points only
interpolate using sample frames that lie inside the loop body -- in other
words, the frames 3 and 8 are not in the picture. The rationale if they
were, we would get glitches at the wraparound points due to abruptly
switching from an interpolation function between frames 7 and 8 to an
interpolation function between 3 and 4. That is why loopStartFrame and
loopEndFrame are central to this approach.

Imagine a slow playback rate in this example (say, 0.01). We'll see this
behavior at a wraparound point:
pos value
7 7
7.01 6.9625
... ...
7.49 5.8375
7.5 5.8
7.51 5.9625
... ...
8.24 4.0375
8.25 4

This output has a continuous linear ramp as it progresses through the
wraparound point.

This doesn't fit my mental model of looping. :-(

If we did not carefully define start and end frames, an implementation
might make use of sample frames 3 and 8 to interpolate at wraparound
points. That could lead to glitchy output like this:
pos value
7 7
7.01 7.01
... ...
7.49 7.49
7.5 3.25 (sudden discontinuity at loop boundary)
7.51 3.26
... ...
8.24 3.99
8.25 4

Of course, this example is made somewhat artificial by the fact that the
buffer contains a linear ramp function which is being looped. In reality
the loop points would probably connect values that were very close to each
other.

This fits my mental model of looping. The way I've been imagining it is
that the buffer section between times 3.25 and 7.5 get tacked on (repeating
"forever") to the buffer after time 7.5. So I would expect the output to
continue to increase to time 7.5-eps and at time 7.5 suddenly jump back to
3.25, as you show above.

I actually expect and want a glitch at that time because the two pieces
aren't continuous.

Perhaps my mental model is wrong. I don't have experience with how other
software does this kind of looping.

—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271588155,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPFiS-9Zv2SU-47lBziybxlZULB9Xks5rQ5TOgaJpZM4A_H6-
.

--
Ray

rtoy on 10 Jan 2017

@rtoy I'm very happy because this means I've succeeded in describing a model sufficiently accurately that someone can disagree with it!

It sounds like you like the second model that I described better. I saw the argument for it too, which is why I presented it as an alternative. But there is a problem with it that bothers me: what if the entire buffer is looped, or (more generally) if there are no sample frames at or after loopEnd? In this case, one needs to somehow interpolate from the last frame in the loop back to the first frame in the loop, as in the approach I've been pushing. However, maybe that's a special case that we can handle by appending a "fake frame" to the end of the buffer that causes the interpolation to behave consistently. For example, if you are looping an entire buffer, this "fake frame" would be the same as the first frame in the buffer.

I will post a second version of the algorithm that approaches things from this angle, to see if that works for you and for others.

joeberkovitz on 10 Jan 2017

On Tue, Jan 10, 2017 at 12:44 PM, Joe Berkovitz notifications@github.com
wrote:

@rtoy https://github.com/rtoy I'm very happy because this means I've
succeeded in describing a model sufficiently accurately that someone can
disagree with it!

Heh.

It sounds like you like the second model that I described better. I saw
the argument for it too, which is why I presented it as an alternative. But
there is a problem with it that bothers me: what if the entire buffer is
looped, or (more generally) if there are no sample frames at or after
loopEnd? In this case, one needs to somehow interpolate from the last
frame in the loop back to the first frame in the loop, as in the approach
I've been pushing. However, maybe that's a special case that we can handle
by appending a "fake frame" to the end of the buffer that causes the
interpolation to behave consistently. For example, if you are looping an
entire buffer, this "fake frame" would be the same as the first frame in
the buffer.

I'll have to think about that.

I will post a second version of the algorithm that approaches things from
this angle, to see if that works for you and for others.

Before you do that, can we agree on the model for looping? Is it really
just concatenating the looped sections together to produce a new effective
buffer? If this is the appropriate model, I think it's fairly
straightforward to describe. (Well, I guess it's complicated if the loop
points aren't on buffer sample boundaries, but we'll just have to pick
something.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271692593,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPFUn_SMWMXq4Ae0i_cwn6cekgJVNks5rQ-3DgaJpZM4A_H6-
.

--
Ray

rtoy on 10 Jan 2017

Here is a description that just concatenates the looped sections. It does use the "fake frame" as a device to make the description work in cases where the loop wrap point doesn't have a sample frame in the buffer.

Other than the fake frame, this description seems simpler. Note that wrapping no longer takes place inside playbackSignal: instead, it's done in the main playback loop. The means of doing so (repeated subtraction) is not efficient but it simplifies the description a lot.

Definitions

Let start correspond to the when parameter to start() if supplied, else 0.
Let offset correspond to the offset parameter to start() if supplied, else 0.
Let stop correspond to the sum of the when and duration parameters to start() if supplied, or the when parameter to stop(), otherwise _Infinity_.
Let buffer be the AudioBuffer employed by the node.
Let bufferRate be the sample rate employed by buffer.
Let bufferLength be the number of frames in buffer.
Let loop, loopStart and loopEnd be the loop-related attributes of the node, with the loop body clamped to the range [0, buffer.length].
A _playhead position_ for buffer is any quantity representing an unquantized time offset in seconds, relative to the time coordinate of the first sample frame in the buffer. Playback rate and AudioContext sample rate are not relevant: this offset is expressed purely in terms of the buffer's audio content and bufferRate.
Let the function playbackSignal(position) be the _playback signal function_ for buffer, which is function that maps from a playhead position to a set of output signal values, one for each output channel. If position corresponds to the location of an exact sample frame in the buffer, this function returns that frame. Otherwise, its return value is determined by a UA-supplied algorithm that interpolates between sample frames in the neighborhood of position.
Let loopEndFrame be ceil(loopEnd * bufferRate - 1) (the last exact frame index within the loop body).
If loopEndFrame >= bufferLength - 1, there are no sample frames following the loop. In this case, to ensure consistent interpolation at the loop point, create a single sample frame after the loop using these steps:
1. Clone the contents of buffer and take this clone as the new value of buffer.
2. Append a single sample frame to buffer, whose signal values are given by playbackSignal(((loopEndFrame + 1) / bufferRate) - (loopEnd - loopStart)). This frame corresponds to the interpolated signal value at a position exactly one sample frame after the last sample frame in the loop, taking wrapping into account. Note that for loops that start on an exact sample frame position, this simplifies to playbackSignal(loopStart).

Buffer Optimization

Let the operation _optimize the buffer_ be any operation that alters both the buffer contents and sample rate in a way that increases the efficiency or quality of rendering, while minimizing changes to the playbackSignal() function. The nature of this operation is up to the UA. Examples of such operations might include upsampling, downsampling, applying a subsample offset, or loop unrolling.

Initialization

Let bufferTime equal -1. This is the _playhead position_ within buffer of the next output sample frame. The initial value -1 indicates that the position has not yet been determined.
Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

Let currentTime be the current time of the AudioContext.
Let dt be 1 / (_context sample rate_).
Let index be 0.
Let computedPlaybackRate be _playbackRate_ * pow(2, _detune_ / 1200).
Optimize the buffer during rendering, if desired.
While index is less than the length of the audio block to be rendered:
1. If currentTime < start or currentTime >= stop, emit silence for the output frame at index.
2. Else,
  1. If bufferTime < 0, set bufferTime to offset + (currentTime - start) * computedPlaybackRate.
  2. Let effectiveBufferTime equal bufferTime.
  3. While loop is true and effectiveBufferTime >= loopEnd,
    1. Let effectiveBufferTime equal effectiveBufferTime - (loopEnd - loopStart).
  4. Emit the result of playbackSignal(effectiveBufferTime) as the output frame at index.
  5. Increase bufferTime by dt * computedPlaybackRate.
3. Increase index by 1.
4. Increase currentTime by dt.
If currentTime >= stop, consider playback to have ended.

joeberkovitz on 10 Jan 2017

On Tue, Jan 10, 2017 at 1:16 PM, Joe Berkovitz notifications@github.com
wrote:

Here is a description that just concatenates the looped sections. It does
use the "fake frame" as a device to make the description work in cases
where the loop wrap point doesn't have a sample frame following it in the
buffer.

Other than the fake frame, this description seems simpler. Note that
wrapping no longer takes place inside playbackSignal: instead, it's done
in the main playback loop. The means of doing so (repeated subtraction) is
not efficient but it simplifies the description a lot.
Definitions

1.

Let start correspond to the when parameter to start() if supplied,
else 0.
2.

Let offset correspond to the offset parameter to start() if supplied,
else 0.
3.

Let stop correspond to the sum of the when and duration parameters to
start() if supplied, or the when parameter to stop(), otherwise
Infinity.
4.

Let buffer be the AudioBuffer employed by the node.
5.

Let bufferRate be the sample rate employed by buffer.
6.

Let bufferLength be the number of frames in buffer.
7.

Let loop, loopStart and loopEnd be the loop-related attributes of the
node, with the loop body clamped to the range [0, buffer.length].
8.

A playhead position for buffer is any quantity representing an
unquantized time offset in seconds, relative to the time coordinate of the
first sample frame in the buffer. Playback rate and AudioContext
sample rate are not relevant: this offset is expressed purely in terms of
the buffer's audio content and bufferRate.
9.

Let the function playbackSignal(position) be the playback signal
function for buffer, which is function that maps from a playhead
position to a set of output signal values, one for each output channel. If
position corresponds to the location of an exact sample frame in the
buffer, this function returns that frame. Otherwise, its return value is
determined by a UA-supplied algorithm that interpolates between sample
frames in the neighborhood of position.
10.

Let loopEndFrame be ceil(loopEnd * bufferRate - 1) (the last exact
frame index within the loop body).
11.

If loopEndFrame >= bufferLength - 1, there are no sample frames
following the loop. In this case, to ensure consistent interpolation at the
loop point, create a single sample frame after the loop using these steps:
1.
  Clone the contents of buffer and take this clone as the new value
  of buffer.
  2.

  Append a single sample frame to buffer, whose signal values are
  given by playbackSignal(((loopEndFrame + 1) / bufferRate) -
  (loopEnd - loopStart)). This frame corresponds to the interpolated
  signal value at a position exactly one sample frame after the last sample
  frame in the loop, taking wrapping into account. Note that for loops that
  start on an exact sample frame position, this simplifies to
  playbackSignal(loopStart).
The spec says loopEnd is constrained to the range (-inf,
buffer.duration]. So loopEndFrame is always inside the buffer. I think.

>

1.
1.

Buffer Optimization

Let the operation optimize the buffer be any operation that
alters both the buffer contents and sample rate in a way that increases the
efficiency or quality of rendering, while minimizing changes to the
playbackSignal() function. The nature of this operation is up to the
UA. Examples of such operations might include upsampling, downsampling,
applying a subsample offset, or loop unrolling.

Initialization

1.

Let bufferTime equal -1. This is the playhead position within buffer
of the next output sample frame. The initial value -1 indicates that the
position has not yet been determined.
2.

Optimize the buffer prior to rendering, if desired.

Rendering a Block of Audio

1.

Let currentTime be the current time of the AudioContext.
2.

Let dt be 1 / (context sample rate).
3.

Let index be 0.
4.

Let computedPlaybackRate be playbackRate * pow(2, detune / 1200).
5.

Optimize the buffer during rendering, if desired.
6.

While index is less than the length of the audio block to be rendered:

If currentTime < start or currentTime >= stop, emit silence for the
output frame at index.

Else,

If bufferTime < 0, set bufferTime to offset + (currentTime -

start) * computedPlaybackRate.

Let effectiveBufferTime equal bufferTime.

While loop is true and effectiveBufferTime >= loopEnd,

Let effectiveBufferTime equal effectiveBufferTime -

(loopEnd - loopStart).

Emit the result of playbackSignal(effectiveBufferTime) as the

output frame at index.

Increase bufferTime by dt * computedPlaybackRate.

Increase index by 1.

Increase currentTime by dt.

7.

If currentTime >= stop, consider playback to have ended.

Based on this algorithm and the example, when currentTime = 8, we would
take the buffer sample at time 3.75, which is what I would expect. I think
the this algorithm is correct as is.

There are some details to be worked out. For example, It seems to me that
we could just initialize bufferTime to offset + computedRate * (ceil(start)

start). I think we don't need effectiveBufferTime. We can just operate on
bufferTime directly.

I don't quite understand the need for index. It just seems to be a
counter and step 6 references it, but step 7 seems to subsume everything
index is intended to do.

On reading the spec for loopStart and loopEnd, I think we're also missing
some details in this algorithm, but no show stoppers.

—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271700389,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPADcIw-3xOlpgXPyQOwUyAYG63O8ks5rQ_UngaJpZM4A_H6-
.

--
Ray

rtoy on 12 Jan 2017

Great. Glad this seems better. I feel good about it too. I'm away this week but I will fix it up further next week in response to your comments.

2.

Append a single sample frame to buffer, whose signal values are
given by playbackSignal(((loopEndFrame + 1) / bufferRate) -
(loopEnd - loopStart)). This frame corresponds to the interpolated
signal value at a position exactly one sample frame after the last sample
frame in the loop, taking wrapping into account. Note that for loops that
start on an exact sample frame position, this simplifies to
playbackSignal(loopStart).

The spec says loopEnd is constrained to the range (-inf,
buffer.duration]. So loopEndFrame is always inside the buffer. I think.

Yes it is; the check isn't to see if it's inside the buffer, but to see if it is the last frame in the buffer. In that case the argument to playbackSignal may be greater than loopEndFrame but less than loopEnd. Appending the fake frame allows interpolation in this tiny sub-frame "tail" to the buffer to wrap around and be well-behaved.

There are some details to be worked out. For example, It seems to me that
we could just initialize bufferTime to offset + computedRate * (ceil(start)

start). I think we don't need effectiveBufferTime. We can just operate on
bufferTime directly.

effectiveBufferTime is concerned with folding loop iterations not with start time. One could move the folding inside pkaybackSignal() which might be clearer.

I don't quite understand the need for index. It just seems to be a
counter and step 6 references it, but step 7 seems to subsume everything
index is intended to do.

On reading the spec for loopStart and loopEnd, I think we're also missing
some details in this algorithm, but no show stoppers.

Great. Thanks so much for the continued feedback!

joeberkovitz on 12 Jan 2017

On Thu, Jan 12, 2017 at 6:33 AM, Joe Berkovitz notifications@github.com
wrote:

Great. Glad this seems better. I feel good about it too. I'm away this
week but I will fix it up further next week in response to your comments.

2.

Append a single sample frame to buffer, whose signal values are
given by playbackSignal(((loopEndFrame + 1) / bufferRate) -
(loopEnd - loopStart)). This frame corresponds to the interpolated
signal value at a position exactly one sample frame after the last
sample
frame in the loop, taking wrapping into account. Note that for loops
that
start on an exact sample frame position, this simplifies to
playbackSignal(loopStart).

The spec says loopEnd is constrained to the range (-inf,
buffer.duration]. So loopEndFrame is always inside the buffer. I think.

Yes it is; the check isn't to see if it's inside the buffer, but to see if
it is the last frame in the buffer. In that case the argument to
playbackSignal may be greater than loopEndFrame but less than loopEnd.
Appending the fake frame allows interpolation in this tiny sub-frame "tail"
to the buffer to wrap around and be well-behaved.

I understand now. As always, an example is best. Assume a sample rate of
1 and the buffer has 2 samples in it. The duration is 2 sec, and loopEnd
can be up to 2. If the playback rate is low, we'll want to take values at
times 1.90, 1.91, 1.92 and so on, but that's past the last value in the
buffer.

I am content to let the browser handle this in some appropriate way
without saying how.

>
>

There are some details to be worked out. For example, It seems to me that
we could just initialize bufferTime to offset + computedRate *
(ceil(start)

start). I think we don't need effectiveBufferTime. We can just operate
on
bufferTime directly.

effectiveBufferTime is concerned with folding loop iterations not with
start time. One could move the folding inside pkaybackSignal() which might
be clearer.

I meant to say that bufferTime would be spec'ed to wrap around instead of
introducing an effectiveBufferTime. So actually, this would be the same as
adding step 4.5 bufferTime = effectiveBufferTime.

I like playbackSignal(t) to be only concerned about producing an
appropriate interpolated sample from the buffer at time t.

It's up to the rest of the algorithm to make sure t is sensible for the
buffer.

>
>

I don't quite understand the need for index. It just seems to be a
counter and step 6 references it, but step 7 seems to subsume everything
index is intended to do.

On reading the spec for loopStart and loopEnd, I think we're also missing
some details in this algorithm, but no show stoppers.

Great. Thanks so much for the continued feedback!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-272177786,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAofPOsDVuXhTVnpC6o9F9eSsS56ttEMks5rRjmjgaJpZM4A_H6-
.

--
Ray

rtoy on 12 Jan 2017

Appending the fake frame allows interpolation in this tiny sub-frame "tail"
to the buffer to wrap around and be well-behaved.

I understand now. As always, an example is best. Assume a sample rate of
1 and the buffer has 2 samples in it. The duration is 2 sec, and loopEnd
can be up to 2. If the playback rate is low, we'll want to take values at
times 1.90, 1.91, 1.92 and so on, but that's past the last value in the
buffer.

I am content to let the browser handle this in some appropriate way
without saying how.

I don't think any of this spec language really says how it all happens, it just defines the observable behavior that is "appropriate". For loops that are aligned with the sample frames like this latest example, it's easier to get away with fuzzy language. But with your earlier example, you highlighted just how a well-intentioned developer (me in this case!) could come up with an interpretation that made no sense to you. So I am concerned that we not leave this example too far open to interpretation.

To my thinking the "fake tail frame" is in no way an implementation strategy, just the simplest way I could find to define how the interpolation should work, in a way that falls back on a simple notion of interpolating between regularly spaced sample frames (even with arbitrary sub-sample loop points). But maybe it's too specific for your taste, even as spec language.

Can you or anyone else provide other language to use in its place? Otherwise I propose to keep this and move ahead for lack of a proposed alternative.

I meant to say that bufferTime would be spec'ed to wrap around instead of
introducing an effectiveBufferTime. So actually, this would be the same as
adding step 4.5 bufferTime = effectiveBufferTime.

I like playbackSignal(t) to be only concerned about producing an
appropriate interpolated sample from the buffer at time t.

It's up to the rest of the algorithm to make sure t is sensible for the
buffer.

Sure, so we can keep the folding outside the call to playbackSignal(t) for better isolation of concerns.

But I believe simply wrapping bufferTime doesn't work because it discards state about which loop iteration one is on. bufferTime encodes this state. Consider a playback rate that changes from, say, playing forward to playing backward. Let's take a sample rate 1 buffer with 3 samples, loopStart = 1, loopEnd = 3.

Let's say playbackRate is 0.01. If time just runs forward, we get a sequence of times like this: 0, 0.01, ... 0.99, 1, 1.01 ... 2.99, 1, 1.01 ... 2.99...

Let's say that at this point (2 iterations have occurred) the computed playback rate now becomes -0.01. We should now get exactly the above sequence in reverse:

2.99, 2.98... 1.01, 1, 2.99, 2.98... 1.01, 1, 0.99... 0.01, 0

This is the reason I speced it as I did: I think if you play a looped buffer backwards, it should behave exactly as if you played an "unrolled" version of the same buffer backwards (with no loop points and a lot of repeated sample frames). Does this make sense to you, or have we identified another area where the behavior isn't correct from your point of view? Do you think we _should_ ignore loop iterations?

joeberkovitz on 17 Jan 2017

This is getting very complex. Even implementations are not that complex. Surely there is a simpler design ?

padenot on 20 Jan 2017

@padenot Please make sure you are looking at https://github.com/WebAudio/web-audio-api/issues/95#issuecomment-271700389 for review purposes.

I would always seek a simpler design wherever possible, and the one referenced in the above link is in fact simplified from a few earlier iterations. I believe it is now about the same length as your original proposal, and it addresses a number of gaps that you did not originally consider (the behavior of sub-sample-rate loop points in particular) so I am a bit puzzled by the critique of "very complex". And perhaps some implementations are not doing everything they should be (sub-sample loop points, again).

At this point, could you please identify specific problems or changes that you would like to see addressed, or else provide a new draft for consideration? At present, having worked through all this with Ray, I don't see a simpler design, although the definition of interpolation at loop point boundaries could be made shorter by being made more vague.

joeberkovitz on 20 Jan 2017

Perhaps things would be easier to review and comment on if we had a document or wiki page that has the algorithm all in one place where we can comment on and perhaps add pull requests. I find it hard to keep it all straight using comments here.

rtoy on 20 Jan 2017

I agree. I think it might be time for a PR, if only to use for markup in this way. I will put one together in the next day or two.

joeberkovitz on 23 Jan 2017

I have just updated the PR referenced above with a full set of illustrations supplementing the ABSN definition. These could use some editorial work but I think they make the behavior of the algorithm much clearer.

joeberkovitz on 7 Feb 2017

Web-audio-api: (AudioBufferSourceNodePlaybackRate): AudioBufferSourceNode.playbackRate not strictly defined

All 65 comments

Definitions

Interpolation and Buffer Optimization

Initialization

Rendering a Block of Audio

Definitions

Buffer Optimization

Initialization

Rendering a Block of Audio

Definitions

Buffer Optimization

Initialization

Rendering a Block of Audio

Related issues