Vexflow: Input events

[harrislapiroff]

Is there any plan to support events (such as hover, click, and tap events) within VexFlow? It could be useful for building music composition apps or music theory games!

(Also, VexFlow is incredible. Really amazing work. 👍)

[gristow]

This is already possible, if you're using SVGContext:

• When a staveNote is drawn, the staveNote.elem property contains the SVG element, to which you can attach event handlers as you normally would to a DOM element.
• For other objects, you can request a "group" element be created by doing something like:

var elem = ctx.openGroup(className, idName);
crescendo.draw();
ctx.closeGroup();
elem.addEventListener("click", function(){
  console.log("Clickity clack.");
});

There are some limitations:

• Events are triggered only when they occur over a painted pixel. So, if a user clicks in the very middle of a whole note, this won't register. If the user wants to click the stem of a note, they've got to be quite exact about it. If you want to catch events based on where you are in relation to stave lines, this isn't terribly helpful.
• It only works with SVGContext.

So, the solution I use is to:

• Disable all pointer events on the rendered SVG (or canvas... or Raphael... I use SVG). svg.style.pointerEvents = 'none';
• Add an empty, floated div sized & positioned directly on top of the SVG.
• Catch touch & pointer events on this div, and write a routine to translate them into the SVG coordinates
• Use the various getBoundingBox routines to create clickable areas, and check intersections with each touch & point event.

Others may have different (better?) approaches.

You can see a running version of this I've done at http://bit.ly/note-draw

If you want to implement a somewhat full-fledged editor function, you'll probably need to go this route (since editing involves catching events on things that have not yet been created). If you just want a user to be able to, say, click on the correct note, the built-in group functionality should suffice.

[harrislapiroff]

Thanks for the info! Very useful example.

[mscuthbert]

Could we leave this issue open either (1) changing it to a Documentation feature. @gristow 's info is too important not to preserve elsewhere (and it'll otherwise just keep coming up). @harrislapiroff -- could you change the title to "Document user interaction with Vexflow" or something like that? _OR_ (2) use it as a feature request to make it so that Gristow's last approach (solution) is easier for most people to implement?

For catching pointer events on a staff (such as adding and moving notes), I use a similar method to @gristow though I have been attaching the pointer events to a div (w/o border or padding) surrounding the SVG/Canvas (I have been using Canvas, but will switch to SVG soon). See music21j's stream.js addEditableCanvas() routine for more details. It is definitely possible, but I think it's within the scope of Vexflow to make this easier.

[harrislapiroff]

I could definitely see it being useful for VexFlow to facilitate/abstract out the logic in @gristow or your solution.

[0xfe]

VexFlow already has some elements instrumented to accept events, e.g., StaveNote. You can directly get the element with note.getElem(), or you can access them via the DOM using the vf-stavenote class. You can also decide if you want to include the stem or modifiers/accidentals in your events via the vf-stem or vf-modifiers classes.

I will happily accept PRs to instrument more VexFlow elements like this. It's quite straightforward -- the hardest part is coming up with decent class names.

[0xfe]

Also, yes, in the least documenting this (and @gristow's bounding box trick) is a good idea. :-)

[gristow]

A few thoughts: The hardest part of implementing this "bounding box trick" is converting co-ordinates in client/UI event space to those used within VexFlow. (This is relatively trivial if you're using a fixed-width svg or canvas; I typically have mine scaling responsively with the browser width.)

Would it make sense to have an addEventListener method on the various context classes which would:

• add a listener of that name on the canvas/svg
• catch the native UI events, and translate the co-ordinates on these events into VexFlow space
• call a callback function with the translated co-ordinates

Or at a minimum, we might add a clientCoordinatesToVexCoordinates method. (Users would still have to figure out how to go about implementing the tricky SVG overlay on their own in this case...)

As I remember (and I wrote the code for this about 13 months ago), the gotchas I encountered, that others might appreciate knowing about before trying this at home, were:

• ctx.scale must be taken into consideration
• we have to account for scrolling by subtracting the canvas/svg's top left corner from all touch & pointer event co-ordinates. This can be attained with getBoundingClientRect()'s top and left properties.
• if the SVG style is set to something responsive (e.g., svg.style.width="100%"), on every event you must also detect the actual rendered width of the SVG at that moment using svg.getBoundingClientRect() to determine if scaling has occurred within the browser outside of ctx.scale
• For SVG, you have to float a div over the SVG. Listening to events on the SVG won't work, because only painted pixels register events. Listening to events on the div containing the SVG also won't work, because if you click on a painted pixel in the SVG, the SVG will register the event, and the SVG's parent DIV won't receive it.

Getting all of this right was easily several days work for me, so it makes me think there might be some use in including this within VexFlow. It's not all that many lines of code. (On the other hand, I'm a musician, not a DOM expert -- this might be a breeze for others.)

[0xfe]

Actually, have you tried pointer-events: bounding-box on the element? Seems like it takes care of everything (I think): https://www.w3.org/TR/SVG2/interact.html#PointerEventsProp

[0xfe]

Looks like I added support for pointer-events: bounding-box to openGroup at some point. It think it worked at the time, but can't remember.

stavenote.js:

ctx.openGroup('flag', null, { pointerBBox: true });
Glyph.renderGlyph(ctx, flagX, flagY, glyph_font_scale, flagCode);
ctx.closeGroup();

[mscuthbert]

I believe that we've dropped support for IE <= 10, right? That's the last major browser not to support pointer-events. This would've been the reason not to publicize this in the past (Thanks @gristow for the difference between Canvas and SVG on what captures the events)

[mscuthbert]

Oh wait I was looking at the CSS/HTML support table; there doesn't seem to be a support table for SVG. (However, turning off pointer-events on the

[gristow]

This is an old thread -- but I've streamlined how I handle VexFlow user interaction for the exercises at uTheory.com. I have a generalized method I think is good which we might consider bringing into the VexFlow codebase.

• Listen for events on the SVG itself (setting svg.style.pointerEvents = "bounding-box" on it)
• If desired, also listen on elements within the svg (which can propagate up or not as desired).
• Convert coordinates in client/browser space to SVG space using an SVGPoint element and matrixTransform. (See this answer on SO.)
• Track low-level browser events and abstract them into five higher-level events: touchStart, touchEnd, drag, hover, mouseOut.

In my case, I do this by extending the SVGContext class directly (we use a slightly customized version of VexFlow). But a generalized version of this, which SVGContext & CanvasContext could extend would be very easy to do.

Do we think this is of use? Here's the full code of how I do it:

/* global document window */
import SVGContext from './SVG';

const isDragSymbol = Symbol('dragging');
const isOutSymbol = Symbol('isOut');
const windowListeners = Symbol('windowListeners');

/**
 * A class to enable touch & mouse interactions on SVGs. Translates browser coordinates to
 * SVG coordinates.
 *
 * Call SVGInteraction.makeInteractive(svg), then override this class's methods to use.
 */
export default class SVGInteraction extends SVGContext {
  /* eslint-disable no-unused-vars */
  // These are here as holders -- override whichever you need when you inherit this class.
  touchStart(e, coords) {}
  touchEnd(e, coords) {}
  drag(e, coords) {}
  hover(e, coords) {}
  mouseOut(e, coords) {}
  /* eslint-enable no-unused-vars */

  makeInteractive(svg = this.svg) {
    // We will add listeners to the SVG bounding box itself:
    svg.style.pointerEvents = 'bounding-box';
    // An SVG point is used to translate div space to SVG space. See Alexander Jank's solution at:
    // https://stackoverflow.com/questions/29261304/how-to-get-the-click-coordinates-relative-to-svg-element-holding-the-onclick-lis
    this.svgPt = svg.createSVGPoint();

    /** ==== Condition Testing & Setting Functions ====
     *
     * These functions are used to translate UI events from the browser into more helpful
     * events from a programatic perspective. They take care of tracking mousestate &
     * touchstate (down or up) to fire drag vs. hover. If a user drags outside of the SVG,
     * they continue to fire the drag event, so that (if desired) things like drag & drop
     * among elements can be implemented.
     *
     */

    // A  not-combinator
    const not = condition => () => !condition();
    // Get whether we're dragging
    const isDragging = () => this[isDragSymbol];
    // Set whether the mouse is down or up.
    const down = () => { this[isDragSymbol] = true; return true; };
    const up = () => { this[isDragSymbol] = false; return true; };

    // Get whether we're outside of the SVG bounds
    const isOutside = () => this[isOutSymbol];
    // Set whether we're in our out; if we move out while dragging we need window listeners briefly.
    const inside = () => {
      this[isOutSymbol] = false;
      this[windowListeners].forEach(([eventType, listener]) => {
        window.removeEventListener(eventType, listener, false);
      });
      return true;
    };
    const outside = () => {
      this[isOutSymbol] = true;
      this[windowListeners].forEach(([eventType, listener]) => {
        window.addEventListener(eventType, listener);
      });
      return true;
    };

    // We'll hold the window listeners here so we can add & remove them as needed.
    this[windowListeners] = [];

    // Utility function to check event conditions & fire class events if needed
    const addListener = ([eventType, callback, ifTrue, el]) => {
      el = el || svg;
      const listener = (evt) => {
        if (ifTrue()) {
          const coords = getCoords(evt, svg, this.svgPt, this);
          callback.call(this, evt, coords);
        }
      };

      if (el !== window) el.addEventListener(eventType, listener);
      else this[windowListeners].push([eventType, listener]);
    };

      // [ type, listener, testFunction (fire listener if true), EventTarget (default this.svg)]
    [
      /* events occuring within SVG */
      ['mousedown', this.touchStart, down], // Touch is started
      ['touchstart', this.touchStart, down], // Touch is started
      ['mouseup', this.touchEnd, up], // Touch ends or mouse up
      ['touchend', this.touchEnd, up], // Touch ends or mouse up
      ['touchmove', this.drag, isDragging], // Dragging
      ['mousemove', this.drag, isDragging], // Dragging
      ['mousemove', this.hover, not(isDragging)], // Hover
      ['mouseout', this.mouseOut, not(isDragging)], // Mouseout
      ['touchcancel', this.mouseOut, not(isDragging)], // Mouseout (touch interrupt)
      ['mouseout', () => {}, () => isDragging() && outside()], // goes out of bounds isDown & set
      ['touchcancel', () => {}, () => isDragging() && outside()], // goes out of bounds isDown & set
      ['touchmove', inside, isOutside], // comes back inside
      ['mousemove', inside, isOutside], // comes back inside
      ['mousedown', inside, isOutside], // comes back inside, this shouldn't happen, but just in case
      ['touchstart', inside, isOutside], // comes back inside, this shouldn't happen, but just in case
      /* out of bounds events */
      ['touchmove', this.drag, () => isOutside() && isDragging(), window], // if outside in window & dragging
      ['mousemove', this.drag, () => isOutside() && isDragging(), window], // if outside in window & dragging
      ['mouseup', this.touchEnd, () => isOutside() && isDragging() && up() && inside(), window], // if outside in window & dragging & released
      ['touchend', this.touchEnd, () => isOutside() && isDragging() && up() && inside(), window], // if outside in window & dragging & released
    ]
    .forEach(addListener);
  }
}

/**
 * Returns coordinates for an event
 * @param {Event} e touch or mouse event
 * @param {SVGPoint} svgPt an SVG point
 * @param {SVGElement} svg the SVG's client bounding rectangle
 *
 * @returns {Object} { x, y, [touches] }
 */
function getCoords(e, svg, svgPt) {
  if ('touches' in e) {
    const touches = e.touches.map(touch => getCoords(touch, svg, svgPt));
    return { x: touches[0].x, y: touches[0].y, touches };
  }

  svgPt.x = e.clientX;
  svgPt.y = e.clientY;
  const svgCoords = svgPt.matrixTransform(svg.getScreenCTM().inverse());
  return { x: svgCoords.x, y: svgCoords.y };
}

[0xfe]

This looks pretty good, Gregory. Mind setting up a quick jsfiddle with an
example to play with?

On Fri, Jun 9, 2017 at 1:01 PM, Gregory Ristow [e-mail]
wrote:

This is an old thread -- but I've streamlined how I handle VexFlow user
interaction for the exercises at uTheory.com https://utheory.com. I
have a generalized method I think is good which we might consider bringing
into the VexFlow codebase.

• Listen for events on the SVG itself (setting svg.style.pointerEvents
  = "bounding-box" on it)
• If desired, also listen on elements within the svg (which can
  propagate up or not as desired).
• Convert coordinates in client/browser space to SVG space using an
  SVGPoint https://developer.mozilla.org/en-US/docs/Web/API/SVGPoint
  element and matrixTransform. (See this answer on SO
  https://developer.mozilla.org/en-US/docs/Web/API/SVGPoint.)
• Track low-level browser events and abstract them into five
  higher-level events: touchStart, touchEnd, drag, hover, mouseOut.

In my case, I do this by extending the SVGContext class directly (we use
a slightly customized version of VexFlow). But a generalized version of
this, which SVGContext & CanvasContext could extend would be very easy to
do.

Do we think this is of use? Here's the full code of how I do it:

/* global document window /import SVGContext from './SVG';
const isDragSymbol = Symbol('dragging');const isOutSymbol = Symbol('isOut');const windowListeners = Symbol('windowListeners');
/* * A class to enable touch & mouse interactions on SVGs. Translates browser coordinates to * SVG coordinates. * * Call SVGInteraction.makeInteractive(svg), then override this class's methods to use. /export default class SVGInteraction extends SVGContext {
/ eslint-disable no-unused-vars /
// These are here as holders -- override whichever you need when you inherit this class.
touchStart(e, coords) {}
touchEnd(e, coords) {}
drag(e, coords) {}
hover(e, coords) {}
mouseOut(e, coords) {}
/ eslint-enable no-unused-vars */

makeInteractive(svg = this.svg) {
// We will add listeners to the SVG bounding box itself:
svg.style.pointerEvents = 'bounding-box';
// An SVG point is used to translate div space to SVG space. See Alexander Jank's solution at:
// https://stackoverflow.com/questions/29261304/how-to-get-the-click-coordinates-relative-to-svg-element-holding-the-onclick-lis
this.svgPt = svg.createSVGPoint();

/** ==== Condition Testing & Setting Functions ====     *     * These functions are used to translate UI events from the browser into more helpful     * events from a programatic perspective. They take care of tracking mousestate &     * touchstate (down or up) to fire drag vs. hover. If a user drags outside of the SVG,     * they continue to fire the drag event, so that (if desired) things like drag & drop     * among elements can be implemented.     *     */

// A  not-combinator
const not = condition => () => !condition();
// Get whether we're dragging
const isDragging = () => this[isDragSymbol];
// Set whether the mouse is down or up.
const down = () => { this[isDragSymbol] = true; return true; };
const up = () => { this[isDragSymbol] = false; return true; };

// Get whether we're outside of the SVG bounds
const isOutside = () => this[isOutSymbol];
// Set whether we're in our out; if we move out while dragging we need window listeners briefly.
const inside = () => {
  this[isOutSymbol] = false;
  this[windowListeners].forEach(([eventType, listener]) => {
    window.removeEventListener(eventType, listener, false);
  });
  return true;
};
const outside = () => {
  this[isOutSymbol] = true;
  this[windowListeners].forEach(([eventType, listener]) => {
    window.addEventListener(eventType, listener);
  });
  return true;
};

// We'll hold the window listeners here so we can add & remove them as needed.
this[windowListeners] = [];

// Utility function to check event conditions & fire class events if needed
const addListener = ([eventType, callback, ifTrue, el]) => {
  el = el || svg;
  const listener = (evt) => {
    if (ifTrue()) {
      const coords = getCoords(evt, svg, this.svgPt, this);
      callback.call(this, evt, coords);
    }
  };

  if (el !== window) el.addEventListener(eventType, listener);
  else this[windowListeners].push([eventType, listener]);
};

  // [ type, listener, testFunction (fire listener if true), EventTarget (default this.svg)]
[
  /* events occuring within SVG */
  ['mousedown', this.touchStart, down], // Touch is started
  ['touchstart', this.touchStart, down], // Touch is started
  ['mouseup', this.touchEnd, up], // Touch ends or mouse up
  ['touchend', this.touchEnd, up], // Touch ends or mouse up
  ['touchmove', this.drag, isDragging], // Dragging
  ['mousemove', this.drag, isDragging], // Dragging
  ['mousemove', this.hover, not(isDragging)], // Hover
  ['mouseout', this.mouseOut, not(isDragging)], // Mouseout
  ['touchcancel', this.mouseOut, not(isDragging)], // Mouseout (touch interrupt)
  ['mouseout', () => {}, () => isDragging() && outside()], // goes out of bounds isDown & set
  ['touchcancel', () => {}, () => isDragging() && outside()], // goes out of bounds isDown & set
  ['touchmove', inside, isOutside], // comes back inside
  ['mousemove', inside, isOutside], // comes back inside
  ['mousedown', inside, isOutside], // comes back inside, this shouldn't happen, but just in case
  ['touchstart', inside, isOutside], // comes back inside, this shouldn't happen, but just in case
  /* out of bounds events */
  ['touchmove', this.drag, () => isOutside() && isDragging(), window], // if outside in window & dragging
  ['mousemove', this.drag, () => isOutside() && isDragging(), window], // if outside in window & dragging
  ['mouseup', this.touchEnd, () => isOutside() && isDragging() && up() && inside(), window], // if outside in window & dragging & released
  ['touchend', this.touchEnd, () => isOutside() && isDragging() && up() && inside(), window], // if outside in window & dragging & released
]
.forEach(addListener);

}
}
/** * Returns coordinates for an event * @param {Event} e touch or mouse event * @param {number} height the internal SVG height * @param {SVGPoint} svgPt an SVG point * @param {SVGElement} svg the SVG's client bounding rectangle */function getCoords(e, svg, svgPt) {
if ('touches' in e) {
const touches = e.touches.map(touch => getCoords(touch, svg, svgPt));
return { x: touches[0].x, y: touches[0].y, touches };
}

svgPt.x = e.clientX;
svgPt.y = e.clientY;
const svgCoords = svgPt.matrixTransform(svg.getScreenCTM().inverse());
return { x: svgCoords.x, y: svgCoords.y };
}

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/0xfe/vexflow/issues/371#issuecomment-307444191, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAOukzSzY9j9s2y7f2ymEJOrpx64uiomks5sCXpygaJpZM4JAh7M
.

--
Mohit Muthanna [mohit (at) muthanna (uhuh) com]

[gristow]

Sure -- here's the gist of it. Not my prettiest code, it was a lot to squeeze into a fiddle!

SVG Interaction Strawman Fiddle

[gloddream]

Sure -- here's the gist of it. Not my prettiest code, it was a lot to squeeze into a fiddle!

SVG Interaction Strawman Fiddle

when i add a barnote, get the errors:
write-SVGInteraction.js:54 Uncaught TypeError: Cannot read property 'style' of null
note.attrs.el is null, when the node is barnote......
@gristow