Norns: dropped OSC messages

[catfact]

we appear to be dropping OSC messages from lua->sc with some number of multiple messages attempted "simultanously"

[antonhornquist]

out of interest - how many messages did you use when you recreated this?

[catfact]

i'm able to consistently get it with 8

[antonhornquist]

how many of those are dropped? does number of dropped msgs vary?

[catfact]

as here:
https://github.com/catfact/dust/blob/elca-sines/lua/emb/elca-test.lua#L127

[catfact]

it seems to vary

[artfwo]

i wonder if this issue reproduces in a pristine environment, i.e. oscsend or some tiny script that sends OSC on one side and OSCResponder on the sclang side.

[antonhornquist]

i guess we can also test simply sending osc messages from sclang to sclang.

[ngwese]

if one is trying to confirm that drops are happening (or have gone away after making a change) the contents of /proc/net/udp might be of use. on my device i see:

  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops             
   59: 00000000:277F 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 6890 2 b8cfd180 0                  
  256: 00000000:0044 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 9748 2 b9776300 0                  
  256: 00000000:0044 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5000 2 b8d0b480 0                  
  340: 00000000:EC98 00000000:0000 07 00000000:00000000 00:00000000 00000000   108        0 4766 2 b8d0a300 0                  
  372: 00000000:22B8 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 6901 2 b8cff100 983                
  421: 00000000:14E9 00000000:0000 07 00000000:00000000 00:00000000 00000000   108        0 4764 2 b8d09f80 0                  
  476: 00000000:DF20 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 6889 2 b8cfd500 565                

...and if i find the pid for say sclang (503 currently on my device) then it is possible to figure out which of the above entries correspond to it using lsof -iUDP -a:

COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sclang  503   pi   13u  IPv4   6889      0t0  UDP *:57120 
matron  504   pi   13u  IPv4   6890      0t0  UDP *:10111 
matron  504   pi   19u  IPv4   6901      0t0  UDP *:8888 

note that the DEVICE value in lsof (6889) corresponds to the inode value on the last line of /proc/net/udp

[catfact]

apologies... can't actually reproduce anymore, starting from something minimal.

i'm using an engine with 64 sine synths, and a script that sets hz on some number of them simultaneously in a (fast or slow) sequnce.

on sc side i always get all N messages at the predicted times and, whats the word... monotonically or whatever.

so i will chalk this up to user error or something weird with the network status. guess i was jumping to conclusions about drops/lags

[antonhornquist]

i think we can keep this open for a while.

i run gong at 4 voice polyphony where it seems to successfulty send 214+ messages to crone.

(every fmthing osc takes 32 params, every pole filter module takes 21 params, then there are a couple more params to set up)

at 6 voice polyphony gong sends over 320 messages to crone, and it appears to be that all do not get over.

i'm using params:bang() after setting up all params. one question would be what number of messages is reasonable to support? i guess one way to address this would be to introduce microdelays in between sending messages in params:bang(). this has tradeoffs of course.

another thing to look into would be to some way of bundling osc events (this might also help lag issues, but i havent gotten to the bottom of this)

[antonhornquist]

also - i'm not sure it's possible to bundle somethnig like 320 events. there are udp packet size limits

[artfwo]

bundles and microdelays are not the answer. bundles will tremendously complicate the logic for sending engine commands (if we decide to build them automatically) or add more monkey business for scripts developers (if done manually).

[catfact]

well ok still seems smart to build something that minimally tests for missing messages
1) sc->sc
2) liblo->sc

assuming there is some limit on "simultaneous" messages, it seems like this implementation+usage of paramset won't work. :bang() updates every param in the set whether it needs updating or not. a simple bandaid would be to maintain a dirty flag in addition to value for each param in the set?

[antonhornquist]

imo update of a large number of params causing loads of osc events (via paramset:bang() or similar) has to work in some way: ie. it's needed when whole new "presets" are loaded.

bundles are transparently handled using anonymous functions in sc (ie s.makeBundle) . not sure if something like this is possible in lua. paramset:bang() should imo be possible to bundle without interfering with user scripts since this is a core norns class.

then again, user scripts may need this for macro controls and similar which potentially could produce a lot of osc messages. of course i'd like the current engine.param(whatever) approach sending atomic messages. a pragmatic alternative would be engine.param(whaterver) caching events lua side followed by a engine.update() or something (we do have screen.update() already, that would be something similar). a bit awkward but perhaps it's needed?

[catfact]

no, i agree - it has to work. but needs to be implemented around limitations of infrastructure; that is, param:bang() needs some kind of workaround

this could mean - on each bang, each N params to update just need to be batched, one way or another... not ideal and better to fix the pipes if possible

but first things i want to find a really minimal test case, see if it is same on norns as other systems

[antonhornquist]

i'll try and see if it's possible to reproduce sc->sc

[antonhornquist]

also: imo, providing additional support in lua for bundling of messages is a good thing in itself. it is included in the osc standard for a reason :)

[catfact]

i can reproduce with a very stripped down case of liblo + sc, on ubuntu + thinkpad. will try norns in a moment.

SC:

~pings = List.new;

~def_start = OSCdef.newMatching(\test_start, {
    |msg, time, addr, recvPort|
    ~pings.clear;
}, '/test/start');

~def_ping = OSCdef.newMatching(\test_ping, {
    |msg, time, addr, recvPort|
    [msg, time, addr, recvPort].postln;
    ~pings.add(msg[1]);
}, '/test/ping');

~def_end = OSCdef.newMatching(\test_end, {
    |msg, time, addr, recvPort|
    postln("pings received: " ++ ~pings.size);
}, '/test/end');

C:

#include <stdio.h>
#include <unistd.h>
#include <lo/lo.h>

int count = 400;

int main(int argc, char *argv[]) {

  lo_address t = lo_address_new("127.0.0.1", "57120");
  lo_send(t, "/test/start", NULL);
  for(int i=0; i<count; ++i) {
    lo_send(t, "/test/ping", "i", i);
  }

  // this sleep is necessary...
  usleep(20000);
  lo_send(t, "/test/end", "");

  return 0;
}

the number varies, but pings stop coming through after 280-300.

[catfact]

pretty much same thing going SC->SC, but bizarrely requires a _longer_ delay (?)

Routine {
n = NetAddr.localAddr;
n.sendMsg('/test/start');
    400.do({ |i|
        n.sendMsg('/test/ping', i);
    });
    0.01.wait;
    n.sendMsg('/test/end');
}.play;

[antonhornquist]

certainly seems to be a sc issue then, when using lots of atomic osc messages

[antonhornquist]

(whether this is a sc issue could i guess be verified by doing a liblio->liblio test case)

btw: for certainty, shouldn't there be a sleep after the '/test/start' message aswell? not sure '/test/start' is guaranteed to arrive prior to '/test/ping' messages?

[artfwo]

we can start by increasing UDP buffer size here: https://github.com/supercollider/supercollider/blob/develop/lang/LangPrimSource/SC_ComPort.cpp#L91

[catfact]

the last test is liblo->liblo. this raises the ceiling substantially, but the issue persists; i can get about 1800 pings through.

#include <stdio.h>
#include <unistd.h>
#include <lo/lo.h>

 int pings;
int done = 0;

 int start_handler(const char* path, const char* types, lo_arg ** argv,
           int argc, void *data, void *user_data) {
   pings = 0;
 }

 int ping_handler(const char* path, const char* types, lo_arg ** argv,
          int argc, void *data, void *user_data) {
   pings += 1;
 }

 int end_handler(const char* path, const char* types, lo_arg ** argv,
         int argc, void *data, void *user_data) {
   fprintf(stderr, "received %d pings \n", pings);
   done = 1;
 }

void error(int num, const char *m, const char *path) {
  fprintf(stderr, "error creating server thread\n");
}

int main(int argc, char *argv[]) {

  lo_server_thread st = lo_server_thread_new("57120", error);

  lo_server_thread_add_method(st, "/test/start", NULL, start_handler, NULL);
  lo_server_thread_add_method(st, "/test/ping", NULL, ping_handler, NULL);
  lo_server_thread_add_method(st, "/test/end", NULL, end_handler, NULL);

  lo_server_thread_start(st);    
  while(!done) { ;; } 
  lo_server_thread_free(st);

  return 0;
}

[catfact]

@antonhornquist i guess you're right, though in practice i get the first N messages reliably in order until some buffer overflows or whatever it is.

i did add the delay as you suggest though - no change that i can see.

[artfwo]

udp send and receive buffer sizes have documented system-specific limits (131071 for the latter it seems). it's possible to decrease them, but increasing above that requires root privileges. this also applies to bundles of course.

i don't think we should to reduce the number dropped messages to zero, some reasonable rate should suffice.

[tehn]

stumbled upon this while working on mlr, which spews ~300 OSC packets at softcut as setup. (i can likely optimize this, but just to have a number).

i was seeing failure when doing a new bootup of matron. the issue was, cleanup was getting run twice, so softcut.reset was getting run twice also, which itself shoots out 175ish packets.

my problem was resolved by fixing the double-cleanup, but i was seeing crone receive about 400 packets during the double-init... which means about 250 were getting dropped. (which is why i noticed mlr not working right).

we might consider moving crone to IPC to open up some OSC bandwidth? or just make the buffer bigger and be ok with timing slop? these massive sends i really only expect for initialization stuff...

side-option specific to softcut... i could implement a single reset command and do all the default vals inside crone

[catfact]

Hard coded reset is easy pickings and solves immediate problem

Another simple thing would be making more cryptic abbreviations for OSC paths

Switching transport layer is actually pretty PITA at this point but of course doable given time. Would try to keep OSC as option in that case for crone compatibility with other environments.

[tehn]

keeping crone OSC friendly makes a ton of sense, i neglected to consider that.

shortening OSC commands where possible is good idea. yesterday @antonhornquist suggested/command -> /cmd

would softcut -> sc be acceptable?

i'll work on a crone-based sc reset.

[ranch-verdin]

Would be great if any IPC transport would provide file descriptor and work with poll and select, for event loop paradigm.

According to a quick bit of googling this morning POSIX IPC or nanomsg would support that, but sysV IPC wouldn't.