Image-sequencer: explore acceleration w WebAssembly
WebAssembly seems like a way we might speed up pixel-by-pixel operations.
https://developer.mozilla.org/en-US/docs/WebAssembly
But would it suffer from the same issues as webworkers, i.e. a cost of passing the image data over?
jywarren
All 51 comments
Also note this way of writing WebAssembly:
jywarren
on 11 Oct 2018
GitMate.io thinks the contributor most likely able to help you is @ccpandhare.
A possibly related issue is https://github.com/publiclab/image-sequencer/issues/237 (Explore OpenCV.js module).
![gitmate[bot] picture](https://avatars.githubusercontent.com/in/9010?v=4&s=40)
gitmate[bot]
on 11 Oct 2018
@jywarren Even I was thinking we can write the pixel manipulation code in c and create a js binding to it, is that the same as web assembly? I mean the code shows up as native code.
tech4GT
on 11 Oct 2018
Interesting -- well, the WebAssembly approach means it'd be runnable at (near?) native speed in the browser, too! I think it could get really fast -- maybe not GPU speed, but maybe closer to video speed. For a speed reference we'd like to get closer to, see the Infragram WebGL demo: https://publiclab.github.io/infragram/
jywarren
on 1 Nov 2018
Walt seems like a good way to easily write webassembly code!
Provide a thin layer of syntax sugar on top of .wat text format. Preferably porting as much of JavaScript syntax to WebAssembly as possible. This improved syntax should give direct control over the WebAssembly output. Meaning there should be minimal to none post optimization to be done to the wast code generated. The re-use of JavaScript semantics is intentional as I do not wish to create a brand new language.
jywarren
on 1 Nov 2018
Also noting imagemagick has been compiled for webassembly... Wow!
https://github.com/KnicKnic/WASM-ImageMagick/blob/master/Readme.md
jywarren
on 8 Nov 2018
I think we could write a version of Pixelmanipulation following this example!
https://github.com/ballercat/walt/wiki/Walt-In-5-Minutes
...but would we be able to run a JavaScript function inside? I'm not sure about that. Would the function have to be compiled as well?
jywarren
on 8 Nov 2018
Docs for reading an image into webassembly: https://developer.mozilla.org/en-US/docs/WebAssembly/existing_C_to_wasm
jywarren
on 8 Nov 2018
This is a great area for testing code snippets!
jywarren
on 8 Nov 2018
Just asked over at the Walt project if their compiler runs in the browser. Otherwise we could look at precompiling these at build time.
jywarren
on 11 Nov 2018
So I think if the simple pixel manipulation function we pass into the PixelManipulation module is close enough to Walt syntax, which is actually WebAssembly, but "looks like" javascript... so if the inside of a function like this could be compiled almost directly, that'd be interesting:
By comparison, here is a section of walt code:
for (i = 0; i < total; i += 1) {
particle = i * particleByteSize;
particle = {
x: width / 2.0,
y: height / 2.0,
vx: ((i * 7 % 200) - 100) / 100.0,
vy: ((i * 19 % 200) - 100) / 100.0
};
}
const x: i32 = 2;
export function echo(): i32 {
const x: i32 = 42;
return x;
}
Walt folks responded to say that walt does indeed compile in the browser, on the fly! Awesome.
jywarren
on 13 Nov 2018
I was reading the above discussions and I think we can implement pixel manipulation code in c and then compile it into wasm file and then add it into the project. After that we can instantiate the WebAssembly module and use the functions written in pixel manipulation file, which is in c, in the js files!
cc @jywarren @tech4GT
Mridul97
on 31 Dec 2018
that sounds great!
On Mon, Dec 31, 2018 at 5:23 AM Mridul97 notifications@github.com wrote:
I was reading the above discussions.. I am thinking to implement pixel
manipulation code in c and then compile it into wasm file and then add it
into the project. After that we can instantiate the module and use the
functions written in pixel manipulation file, which is in c, in the js
files.
cc @jywarren https://github.com/jywarren @tech4GT
https://github.com/tech4GT—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-450629581,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ_pwxOf22XeHB0vwioPE21wwQGCXks5u-eVxgaJpZM4XXM4q
.
jywarren
on 2 Jan 2019
Can I work on this issue?
Divy123
on 1 Mar 2019
yes please!!!
On Fri, Mar 1, 2019 at 5:05 PM Slytherin notifications@github.com wrote:
Can I work on this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-468828032,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ8aHsoHsbGj20VY8vcgdXh2_6iaZks5vSaQmgaJpZM4XXM4q
.
jywarren
on 1 Mar 2019
@jywarren @publiclab/is-reviewers nectarJs claims that it can compile js to native OS binaries for all platforms. This can be used to compile the lib for CLI maybe? It says that it can also create webassembly from js code. (It is under development though)
NectarJs github page
harshkhandeparkar
on 2 Mar 2019
@jywarren your thoughts on nectarJs please?
Divy123
on 5 Mar 2019
Hmm, i thought webassembly is cross-platform anyways? But that is very cool!
Ah, i see that nectarJs seems to have an option to compile to WASM (which
should then be cross-platform as that's the WebAssembly format). Perhaps we
could do that with some portions of our code, like PixelManipulation, to
achieve our goal?
On Sat, Mar 2, 2019 at 12:09 AM Harsh Khandeparkar notifications@github.com
wrote:
@jywarren https://github.com/jywarren @publiclab/is-reviewers
https://github.com/orgs/publiclab/teams/is-reviewers nectarJs
https://codeburst.io/nectarjs-compiling-javascript-into-native-binaries-for-every-platform-2efb2083a4a
claims that it can compile js to native OS binaries for all platforms. This
can be used to compile the lib for CLI maybe?NectarJs github page https://github.com/NectarJS/nectarjs webpage
https://nectar-lang.com—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-468886125,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ-kudtyWRA9nW-5Io2vxsoSk2PIIks5vSgdcgaJpZM4XXM4q
.
jywarren
on 5 Mar 2019
@jywarren in that case may I proceed with nectarJs for applying the same to our PixelManipulation.js ?
Divy123
on 5 Mar 2019
I would love to see a prototype of this in a PR, yes please!
On Mon, Mar 4, 2019 at 6:32 PM Slytherin notifications@github.com wrote:
@jywarren https://github.com/jywarren in that case may I proceed with
nectarJs for applying the same to our PixelManipulation.js ?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-469470300,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ7PbPE_HUQ839p_uyAA93tzOY76Gks5vTazagaJpZM4XXM4q
.
jywarren
on 5 Mar 2019
Excited for this.
Thanks!
Divy123
on 5 Mar 2019
same!!!
On Mon, Mar 4, 2019 at 6:35 PM Slytherin notifications@github.com wrote:
Excited for this.
Thanks!—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-469471003,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ7BHsEFgRSJit5x80EFiOYjYEodZks5vTa2EgaJpZM4XXM4q
.
jywarren
on 5 Mar 2019
Really nice walkthrough of webassembly here! https://medium.freecodecamp.org/get-started-with-webassembly-using-only-14-lines-of-javascript-b37b6aaca1e4
jywarren
on 5 Mar 2019
@jywarren just a note. In the NectarJs repo, there was a discussion about it not being fully open source. I don't know what it meant. I know one thing for sure and that is that the user needs to have an account on https://nectar-lang.com and should have a key to use NectarJs. Just wanted to warn everybody before using this. The dev said he wanted help from everyone to make Nectar fully open source(maybe it is already done?)
cc @Divy123
harshkhandeparkar
on 5 Mar 2019
Ok! There are other ways of making wasm files listed above if we run into
trouble. Thanks!
On Tue, Mar 5, 2019, 12:17 PM Harsh Khandeparkar notifications@github.com
wrote:
@jywarren https://github.com/jywarren just a note. In the NectarJs
repo, there was a discussion about it not being fully open source. I don't
know what it meant. I know one thing for sure and that is that the user
needs to have an account on https://nectar-lang.com and should have a key
to use NectarJs. Just wanted to warn everybody before using this. The dev
said he wanted help from everyone to make Nectar fully open source(maybe it
is already done?)cc @Divy124
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-469769521,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ5sfAnS7tiFl5vRbVTPqbul2Ph-cks5vTqaHgaJpZM4XXM4q
.
jywarren
on 5 Mar 2019
Thanks @HarshKhandeparkar .
Divy123
on 8 Mar 2019
@jywarren I looked into WASM, made these two little projects,
https://github.com/Divy123/canvas-wasmvas
https://github.com/Divy123/prime-with-wasm
I suggest we can proceed with writing native C code and compiling that with emscripten and utilizing the genrated glue JS files in our code which I feel can be quite handy.
Your opinion please!!
Divy123
on 12 Mar 2019
Ok, however did you see the library to generate wasm from JavaScript like
syntax? In either case, we should make a src/webassembly folder perhaps to
contain our source code and put the wasm files in dist, right? Thanks!!!
On Tue, Mar 12, 2019, 1:49 PM Slytherin notifications@github.com wrote:
@jywarren https://github.com/jywarren I looked into WASM, made these
two little projects,
https://github.com/Divy123/canvas-wasmvas
https://github.com/Divy123/prime-with-wasmI suggest we can proceed with writing native C code and compiling that
with emscripten and utilizing the genrated glue JS files in our code which
I feel can be quite handy.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-472110012,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ7HecZ8GWkdjxc5adD0tRiQmB009ks5vV-icgaJpZM4XXM4q
.
jywarren
on 12 Mar 2019
Ya I looked into but perhaps feel this can be more easy to go with as then we don't have the overhead of a library.
Yes we have to keep wasm files in src/webassembly and keep it in dist!!
What do you think?
Divy123
on 12 Mar 2019
OK, sounds good. We can always refactor to use a different wasm generator
later, so this could even be solved in parts. Best first get a
demonstration running of it working at all!
On Tue, Mar 12, 2019 at 3:04 PM Slytherin notifications@github.com wrote:
Ya I looked into but perhaps feel this can be more easy to go with as then
we don't have the overhead of a library.
Yes we have to keep wasm files in src/webassembly and keep it in dist!!
What do you think?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-472139271,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJwXDb0z2BeSITLXSafke6Uq0qrUzks5vV_oggaJpZM4XXM4q
.
jywarren
on 13 Mar 2019
Sure !!
Divy123
on 15 Mar 2019
@jywarren just one thing that if you can suggest apart from PixelManipulation.js what are some other areas that we can apply wasm to include in our code base like can we add to different modules as well?
Thanks!!
Divy123
on 22 Mar 2019
Hi! Well, my thought was that PixelManipulation is used a LOT so if we
improve that, we'll see improvements across a lot of our codebase. And we
can start there to get a big improvement in one step, and then follow up to
see where else we could apply this. Many modules use PixelManipulation in
the same way, so it's a great leverage point to make a wide-ranging
optimization.
On Fri, Mar 22, 2019 at 5:06 PM Slytherin notifications@github.com wrote:
@jywarren https://github.com/jywarren just one thing that if you can
suggest apart from PixelManipulation.js what are some other areas that we
can apply wasm to include in our code base like can we add to different
modules as well?
Thanks!!—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-475782837,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJzbWiaoooj_YT0miRCRTiL_tRc01ks5vZUWtgaJpZM4XXM4q
.
jywarren
on 22 Mar 2019
Yes, this is absolutely true, the heavy lifting is done by pixelManipulation for almost all modules, so that should give us a significant boost in performance!
tech4GT
on 24 Mar 2019
What abt gpu.js which I am trying to implement for PixelManipulation? It will also work in node and will process the pixels on gpu. Can that also be compiled to wasm? It uses a node library headless-gl(will use in the future). Currently it uses the canvas api.
harshkhandeparkar
on 24 Mar 2019
Hi Harsh!
Can you explain a bit of your work here with gpu.js?
Divy123
on 24 Mar 2019
You can have a look at issue #849. Essentially it will manipulate all the pixels at the same time in parallel on a gpu.
harshkhandeparkar
on 24 Mar 2019
I think we can pursue gl acceleration in a parallel track and use our new
benchmarks to compare!
On Sun, Mar 24, 2019, 3:49 AM Harsh Khandeparkar notifications@github.com
wrote:
You can have a look at issue #849
https://github.com/publiclab/image-sequencer/issues/849. Essentially it
will manipulate all the pixels at the same time in parallel on a gpu.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400#issuecomment-475936367,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ352UrS2hQaM9Ri8hD6SvhLB-99Cks5vZy3cgaJpZM4XXM4q
.
jywarren
on 24 Mar 2019
@jywarren @tech4GT I have been through PixelManipulation.js file and what I am able to understand that the whole code here can not be converted as there are lots of things like get-pixels and save-pixels that have to be passed to our C code. So can I get some help regarding exactly what parts of the code do we want to convert to C code or the whole PixelManipulation.js code?
Thanks !!
Divy123
on 27 May 2019
@jywarren can you please help on this?
Divy123
on 27 May 2019
Also noting @MargaretAN9 is interested in this!
Hmm. I think this may be a good question to ask @HarshKhandeparkar for a bit more info here. I had hoped we could take everything /between/ get-pixels and save-pixels and run that in webassembly.
jywarren
on 27 May 2019
Ok sure!! Thanks!
@MargaretAN9 can we collaborate on this??
Also this is my this week's target.
@HarshKhandeparkar any insights?
@jywarren this is my task for this week. So should I start working on this or wait for Harsh's reply?
Divy123
on 27 May 2019
If we don't get input from others in time, perhaps we could swap this
week's task with another to give people a little more time? Thanks!
On Mon, May 27, 2019 at 2:03 PM Slytherin notifications@github.com wrote:
Ok sure!! Thanks!
@MargaretAN9 https://github.com/MargaretAN9 can we collaborate on this??
Also this is my this week's target.
@HarshKhandeparkar https://github.com/HarshKhandeparkar any insights?@jywarren https://github.com/jywarren this is my task for this week. SO
should I start working on this or wait for Harsh's reply?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400?email_source=notifications&email_token=AAAF6J26JQATHEKNGPEV2V3PXQO6JA5CNFSM4F24ZYVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWKJNTY#issuecomment-496277199,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAF6J45ZUG2K5JD2UFNJJ3PXQO6JANCNFSM4F24ZYVA
.
jywarren
on 28 May 2019
@jywarren can I just send here a PR with a working demo and then proceed with everyone's suggestions. Meanwhile I can make preparations for the next weeks as well to speed up the things there??
Divy123
on 28 May 2019
oh that'd be great! always great to see working code to offer input on.
thank you!!!
On Tue, May 28, 2019 at 4:13 PM Slytherin notifications@github.com wrote:
@jywarren https://github.com/jywarren can I just send here a PR with a
working demo and then proceed with everyone's suggestions. Meanwhile I can
make preparations for the next weeks as well to speed up the things there??—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/400?email_source=notifications&email_token=AAAF6J44CEEVYPFYG7DCLHTPXWG5HA5CNFSM4F24ZYVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWNJTUY#issuecomment-496671187,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAF6JZBAPVKKH44H2VWERDPXWG5HANCNFSM4F24ZYVA
.
jywarren
on 28 May 2019
Thanks a lot Jeff!! :smile:
Divy123
on 28 May 2019
I have made a wasm file from wasm fiddle and here is my C code
int getPixels(int , int , int );
int setPixels(int, int, int, int);
int * changePixel(int, int, int, int, int, int);
void paceOp();
void consoleLog(int);
void manipulatePixel(int width, int height, int inBrowser, int test){
consoleLog(100);
for (int x = 0; x < width; x++) {
for (int y = 0; y <height; y++) {
int* pixel =changePixel(
getPixels(x, y, 0),
getPixels(x, y, 1),
getPixels(x, y, 2),
getPixels(x, y, 3),
x,
y
);
setPixels(x, y, 0, pixel[0]);
setPixels(x, y, 1, pixel[1]);
setPixels(x, y, 2, pixel[2]);
setPixels(x, y, 3, pixel[3]);
if (!inBrowser && !test) paceOp();
}
}
}
I was trying to convert this part of code for testing from PixelManipulation.js file
for (var x = 0; x < pixels.shape[0]; x++) {
for (var y = 0; y < pixels.shape[1]; y++) {
let pixel = options.changePixel(
pixels.get(x, y, 0),
pixels.get(x, y, 1),
pixels.get(x, y, 2),
pixels.get(x, y, 3),
x,
y
);
pixels.set(x, y, 0, pixel[0]);
pixels.set(x, y, 1, pixel[1]);
pixels.set(x, y, 2, pixel[2]);
pixels.set(x, y, 3, pixel[3]);
if (!options.inBrowser && !process.env.TEST) pace.op();
}
}
After adding the wasm file, the code looks like:
const imports = {
env:{
changePixel:options.changePixel,
getPixels: pixels.get,
setPixels: pixels.set,
paceOp: require('pace').op,
consoleLog:console.log // for testing
}
}
const inBrowser = (options.inBrowser)?1:0;
const test = (process.env.TEST)?1:0
WebAssembly.instantiateStreaming(fetch('./program.wasm'), imports)
.then(wasm => {
wasm.instance.exports.manipulatePixel(pixels.shape[0],pixels.shape[1],inBrowser, test)
});
But I think the wasm file is not getting recognized, is it some bundler issue?
As of now I have kept it in src/no_module but is there something else I need to consider??
Am I required to include it in dist?If yes, then how to? I am not accustomed to it.
@jywarren @tech4GT @HarshKhandeparkar need your help here!!
Its something urgent. Please help !
Divy123
on 29 May 2019
Hi @Divy123 -- can you share a PR link, so we can take a look at how you've gotten it set up? Help us reproduce your error so we can help you solve it! I've also pinged folks in the weekly check-in to see if anyone there can help.
jywarren
on 29 May 2019
Thanks a lot @jywarren !!
Making a PR now
Divy123
on 29 May 2019
@jywarren here is the link:
https://github.com/publiclab/image-sequencer/pull/1093
Please look into it.
Divy123
on 29 May 2019
Related issues
blurry-x-face
·
4Comments
harshkhandeparkar
·
4Comments
harshkhandeparkar
·
4Comments
Divy123
·
5Comments
kindanduseful
·
5Comments
Most helpful comment
I would love to see a prototype of this in a PR, yes please!
On Mon, Mar 4, 2019 at 6:32 PM Slytherin notifications@github.com wrote: