Image-sequencer: Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node

[jywarren]

https://www.npmjs.com/package/gl

gl lets you create a WebGL context in node.js without making a window or loading a full browser environment.

This would no longer be pure JavaScript, but for some modules this is interesting. For example, the FisheyeGL module #27 can currently only be run in a browser, and webgl-distort #64 also would be this way.

Long-term project!

---

Hmm, maybe also these resources:

• non graphics card GL functions could be shimmed with https://github.com/Overv/JSGL
• or this: https://github.com/stackgl/headless-gl

[Divy123]

Sorry for being late on the fisheyeGl issue but I was facing some issues which I am going through.
Will soon come up with a solution.

[tech4GT]

Okay I brainstormed this a lot and I came up with 2-3 solutions

1. We can run a headless browser like puppeteer and run this module in it's context, then get the result using the callback.
2. We can try to use jsdom in conjunction with gl and make a global document object that can get a web-gl context. I have been trying this out but so far it looks like this would require some changes in the fisheyeGL library by @jywarren . Since the library uses new Image() and I was not able to find anything to emulate that in node, I tried using jsdom but it does not work with the constructor, maybe we can change this to

document.querySelector("#image-node") || new Image();

1. We can enable web-gl support in our modules by automatically requiring gl in node context. This would however mean that code for browser and node wold have to be different, which is what I am trying to avoid here.
   cc @jywarren thoughts?

[tech4GT]

Also there's an issue with node_version for gl It was compiled for node 8.15.0 latest, so If we go with this we would have to use that version
Also please do see the warning on their npm page

[Divy123]

@tech4GT @jywarren I have done some research and here are some points on this:
First of all the gl produces the canvas element that does not have texImage2D function method on it .
So that's a drawback I think.
Also @jywarren can you please explain how can https://github.com/Overv/JSGL be useful here.
While researching for fisheyeGl module I went through but there seemed no good resources to move with it.
Also I came across basicHTML but not of much use as there canvas.getContext() not implemented.
https://github.com/WebReflection/basicHTML/issues/32.
I was trying to learn webGl so that I can be helpful with this . Can you people please suggest some good resource to go forward?
And also puppeteer can be quite good to go with.

[Divy123]

@jywarren request your help here.

[jywarren]

Hmm, to @tech4gt I think it's ok to have different code in browser or node if that code split exists only inside the module context. Although we could eventually try to generalize parts to make it easier to write webgl modules.

[jywarren]

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl
though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

>

[jywarren]

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <[email protected] wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl
though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

>

[jywarren]

Although we could likely generate a dataurl from puppeteer instead of
screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

>

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <[email protected] wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of
fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

>

[jywarren]

It sounds reasonable, and we could try to improve on the loadtime later?

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

Although we could likely generate a dataurl from puppeteer instead of
screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

>

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <[email protected] wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of
fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

>

[jywarren]

https://github.com/GoogleChrome/puppeteer/issues/1260

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

It sounds reasonable, and we could try to improve on the loadtime later?

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

Although we could likely generate a dataurl from puppeteer instead of
screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <[email protected] wrote:

>

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <[email protected] wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of
fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

>

[Divy123]

GoogleChrome/puppeteer#1260

In that case it can't be.

[jywarren]

Ah but see here:
https://github.com/GoogleChrome/puppeteer/issues/1260#issuecomment-348878456

On Sun, Feb 17, 2019, 4:40 PM Slytherin <[email protected] wrote:

GoogleChrome/puppeteer#1260
https://github.com/GoogleChrome/puppeteer/issues/1260
In that case it can't be.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464535727,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ5hDYfl-DMWiCuW6zIEBxYZTAxJuks5vOfZ0gaJpZM4TceLs
.

[Divy123]

Use at your own risk though, since @aslushnikov says Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

[jywarren]

Hmm, do they say why?

On Sun, Feb 17, 2019, 5:18 PM Slytherin <[email protected] wrote:

Use at your own risk though, since @aslushnikov
https://github.com/aslushnikov says
Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464542054,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJyMKGhYRvpz5y4v8a_K8SfsP97O3ks5vOf86gaJpZM4TceLs
.

[jywarren]

I think potentially we just have to start trying things and see what we can
get to work.

On Sun, Feb 17, 2019, 5:20 PM Jeffrey Warren <[email protected] wrote:

Hmm, do they say why?

On Sun, Feb 17, 2019, 5:18 PM Slytherin <[email protected] wrote:

Use at your own risk though, since @aslushnikov
https://github.com/aslushnikov says
Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464542054,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJyMKGhYRvpz5y4v8a_K8SfsP97O3ks5vOf86gaJpZM4TceLs
.

[Divy123]

Sure @jywarren
Not mentioned specifically something about it.
I think that's the only way to work around right now.

[tech4GT]

@jywarren Are we open to modifying your fisheye-gl coed then? I can make the changes and put in a pr.

[Divy123]

@tech4GT is there something I can help out with?

[tech4GT]

@Divy123 Sure! Once we finalize what path we want to follow here, I'll do some groundwork and then you can help me build on it :smile:

[Divy123]

Sure thanks!

[jywarren]

Absolutely we can do work upstream!

Noting also that I'm really not sure what happens when you run WebGL code
in Node environment on a server that has no GPU...?? But I'm asking someone
at Google!

On Tue, Feb 19, 2019 at 8:44 AM Slytherin [e-mail] wrote:

Sure thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-465212325,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ9DcGiU7M4XV1efthAKstaOUSGOiks5vPCm2gaJpZM4TceLs
.

[harshkhandeparkar]

NOOOCL is a nodejs package which runs openCL (not GLSL) shaders in nodejs. https://github.com/unbornchikken/nooocl

[jywarren]

oh whoaaaaa cool!

On Tue, Mar 12, 2019 at 2:45 PM Harsh Khandeparkar [e-mail]
wrote:

NOOOCL is a nodejs package which runs openCL (not GLSL) shaders in nodejs.
https://github.com/unbornchikken/nooocl

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-472132423,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJz18jaKfB69RJtSTphu8AG8tEqSTks5vV_WZgaJpZM4TceLs
.

[harshkhandeparkar]

Hello! Last commit for headless-gl https://github.com/stackgl/headless-gl was 4d ago. It will compatible with latest node versions. A new version was released recently and gpu.js also will use headless-gl for v2 instead of NOOOCL

[harshkhandeparkar]

NOOOCL has no release. Only a pre-release, so i don't recommend using it.

[harshkhandeparkar]

I would suggest going with headless-gl since it is maintained to work in all recent versions of node(thought the devs say that it is not actively developed, so no new features but compatibility)

[Divy123]

Can you please provide some more details about headless-gl? I went through their doc but found no way to input an image?
What do you think?

[harshkhandeparkar]

You can pass in an unit8Array or float32Array or any array. An example would be pixels.data array which has a size of width*height*channels. This array can be given to the readPixels method of the module. Also this module provides a full canvas context, so any canvas methods available in the browser should be available here. I hope this helps.

[Divy123]

Thanks a lot @HarshKhandeparkar .

[jywarren]

Just be aware of what will require an actual GPU and could in theory fail
on Travis or in the cloud (if there is no GPU on cloud servers, for
example) or what would have a fall-back solution. But I don't really know
how it works, so perhaps the best way would be to try it out!

On Mon, Mar 18, 2019 at 7:42 AM Slytherin [e-mail] wrote:

Thanks a lot @HarshKhandeparkar https://github.com/HarshKhandeparkar .

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473876386,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ6JKJBZI9T1-SGRIi7FpR-zg8QFmks5vX3t_gaJpZM4TceLs
.

[harshkhandeparkar]

Does webgl in browser fallback to cpu mode?

[harshkhandeparkar]

@Divy123 @jywarren if everyone uses laptop or mac, I can try this stuff on my pc with gpu and then remove the gpu and test again as I have a custom built pc. @Divy123 you can do the initial work and open a pr or tell me your branch and I will test. If you have a custom pc which you can modify, you can try it out yourself. I'm going test out gpu.js with and without gpu anyways so I can also test this if needed. Shall I do that? Please note that I will only be available from april 16th to may 2nd or 3rd, after that I cannot do anything. So if you are going to start work on this, try to do it fast. Thanks.

[Divy123]

Sure @jywarren @HarshKhandeparkar
Applying this year for SoC, so can I include this in my summer proposal?
I will be brainstorming now and do some initial work and putting this on hold till then, is that fine?

[Divy123]

I have a list of points I want to implement this summer which includes this one and the web workers.
I have already started writing the web assembly code and I think these three can be a great addition to the performance.
@jywarren what do you think upon this?

[jywarren]

This all sounds great, we'd love to have this in a SoC proposal too. Just
noting however that we ideally want to try to do the GPU modules /somehow/
before June, for MapKnitter purposes (webgl-distort module) but the other
optimizations can happen a bit slower. Thanks, all!

Noting also - some CPUs have an integrated GPU. I wonder if they will fall
back to that. I also wonder if Travis and other cloud-based systems skip
the integrated GPU since they may be purpose-built for cloud computing, and
if they would have no GPU at all. Finally i wonder if some libs may fall
back to pure CPU-based calculations if there's nothing else. Some testing
should help us figure all this out, but one of the major target platforms
will be Google Cloud or Travis, so perhaps we should consider testing on
those where we can. Thank you all!

On Mon, Mar 18, 2019 at 12:10 PM Slytherin [e-mail] wrote:

I have a list of points I want to implement this summer which includes
this one and the web workers.
I have already started writing the web assembly code and I think these
three can be a great addition to the performance.
@jywarren https://github.com/jywarren what do you think upon this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473981334,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ-_0Vtmk-hWalM0rEvcIhfSZI5bEks5vX7ppgaJpZM4TceLs
.

[harshkhandeparkar]

Uh, I don't know much about SoC. So do you want me to test this out? If yes, when will you be able to produce the (let's say) scaffolding for running gl headlessly? Btw my implementation of gpu.js will not take time and can be done in a few days. What I 'm waiting for is the next major release of gpu.js which will allow using gpu in node for GPGPU. See issue #849 for reference.

[harshkhandeparkar]

My cpu has a built in gpu, so I can test it on that and we can maybe benchmark it to see if it used the gpu or used the cpu. It can be run on travis which will mostly not have a gpu and we can test if it falls back to pure cpu computation which will mostly take more time than on gpu but it can also take less time since they will have a good cpu.

[harshkhandeparkar]

gCloud will probably have a gpu since it is also used for ML and other similar purposes.

[Divy123]

@HarshKhandeparkar will explore it better and reach out to you soon.
Also thank you soo much for your such a great help.
I think we have to test it on your cpu.
Also @jywarren I will start working on this part as soon as wasm part is done on an initial basis probably by the mid of april and will try to complete it within May itself .

[harshkhandeparkar]

Ok sure. 😊

[harshkhandeparkar]

Btw when is SoC going to start?

[Divy123]

SoC will start from may 27 but work will never stop because ultimately it's all about our performance.

[jywarren]

:-) I love the energy! We do also want to be sure that everyone
participates at a healthy pace that ensures a good life/work balance. Thank
you so much!!!

On Mon, Mar 18, 2019 at 12:45 PM Slytherin [e-mail] wrote:

SoC will start from 27 but work will never stop because ultimately it's
all about our performance.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473999747,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ1cVWFklPlf5HdRB2XiEVrXlxgBgks5vX8KfgaJpZM4TceLs
.

[Divy123]

Thank you @jywarren for making this a super awesome learning platform for us.

[harshkhandeparkar]

@jywarren chinmay was a gsoc student right? Why doesn't he contribute anymore? Sorry if I asked something wrong or somewhere wrong. I was just curious.

[harshkhandeparkar]

I'm unpinning this for the time being to oin the community check in

[jywarren]

Chinmay was, yes - he got a job and wasn't able to keep contributing - this
happens sometimes (esp. due to work restrictions sometimes) but I'm sure he
is very happy to see all the work on this project that he helped to kick
off!

On Mon, Mar 18, 2019 at 1:48 PM Harsh Khandeparkar [e-mail]
wrote:

I'm unpinning this for the time being to oin the community check in

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474028266,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ4547iMw9wBibcJq464XL5NDH-C3ks5vX9FOgaJpZM4TceLs
.

[Divy123]

@jywarren staring with headlessGl here.
Just one question should I start converting the fisheye module to test out this?
Or is there something else I can continue with for this?

[jywarren]

Yes please if you could start with the webgl-distort library that would be
great!

https://github.com/jywarren/webgl-distort

Cc @tech4gt ! 🎉🎉

On Tue, Mar 19, 2019, 10:48 AM Slytherin [e-mail] wrote:

@jywarren https://github.com/jywarren staring with headlessGl here.
Just one question should I start converting the fisheye module to test out
this?
Or is there something else I can continue with for this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474404009,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ9TvHFDCtcqp5vuDUM7tHto3XOX3ks5vYPifgaJpZM4TceLs
.

[Divy123]

Sure!
Thanks.

[VibhorCodecianGupta]

@jywarren @Divy123 seeing as this is one of the GSoC projects this year and is already being worked upon, is this still open to incoming proposals or shall I draft one pertaining to real-world use cases of IS? I have the necessary research for both, but this looks like it's already being worked upon?

[Divy123]

I think you can go on with real world use cases one if you want to and I would be really thankful if you can share your research here as I have already started working on this.
Also I think we can discuss this with Jeff.
Thanks!!
@jywarren your thoughts please.

[jywarren]

I also think it's fine to assume that even if we have an initial
implementation for any of these, we might want to refine it as part of SoC!

On Tue, Mar 19, 2019 at 12:15 PM Slytherin [e-mail] wrote:

I think you can go on with real world use cases one and I would be really
thankful if you can share your research here.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474451020,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJyWUkE6nRchSkKv1-ZIDLCh7-lOOks5vYQzxgaJpZM4TceLs
.

[Divy123]

I also think it's fine to assume that even if we have an initial
implementation for any of these, we might want to refine it as part of SoC!

Can you please explain a bit more on this?

[Divy123]

Actually me and Vibhor both are applying for GSoC. So what's your view how should we proceed on this one?

[jywarren]

Oh i just mean that -- think of any feature, the first version can be good
but may still be improve-able, right, so there's almost always room to
expand on a feature later. Coming up with a solution now need not mean
there can't be a Summer of Code proposal for it later too!

On Tue, Mar 19, 2019 at 3:17 PM Slytherin [e-mail] wrote:

I also think it's fine to assume that even if we have an initial
implementation for any of these, we might want to refine it as part of SoC!
Can you please explain a bit more on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474535518,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ53xu-C2z9W2dP8CWgz6y90G0TgZks5vYTetgaJpZM4TceLs
.

[jywarren]

And we also allow multiple people working on the same project - we advise
folks to find ways to break up a problem into parts for better
collaboration.

On Tue, Mar 19, 2019 at 3:19 PM Jeffrey Warren [e-mail] wrote:

Oh i just mean that -- think of any feature, the first version can be good
but may still be improve-able, right, so there's almost always room to
expand on a feature later. Coming up with a solution now need not mean
there can't be a Summer of Code proposal for it later too!

On Tue, Mar 19, 2019 at 3:17 PM Slytherin [e-mail]
wrote:

I also think it's fine to assume that even if we have an initial
implementation for any of these, we might want to refine it as part of
SoC!
Can you please explain a bit more on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474535518,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ53xu-C2z9W2dP8CWgz6y90G0TgZks5vYTetgaJpZM4TceLs
.

[Divy123]

And we also allow multiple people working on the same project - we advise
folks to find ways to break up a problem into parts for better
collaboration.

@jywarren I think that can be the best way to go on.
Thanks

[Divy123]

You can pass in an unit8Array or float32Array or any array. An example would be pixels.data array which has a size of widthheightchannels. This array can be given to the readPixels method of the module. Also this module provides a full canvas context, so any canvas methods available in the browser should be available here. I hope this helps.

@HarshKhandeparkar you said that it provides full canvas context, so like if I do canvas.toDataURI() in browser, what can be done in here?
Can you please explain a bit here?

[harshkhandeparkar]

It is the full canvas API Available in nodejs. Whatever works in the browser will work here.

[Divy123]

you said that it provides full canvas context, so like if I do canvas.toDataURI() in browser, what can be done in here?

[harshkhandeparkar]

Same can be done

[Divy123]

But canvas is not there so gl.?

[harshkhandeparkar]

Yes. You can also assign it to a var named canvas

[Divy123]

Means instead I can do
var canvas = require('gl')(....)

[harshkhandeparkar]

Yes

[Divy123]

Coool!!

[harshkhandeparkar]

@Divy123 i found something important. Please have a look at it.
https://github.com/stackgl/headless-gl/blob/master/README.md#system-dependencies

[Divy123]

I looked into.
Do you find some problem in this?

[harshkhandeparkar]

You need to be careful as this might not work on every system. Might need some system dependencies to be installed

[Divy123]

For sure, Thanks.

[jywarren]

Looks like for Travis it may require a travis.yml change...
https://github.com/stackgl/headless-gl/blob/master/README.md#how-can-i-use-headless-gl-with-a-continuous-integration-service
https://github.com/stackgl/headless-gl/blob/master/README.md#how-can-i-use-headless-gl-with-a-continuous-integration-service

This is great news as it answers our questions about needing a GPU on a
cloud container!

On Wed, Mar 20, 2019, 7:21 AM Slytherin [e-mail] wrote:

For sure, Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474788803,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ4FWd3HC1rZsWIaWoacCteeekrFUks5vYhmlgaJpZM4TceLs
.

[Divy123]

I think this can be solved now without any worries.
Working on webgl-distort in progress..
Cc:@jywarren

[Divy123]

Here are the steps I am planning to take with converting webgl-distort to be node compatible:
1). Creating an image with jsdom
2) Creating headless-gl context.
3)Adding Image to the texture.
4)Applying perspective with the given matrices.
5)Converting gl image texture back to dataURI .

I am a bit unsure on point 4 above.
@jywarren @tech4GT @HarshKhandeparkar can you please tell if I am in the right direction?

[jywarren]

That's right! Try just moving a couple corners by a hundred pixels or so,
something simple. The module will accept new x,y coordinates for each of
the four corners.

Sounds great!

On Wed, Mar 20, 2019, 9:07 AM Slytherin [e-mail] wrote:

Here are the steps I am planning to take with converting webgl-distort to
be node compatible:
1). Creating an image with jsdom
2) Creating headless-gl context.
3)Adding Image to the texture.
4)Applying perspective with the given matrices.
5)Converting gl image texture back to dataURI .

I am a bit unsure on #4
https://github.com/publiclab/image-sequencer/issues/4.
@jywarren https://github.com/jywarren @HarshKhandeparkar
https://github.com/HarshKhandeparkar can you please tell if I am in the
right direction?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474820468,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJzz7QDamlhgETpQ-5C75i2eSo1HAks5vYjKBgaJpZM4TceLs
.

[harshkhandeparkar]

Try translation first to test if headless-gl works.

[harshkhandeparkar]

Try tanslating or rotating using simple matrices first. Even scale is ok

[Divy123]

Sure @HarshKhandeparkar .

[Divy123]

This is some of the code I have written to apply some basic filters to an image in headless-gl.
But I am pretty unsure about how to fetch the results out of it.
There we do have canvas in webGL but here I don't know.
@jywarren please help in this.

const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { document } = (new JSDOM(`...`)).window;


const image = document.createElement('img');
image.crossOrigin = "anonymous";
image.src = './examples/example-1024.jpg';
image.onload = function () {

  var width = 64
  var height = 64
  var gl = require('gl')(width, height, { preserveDrawingBuffer: true })


  gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);
  gl.clearColor(1.0, 0.8, 0.1, 1.0);
  gl.clear(gl.COLOR_BUFFER_BIT);

  const vertShaderSource = '
    attribute vec2 position;
    varying vec2 texCoords;
    void main() {
      texCoords = (position + 1.0) / 2.0;
      texCoords.y = 1.0 - texCoords.y;
      gl_Position = vec4(position, 0, 1.0);
    }
  ';

  const fragShaderSource = '
    precision highp float;
    varying vec2 texCoords;
    uniform sampler2D textureSampler;
    void main() {
      float warmth = -0.2;
      float brightness = 0.2;
      vec4 color = texture2D(textureSampler, texCoords);
      color.r += warmth;
      color.b -= warmth;
      color.rgb += brightness;
      gl_FragColor = color;
    }
  ';

  const vertShader = gl.createShader(gl.VERTEX_SHADER);
  const fragShader = gl.createShader(gl.FRAGMENT_SHADER);

  gl.shaderSource(vertShader, vertShaderSource);
  gl.shaderSource(fragShader, fragShaderSource);

  gl.compileShader(vertShader);
  gl.compileShader(fragShader);

  const program = gl.createProgram();
  gl.attachShader(program, vertShader);
  gl.attachShader(program, fragShader);

  gl.linkProgram(program);

  gl.useProgram(program);

  const vertices = new Float32Array([
    -1, -1,
    -1, 1,
    1, 1,

    -1, -1,
    1, 1,
    1, -1,
  ]);

  const vertexBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
  gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);

  const positionLocation = gl.getAttribLocation(program, 'position');

  gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);
  gl.enableVertexAttribArray(positionLocation);

  const texture = gl.createTexture();
  gl.activeTexture(gl.TEXTURE0);
  gl.bindTexture(gl.TEXTURE_2D, texture);

  var getPixels = require("get-pixels")

getPixels('./examples/example-1024.jpg', function(err, pixels) {
  if(err) {
    console.log("Bad image path")
    return
  }
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA,  2,2,gl.RGBA,gl.UNSIGNED_BYTE, new Uint8Array(pixels.data));
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);

  gl.drawArrays(gl.TRIANGLES, 0, 6);
  console.log(pixels.data)

})  
}();

[Divy123]

@HarshKhandeparkar

[harshkhandeparkar]

That is a lot to digest. I will answer in a few hrs. I am busy. Sorry.

[Divy123]

No issues .
Thanks

[Divy123]

This is some of the code I have written to apply some basic filters to an image in headless-gl.
But I am pretty unsure about how to fetch the results out of it.
There we do have canvas in webGL but here I don't know.
@jywarren please help in this.

const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { document } = (new JSDOM(...)).window;

const image = document.createElement('img');
image.crossOrigin = "anonymous";
image.src = './examples/example-1024.jpg';
image.onload = function () {

var width = 64
var height = 64
var gl = require('gl')(width, height, { preserveDrawingBuffer: true })

gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);
gl.clearColor(1.0, 0.8, 0.1, 1.0);
gl.clear(gl.COLOR_BUFFER_BIT);

const vertShaderSource = '
attribute vec2 position;
varying vec2 texCoords;
void main() {
texCoords = (position + 1.0) / 2.0;
texCoords.y = 1.0 - texCoords.y;
gl_Position = vec4(position, 0, 1.0);
}
';

const fragShaderSource = '
precision highp float;
varying vec2 texCoords;
uniform sampler2D textureSampler;
void main() {
float warmth = -0.2;
float brightness = 0.2;
vec4 color = texture2D(textureSampler, texCoords);
color.r += warmth;
color.b -= warmth;
color.rgb += brightness;
gl_FragColor = color;
}
';

const vertShader = gl.createShader(gl.VERTEX_SHADER);
const fragShader = gl.createShader(gl.FRAGMENT_SHADER);

gl.shaderSource(vertShader, vertShaderSource);
gl.shaderSource(fragShader, fragShaderSource);

gl.compileShader(vertShader);
gl.compileShader(fragShader);

const program = gl.createProgram();
gl.attachShader(program, vertShader);
gl.attachShader(program, fragShader);

gl.linkProgram(program);

gl.useProgram(program);

const vertices = new Float32Array([
-1, -1,
-1, 1,
1, 1,

-1, -1,
1, 1,
1, -1,

]);

const vertexBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);

const positionLocation = gl.getAttribLocation(program, 'position');

gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);
gl.enableVertexAttribArray(positionLocation);

const texture = gl.createTexture();
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, texture);

var getPixels = require("get-pixels")

getPixels('./examples/example-1024.jpg', function(err, pixels) {
if(err) {
console.log("Bad image path")
return
}
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 2,2,gl.RGBA,gl.UNSIGNED_BYTE, new Uint8Array(pixels.data));
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);

gl.drawArrays(gl.TRIANGLES, 0, 6);
console.log(pixels.data)

})
}();

@jywarren Please look into this.

[jywarren]

Hmm, is there a gl.toDataURL method?

canvas.toDataURL('image/png')

https://github.com/jywarren/webgl-distort/blob/f8a86cb7c60b25a1728ffb532c8ec2db116821f7/dist/webgl-distort.js#L3-L28

On Mon, Mar 25, 2019 at 2:15 PM Slytherin [e-mail] wrote:

This is some of the code I have written to apply some basic filters to an
image in headless-gl.
But I am pretty unsure about how to fetch the results out of it.
There we do have canvas in webGL but here I don't know.
@jywarren https://github.com/jywarren please help in this.

const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { document } = (new JSDOM(...)).window;

const image = document.createElement('img');
image.crossOrigin = "anonymous";
image.src = './examples/example-1024.jpg';
image.onload = function () {

var width = 64
var height = 64
var gl = require('gl')(width, height, { preserveDrawingBuffer: true })

gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);
gl.clearColor(1.0, 0.8, 0.1, 1.0);
gl.clear(gl.COLOR_BUFFER_BIT);

const vertShaderSource = '
attribute vec2 position;
varying vec2 texCoords;
void main() {
texCoords = (position + 1.0) / 2.0;
texCoords.y = 1.0 - texCoords.y;
gl_Position = vec4(position, 0, 1.0);
}
';

const fragShaderSource = '
precision highp float;
varying vec2 texCoords;
uniform sampler2D textureSampler;
void main() {
float warmth = -0.2;
float brightness = 0.2;
vec4 color = texture2D(textureSampler, texCoords);
color.r += warmth;
color.b -= warmth;
color.rgb += brightness;
gl_FragColor = color;
}
';

const vertShader = gl.createShader(gl.VERTEX_SHADER);
const fragShader = gl.createShader(gl.FRAGMENT_SHADER);

gl.shaderSource(vertShader, vertShaderSource);
gl.shaderSource(fragShader, fragShaderSource);

gl.compileShader(vertShader);
gl.compileShader(fragShader);

const program = gl.createProgram();
gl.attachShader(program, vertShader);
gl.attachShader(program, fragShader);

gl.linkProgram(program);

gl.useProgram(program);

const vertices = new Float32Array([
-1, -1,
-1, 1,
1, 1,

-1, -1,
1, 1,
1, -1,

]);

const vertexBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);

const positionLocation = gl.getAttribLocation(program, 'position');

gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);
gl.enableVertexAttribArray(positionLocation);

const texture = gl.createTexture();
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, texture);

var getPixels = require("get-pixels")

getPixels('./examples/example-1024.jpg', function(err, pixels) {
if(err) {
console.log("Bad image path")
return
}
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 2,2,gl.RGBA,gl.UNSIGNED_BYTE, new
Uint8Array(pixels.data));
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);

gl.drawArrays(gl.TRIANGLES, 0, 6);
console.log(pixels.data)

})
}();

@jywarren https://github.com/jywarren Please look into this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476317958,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ9i8dm5lYEV1Ux9eoU3WW7H_F1-bks5vaRIqgaJpZM4TceLs
.

[harshkhandeparkar]

I think there is a canvas.toDataUri() method. @Divy123

[harshkhandeparkar]

Oh we both answered together. Ok I googled it. My answer is wrong. It is toDataURL()

[Divy123]

there is one on canvas but here there is no such method available.
@HarshKhandeparkar

[jywarren]

:-))))))

On Mon, Mar 25, 2019 at 2:19 PM Harsh Khandeparkar [e-mail]
wrote:

I think there is a canvas.toDataUri() method. @Divy123
https://github.com/Divy123

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476319322,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ-UB2DY_0p-3SCSdWf8NcVtA-13Iks5vaRMTgaJpZM4TceLs
.

[harshkhandeparkar]

Are you sure? Did you try it out?

[Divy123]

yes

[harshkhandeparkar]

The docs for headless-gl say that it returns a WebGLRenderingContext object. Here is a link to MDN docs about WebGLRenderingContext

[Divy123]

But I console.logged this and didn't get anything.

On Mon 25 Mar, 2019, 11:56 PM Harsh Khandeparkar <[email protected]
wrote:

The docs for headless-gl say that it returns a WebGLRenderingContext
object. Here
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext
is a link to MDN docs about WebGLRenderingContext

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476322284,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Adx0r8lw2el4xakGIAo_ebRXZR_PMLWWks5vaRTYgaJpZM4TceLs
.

[harshkhandeparkar]

Try doing
```js
console.log(typeof gl.toDataURL)

[Divy123]

ok wait!!

[Divy123]

Still undefined @HarshKhandeparkar

[harshkhandeparkar]

Even I 'm not sure now. Try searching for some other suitable method in the MDN docs link i just gave you. Maybe get-pixels accepts a buffer as an input? I'm not sure..

[Divy123]

I am a bit unsure about how to get the changed pixels out of the gl format like you can see the code above here.
Like there we have the context from canvas but here we directly do have gl. @HarshKhandeparkar
@jywarren Is there a way even in webGL that I can have the changed texture data as per the transformations without using canvas.

[jywarren]

Actually webgl is a variant of 2d canvas, so it always has a canvas
context. Let me look a sec..

On Mon, Mar 25, 2019, 2:40 PM Slytherin [e-mail] wrote:

I am a bit unsure about how to get the changed pixels out of the gl format
like you can see the code above here.
Like there we have the context from canvas but here we directly do have
gl. @HarshKhandeparkar https://github.com/HarshKhandeparkar
@jywarren https://github.com/jywarren Is there a way even in webGL that
I can have the changed texture data as per the transformations without
using canvas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476327274,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ_JjiBtMD5fvx6oqy99wCPkk3DnYks5vaRgRgaJpZM4TceLs
.

[jywarren]

Ah, readpixels- see the example?

https://github.com/stackgl/headless-gl#example

On Mon, Mar 25, 2019, 2:44 PM Jeffrey Warren [e-mail] wrote:

Actually webgl is a variant of 2d canvas, so it always has a canvas
context. Let me look a sec..

On Mon, Mar 25, 2019, 2:40 PM Slytherin [e-mail] wrote:

I am a bit unsure about how to get the changed pixels out of the gl
format like you can see the code above here.
Like there we have the context from canvas but here we directly do have
gl. @HarshKhandeparkar https://github.com/HarshKhandeparkar
@jywarren https://github.com/jywarren Is there a way even in webGL
that I can have the changed texture data as per the transformations without
using canvas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476327274,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ_JjiBtMD5fvx6oqy99wCPkk3DnYks5vaRgRgaJpZM4TceLs
.

[harshkhandeparkar]

So let me ask a few questions:

1. Are all of the pixels saved to a texture?
2. Do you specifically save them to a texture or are they required from the RAM on the go?
3. If you do save the data to a texture, there are some methods(i think) which return a buffer from the texture. I think you will have to explore the MDN docs.
4. In normal GLSL(the shader language), what you do is save the data to a texture, process the data on the GPU, get the data back from the gpu after rasterisation(converting to a bitmap). Now this data can be processed in many different ways. WebGL gives you a method to directly convert it to a DataURL but there may be other methods which return different things too. Try exploring.

[jywarren]

Also is it possible you need to use extra config on your node call like
shown here, for it to work?

https://github.com/stackgl/headless-gl#how-can-headless-gl-be-used-on-a-headless-linux-machine

On Mon, Mar 25, 2019, 2:45 PM Jeffrey Warren [e-mail] wrote:

Ah, readpixels- see the example?

https://github.com/stackgl/headless-gl#example

On Mon, Mar 25, 2019, 2:44 PM Jeffrey Warren [e-mail] wrote:

Actually webgl is a variant of 2d canvas, so it always has a canvas
context. Let me look a sec..

On Mon, Mar 25, 2019, 2:40 PM Slytherin [e-mail] wrote:

I am a bit unsure about how to get the changed pixels out of the gl
format like you can see the code above here.
Like there we have the context from canvas but here we directly do have
gl. @HarshKhandeparkar https://github.com/HarshKhandeparkar
@jywarren https://github.com/jywarren Is there a way even in webGL
that I can have the changed texture data as per the transformations without
using canvas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476327274,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ_JjiBtMD5fvx6oqy99wCPkk3DnYks5vaRgRgaJpZM4TceLs
.

[harshkhandeparkar]

The links you provided don't seem to be working. @jywarren

[jywarren]

Oh really? i just tried, them can you try again? It also looks like they're recommending get-pixels, but not sure if that's after an image has been exported. But these links show some examples for fetching per-pixel data!

[harshkhandeparkar]

No luck. It just opens the repo homepage i guess but not the readme. Both of them. Maybe it is because I use my phone? You may never know, github breaks in a different way every new day.

[Divy123]

Well I saw how to get data out of texture but no success till now! Still exploring.
Please do help if you find something @HarshKhandeparkar @jywarren .

[jywarren]

Ah yes, on mobile view it doesn't open the full readme. But skim down the
readme, it's quite informative!

On Mon, Mar 25, 2019 at 3:24 PM Slytherin [e-mail] wrote:

Well I saw how to get data out of texture but no success till now! Still
exploring.
Please do help if you find something @HarshKhandeparkar
https://github.com/HarshKhandeparkar @jywarren
https://github.com/jywarren .

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476342360,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJxEKl5MvfYD5oD8p75KRmpragcksks5vaSJYgaJpZM4TceLs
.

[harshkhandeparkar]

Ok.

[harshkhandeparkar]

O don't know what this does but it is included in the readme. Is it helpful?

process.stdout.write(['P3\n# gl.ppm\n', width, " ", height, '\n255\n'].join('')) for(var i=0; i<pixels.length; i+=4) { for(var j=0; j<3; ++j) { process.stdout.write(pixels[i+j] + ' ') }

[Divy123]

I have looked into this but the main issue how to get the data out of texture.

[harshkhandeparkar]

What does the code i posted above do?

[harshkhandeparkar]

There is a getImageData() method but it is a method of CanvasRenderingext2D object.

[Divy123]

Its writing image or pixles data as a PPM formatted image.

[Divy123]

There is a getImageData() method but it is a method of CanvasRenderingext2D object.

Again with canvas.

[harshkhandeparkar]

WebGLRenderingContext object has a canvas property which will either return a offscreenCanvas element or null. Try this out.

[tech4GT]

https://github.com/GoogleChrome/puppeteer/issues/1260
This solves an issue I was earlier facing with the puppeteer implementation, Just adding it here since it may be useful!

[Divy123]

@HarshKhandeparkar this is still not available.
@tech4GT hope you can elaborate on your idea of this.

[tech4GT]

Sure @Divy123 Anything in particular you want my help on?

[Divy123]

Like I tried using headless-gl for transforming webgl-distort to be node compatible.
But the problem I am facing is due to no canvas being available and attached to the context, I can not get data out of a texture. Googled a alot but no success till now.
But I know a bit about headless browser concept and would be delighted if you can help me with that like some basic steps that how can we use headless browser here.
I am already exploring but it would be great if you help me out with the approach we can take with puppeteer.
Thanks a lot!!

[tech4GT]

Oh, the approach I tried on this was nothing special as well, I was running the headless browser inside node and the module was running on the sequencer inside the headless browser!

[tech4GT]

The issue I posted earlier relates to the fact that by default gl is disabled in puppeteer. You need to explicitly enable it.

[tech4GT]

This is not an ideal solution though since we would want to avoid running the headless browser if possible(coz it'll take up a lot of resources)

[jywarren]

Even if it takes a lot of resources (starting up the browser) I still think we should move ahead with this implementation -- it's the closest I've seen to actually working so far. We could refactor and attempt another approach afterwards, but I think we have to try! Thank you all!!!

[jywarren]

Hi @Divy123 could you open a PR with your attempt so far? I'd like to try a few things.

For example, https://github.com/stackgl/headless-gl#example says we can access the pixels like:

//Create context
var width   = 64
var height  = 64
var gl = require('gl')(width, height, { preserveDrawingBuffer: true })

//Clear screen to red
gl.clearColor(1, 0, 0, 1)
gl.clear(gl.COLOR_BUFFER_BIT)

//Write output as a PPM formatted image
var pixels = new Uint8Array(width * height * 4)
gl.readPixels(0, 0, width, height, gl.RGBA, gl.UNSIGNED_BYTE, pixels)
process.stdout.write(['P3\n# gl.ppm\n', width, " ", height, '\n255\n'].join(''))
for(var i=0; i<pixels.length; i+=4) {
  for(var j=0; j<3; ++j) {
    process.stdout.write(pixels[i+j] + ' ')
  }
}

[tech4GT]

@jywarren I have an idea, maybe I can create a binding between gl and node-canvas and add the gl based functions on node-canvas using gl! I think this might be worth a shot.

[tech4GT]

If this works, it would allow for 100% code sharing between browser and node.

[tech4GT]

@jywarren @Divy123 I think I have figured this out, what I am doing is using 2 libraries:
node-canvas and gl, and I have overriden the default canvas.getContext to return an instance of 'node-gl'
Apart from that I needed some more changes in the fisheye-gl code and I think this might be ready.
Now do you guys think we should modify jywarren/fisheyegl or I should bake the node-specific code in the module itself?

[jywarren]

Can you push what code you have and we can see what changes would be best
in which module? Thanks!!!!

On Sun, Mar 31, 2019, 4:51 AM Varun Gupta [e-mail] wrote:

@jywarren https://github.com/jywarren @Divy123
https://github.com/Divy123 I think I have figured this out, what I am
doing is using 2 libraries:
node-canvas and gl, and I have overriden the default canvas.getContext to
return an instance of 'node-gl'
Apart from that I needed some more changes in the fisheye-gl code and I
think this might be ready.
Now do you guys think we should modify jywarren/fisheyegl or I should bake
the node-specific code in the module itself?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-478323673,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ07a33j86jwsMQncyolIYslkh85qks5vcHcJgaJpZM4TceLs
.

[tech4GT]

@jywarren I have figured out a lot of things, can you explain a little about how this code is important though.
https://github.com/jywarren/fisheyegl/blob/89d28f0dee943681192a708b59c076e3d5854652/src/fisheyegl.js#L204-L217
Since we don't have window available in node context, how should we change this code?

[tech4GT]

The current code is outputting a black image and I am unable to figure out why.

[jywarren]

I think we can skip this in node because it seems to be to limit the
refresh of the library to the frame rate of the browser window refresh:
https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

Hmm, maybe try a simpler webgl task to be sure the basic idea works?

On Sun, Mar 31, 2019, 10:08 AM Varun Gupta [e-mail] wrote:

The current code is outputting a black image and I am unable to figure out
why.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-478345105,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ5r3Wmz8WRn8R9zw5nQ92-TCbsbOks5vcME0gaJpZM4TceLs
.

[tech4GT]

Okay there is clearly some issue with the gl module, so for now I think we should proceed with the headless browser implementation. We can circle back to the other implementation once the issues are rectified.
cc @jywarren @Divy123

[Divy123]

I agree @tech4GT .
Exploring puppeteer nowadays to work on this.
Also I request you and @jywarren to please review my SoC proposal :
https://publiclab.org/notes/lit2017001/04-01-2019/soc-proposal-image-sequencer-v3-boosting-the-performance.
Thanks!!

[jywarren]

Did you open an issue to ask more about it, and if so could you link there for reference? Thank you!

[jywarren]

I will review ASAP, apologies this has been taking a lot of time this week 😄 thanks for your patience!

[Divy123]

Thanks a lot @jywarren .

[tech4GT]

@jywarren here https://github.com/stackgl/headless-gl/issues/149

[jywarren]

oh awesome. Thanks.

On Thu, Apr 4, 2019 at 10:16 AM Varun Gupta [e-mail]
wrote:

@jywarren https://github.com/jywarren here stackgl/headless-gl#149
https://github.com/stackgl/headless-gl/issues/149

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/216#issuecomment-479915661,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ016CuB23ccHHGx4DiX3hvfjkthXks5vdgk6gaJpZM4TceLs
.

[tech4GT]

@jywarren @Mridul97 So I was trying this out and I think the initial reload we do for the pwa results in puppeteer execution context being destroyed on sequencer.publiclab.org, I think we need to work around it somehow. That was a temporary solution anyway I guess.

[tech4GT]

@publiclab/is-reviewers Can someone point me to the link for dist files.
I am trying to add image-sequencer.min.js as a script on a page.

[harshkhandeparkar]

We are getting help from @robertleeplummerjr in using headless-gl
cc @tech4GT @jywarren @Divy123 @publiclab/is-reviewers

[robertleeplummerjr]

I don't have a ton of time I can offer, already on borrowed, but I can look into getting the fisheye project running on headless-gl, and then we can connect the dots backwards to this project, so it can be loaded in. Does that sound reasonable?

[jywarren]

Wow that is very generous, thank you! We are very interested in this bc our puppeteer based approach (running a whole headless Chrome) is not very efficient. We would /love/ to work with whatever you can figure out and work to generalize a narrow case.

For what it's worth, this is another WebGL module we use that has simpler parameters (four corner coordinate pairs): https://github.com/jywarren/webgl-distort

You can see them both running in the browser here with a standard input and output:

• https://sequencer.publiclab.org/examples/#steps=webgl-distort
• https://sequencer.publiclab.org/examples/#steps=fisheye-gl

Thank you!!!

[robertleeplummerjr]

If I use headlessgl like this:

const gl = require('gl');
const context = gl(1,1);
const getPixels = require('get-pixels');
const fisheyeGL = require('../../src/fisheyegl.js');
getPixels('example/images/grid.png', function(err, pixels) {
  if (err) {
    console.log(err);
    return;
  }

  fisheyeGL({
    image: pixels.data,
    getGLContext: () => context,
  });
});

And in fisheyegl, if I force the flow to use the pixels lib mentioned above, and I change line 157 from:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);

to:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);

I believe it works.

Here is the full diff I did:

diff --git a/src/fisheyegl.js b/src/fisheyegl.js
index db7369b..0117c63 100644
--- a/src/fisheyegl.js
+++ b/src/fisheyegl.js
@@ -41,7 +41,7 @@ var FisheyeGl = function FisheyeGl(options){
   var image = options.image || "images/barrel-distortion.png";

   var selector = options.selector || "#canvas";
-  var gl = getGLContext(selector);
+  var gl = options.getGLContext ? options.getGLContext() : getGLContext(selector);

   var shaders = require('./shaders');

@@ -172,7 +172,7 @@ var FisheyeGl = function FisheyeGl(options){

     gl.bindTexture(gl.TEXTURE_2D, texture);

-    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);
+    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);

     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); //gl.NEAREST is also allowed, instead of gl.LINEAR, as neither mipmap.
@@ -201,6 +201,13 @@ var FisheyeGl = function FisheyeGl(options){
     return texture;
   }

+  function loadImageFromPixels(gl, pixels, callback){
+    console.log('pixels', pixels);
+    var texture = gl.createTexture();
+    loadImage(gl, pixels, callback, texture);
+    return texture;
+  }
+
   function run(animate, callback){
     var f = window.requestAnimationFrame || window.mozRequestAnimationFrame ||
       window.webkitRequestAnimationFrame || window.msRequestAnimationFrame;
@@ -272,7 +279,11 @@ var FisheyeGl = function FisheyeGl(options){
     });
   }

-  setImage(image);
+  if (typeof image === 'string') {
+    setImage(image);
+  } else {
+    loadImageFromPixels(gl, image);
+  }

   // asynchronous!
   function getImage(format) {

[robertleeplummerjr]

You'll notice I only had an image size of 1x1 above, so not a practical solution, but proof you can load in the pixels.

[harshkhandeparkar]

Wow! Thanks @robertleeplummerjr
@jywarren @tech4GT @VibhorCodecianGupta @Divy123