Pixi.js: How to improve fromImage texture load time with big image?

[hellraisercenobit]

Expected Behavior

When we want to load a texture on the fly with fromImage to replace an old one, it should be fast without any artifact (1600x1600 img)

Current Behavior

Load time seems to be too slow and we can see flickering/clipping between interval to render
After some pass, all become smoothy.

Possible Solution

It is possible to create a texture from compressedTexture without loader to improve load time?
Or another way?

Steps to Reproduce

    const app = new PIXI.Application({
      width: window.innerWidth,
      height: window.innerHeight,
      view: <HTMLCanvasElement>
    });

let currentSprite = something

setInterval(() => {
      const texture: PIXI.Texture = PIXI.Texture.fromImage('data:image/jpeg;base64,DYNAMICNEWENCODEDIMG');
      const sprite: PIXI.Sprite = new PIXI.Sprite(texture);
      app.stage.addChild(sprite);
      app.removeChild(currentSprite);
      currentSprite = sprite;
}, 500);

Environment

• pixi.js version: _e.g. 4.7.1_
• Browser: _e.g. Chrome 67_
• Device: _e.g. Desktop_

[englercj]

The blocking part you notice is likely the upload of the image to the GPU, which if you are using a common image format (png, jpg, etc) includes a decoding step that is CPU intensive.

One potential solution is to use "compressed textures", this will increase the download time (non-blocking) but reduce the upload/decode time (blocking). This may require additional work since pixi doesn't natively support compressed textures as texture sources. Though this may help.

Another is to use createImageBitmap in a worker. That way you explicitly control the decode step and force it to be on a worker thread (non-blocking). This may require additional work as pixi doesn't natively support the resultant ImageBitmap as a texture source.

Of course, these are all ways to move the work off the main thread so user's don't notice. However, they probably both will increase the time it takes to get a texture from the network into your scene graph. It just won't block as much (async methods are almost always slower than sync methods).

[hellraisercenobit]

@englercj thx, in my case load time from network is not a big deal.
I'm working only with generated base64 encoded images on the fly like an img stream.

createImageBitmap seems to be a good choice but no Edge support :'(.
I will try this (the disadvantage is to manage async result to render next img and not the first loaded).

I have already tried a lot of things:

• Reduce images sizes but load time seems to be laggy too (power of 2 1024 low res instead of 1600) so GPU load time seems to not depend really on image size (maybe on HDRES).
• Reduce fps inside the ticker too (the only one improvement).
• Prerender all my sprites in my stage and toggle them with (addChild, removeChild) but the expected result is bad (laggy too).
• Use the loader to cache everything before starting the app (noooot working, same as others).
• Swap texture vs new Sprite (new Sprite win)
• Binary to canvas (convert buffer to texture resource is too slow)

Could splitted textures decreased GPU load time (like 512X4)?
I have already seen your link on compressed texture plugins, hopeful!
No way to use direct input instead a real dds file (i mean like a base64 img)?
Do you know a tool to convert img to dds or a good node module to do that easily?

So next steps: ImageBitmap and DXT

[englercj]

Honestly your best bet is to profile and see what is taking up most of your time. I'm guessing it is upload/decode, but I don't know without profiling :)

[andrewstart]

For creating .dds files from .png/.jpg, I have found https://github.com/BKcore/crunch-osx to be useful. I made a wrapper for it primarily aimed at creating the 'crunch' compressed format, but it should be capable of producing .dds files (haven't spent much time testing it). If you go with .dds, you'll want to try to make sure that the server is sending them gzipped, otherwise the network load time might become a big deal. If you decide that the crunched images are what you want due to filesize/network load, then you'll want to set up the crunch decoding in a webworker. When I changed from PNGs to crunched textures for locally loaded files on a phone, the overall load + upload time was about the same as when I used pngs, but there was no stutter because the main thread was never blocked.

[hellraisercenobit]

@englercj yeah it's just upload/decode, i'll profiling just in case.
My sample is really simple, just have 100 pictures and animate like a spritesheet at 25fps (with textures > 1024x1024).

I have tried with compressed textures dxt, it 's seems to be better (but need a preload not my target).
I don't know if we can load data with the plugin to skip one step.

@andrewstart thx andrew i'm on mac too ^^, do you have a sample of crunch decoding? with v4? :)
My target is phone/tablet/desktop too.
It could be helpfull

V5 alpha is supposed to do that for us no?

I will provide some examples to share.

Another approach:
Better to do putImageData + fromCanvas instead of fromImage?

Thx a lot guys.

[hellraisercenobit]

@englercj in another ticket you expose the fact pixi v5 support ImageBitmap.
How should i create texture with ImageBitmap Worker with alpha v2?
I would like to create a sample with pixi V5 to see the diff.

[ivanpopelyshev]

https://github.com/pixijs/pixi.js/blob/next/packages/core/src/textures/resources/ImageResource.js#L156

ImageResource creates bitmap before it actually uploads texture to videomemory. Texture might be just a blank for a pair of frames while decoding happens in separate thread.

[ivanpopelyshev]

Its enabled by default.

If you want to get more info, debug that thing and read the source code of texture resources.

There are no demos yet and all the people who understand how it works are currently busy.

Compared to v4, the issue went down from requires complete knowledge of pixijs to requires knowledge of texture-resource mechanism. Maybe later it will be even easier to explain/configure, but right now you have to understand how it works, there's no way out of it.

[hellraisercenobit]

@ivanpopelyshev no problem man, i understand, i was already on sources (ImageResource.js) to know how it work.

Thx a lot for your answers

[ivanpopelyshev]

Then you are our first tester :)

Also there might be bugs, we had one with premulitplyAlpha, and i'm sure more problems are coming due to async nature of createImageBitmap. I hope our new architecture is good enough for you to do necessary hacks for your app.

I think we need to change loader that way it can construct custom resource for the image, ImageResource by default.

[ivanpopelyshev]

By the way, I'm going to dig into those resources in a hour or two, to make compressed textures support, so if you join pixijs slack (tell me your email!) you'll be able to get help online for today.

[hellraisercenobit]

@ivanpopelyshev my mail => [email protected]
I don't know if i will have time tonight after work (i'm UTC+2).
I will be happy to join slack and try to do not ask too many questions.
I have seen implementation and it do not use inline webworker now but just the createBitmapdata promise (a good start).
Maybe an improvement? :)
It will be hard to make a webperf diff, v5 seems to me slower on bunnymark (normal it's a wip).
Thx for your help.

[andrewstart]

Unfortunately, I don't have an example of the crunch decoding because it is internal code and somewhat dependent on our own loading system. However, at its core it is code from https://github.com/toji/texture-tester/blob/master/js/webgl-texture-util.js, just modified to be just a webworker for loading crunch/dds binary data. The original code runs in both the worker and the main thread, and does different things depending on the mode.

[hellraisercenobit]

@andrewstart interesting, how did you use decoded result with pixi? you use v5 alpha to be able to use ImageData?
If it's v4, i suppose you don't use compressed textures plugin to do this.

[themoonrat]

@hellraisercenobit Did you make any progress on this? (wondering if this specific issue should be closed or not) It might be a v5 addition if it's not already @ivanpopelyshev ?

[ivanpopelyshev]

v5 supports imageBitmap, we have compressed textures PRs, as for partial upload of big textures nobody made this kind of plugin so far. Also, original poster contacted us in slack ;) You may close this ticket.

[themoonrat]

:+1: I need to catch up looking at the v5 branch!

[hellraisercenobit]

@themoonrat best approach was to manage my textures manually one by one with fromImage and update event.
I have tried with v5 to load my texture as binary format but you'll be able to do this only with raw data (not jpeg or png).
The size of your image will be bigger (a lot) and load time too.
Too much factor could be an issue for this usecase.
Depend on client hardware so I have deciced to work with smaller image to reduce load time.
You will not be able to load big image (1024 or sup) fast enough to have a smooth live animation without any preload.
Could be better with dds/compressed on the fly (gpu love it).

[themoonrat]

Cool, thanks for the feedback, useful tips!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.