Three.js: Asset management problems in architecture

[Usnul]

This is something I have been struggling with for a while. Three.js doesn't manage your assets very well. It creates duplicates, specifically my problem is with duplication of textures and materials.

When you load a texture - you get a brand new Texture object, the TextureLoader under the hood uses Cache, which is a global singleton, meaning that everything that you ever load will be in Cache. There is no eviction, meaning that it's very possible to leak memory here if you're loading dynamic things, like say, if you created a model viewer or an editor of some kind, user loads different models over and over again, and Cache just retains all of that. As a user you have just 2 options here:

1. Hunt down specific object and use Cache.remove(key), or
2. Use Cache.clear() which is a nuclear option that results in removal of everything in cache

The Cache is a key-value store, which is not so much of an issue, but it gets used with URL of an asset being loaded as the key - this means that given a URL - you can only ever have 1 representation of the data under that URL. Imagine following scenario:

You loaded a texture "happy-cat.png" using TextureLoader, then, down the road, you want to access binary data of the PNG specifically to parse it using some PNG library, suppose you want to access metadata stored in there, such as user name, palette, creation date, creation software etc. So, you use FileLoader. FileLoader uses same Cache, and it will also use the same url ("happy-cat.png") to access the Cache, so it will return you, not a "bytearray" like you asked, but, in fact, a DOM Image. There are more scenarios where this can happen and be an issue. As a minimum, an Asset Cache needs to be able to distinguish typed of Assets and store data from the same source in multiple representations, such as Image and ByteArray.

Suppose you are Ms Smarty-Pants and you decided to take all the models in your scene and pack textures for those into a single image, this way you can reduce texture switching to 0! That's crazy good, right? Well... When you load your models, say "Tree", "House" and "Cat" - each of these models will get a new Material, even if that said Material will be identical in your case. Each of these Materials will get its own new Texture, and that means that there will be 3 unique copies of that carefully packed image loaded to the GPU for your scene. Here's the relevant piece of code:
https://github.com/mrdoob/three.js/blob/733eb1d58bd70a7e161f3ced21093f186f0e8b9f/src/renderers/webgl/WebGLTextures.js#L294-L324
As you can see, "uploadTexture" happens based on texture.version, it doesn't care that this texture is exactly the same as another texture that we already have in memory.

I am ashamed to admit that I was that Mr Smarty-Pants, and only caught this when doing performance profiling today. I found out that my image is being decoded multiple times and is being uploaded to the GPU multiple times, here's a helpful piece of code to identify such cases that you can put around the "uloadTexture":

if (!texture.isDataTexture && texture.image !== undefined && texture.image.src) {
   console.time("Loading Texture " + texture.image.src);
}

uploadTexture(textureProperties, texture, slot);

if (!texture.isDataTexture && texture.image !== undefined && texture.image.src) {
   console.timeEnd("Loading Texture " + texture.image.src);
   console.log(textureProperties, texture, slot);
}

I propose that there should be a way to reuse static textures, one way to do that would be to de-couple the Texture Data from Texture Settings and use equality checks (with hashing to make things faster). Similar approach could be applied for Materials too, we'd be able to clean up some of the code that does something functionally similar for materials currently (when compiling shaders).

[titansoftime]

I had similar issues. My "solution" was to disable the THREE.Cache entirely. Glad someone else is looking into this =]

[Usnul]

I was thinking about how to mitigate the Texture duplication issue, the simplest solution seems checking for equality. For that Texture would need extra 2 methods:

texture.equals( other : Texture ) : boolean
texture.hash() : number

the hash could be a string, but honestly - i dislike the notion of super-long strings as hashes, I would prefer a fixed-length primitive, such as number.

Inside WebGLTextures we could have a Map<Texture, TextureUnit> and Map<Hash, Texture[]> this way you could easily find if texture with the same data/parameters is already uploaded and get the TextureUnit (just a number) of it.

const texture2textureUnit = new Map();
const hash2textures = new Map();

...

function tryUploadTexture(texture){
    const tHash = texture.hash();

    const possibleMatches = hash2textures.get(tHash);

    if(possibleMatches !== undefined){
        for(const t of possibleMatches){
           if(t.equals(texture)){
                // found exact match
                const textureUnit = texture2textureUnit.get(t); 
                state.activeTexture(textureUnit);
                return;
           }
        }
    }

    uploadTexture(texture);
}

I think you'd need an extra layer of abstraction, like TextureUnitData or something like that, to keep track of reference counts and prevent modification to Textures from polluting hash map.

just some thoughts. I was thinking I'd implement something like that quickly, but after looking at the existing code in the uploadTexture with all the different texture type clauses - I figured a quick fix without those 2 method would be too specific and not handle all the texture types.

[Usnul]

A small update. I have been mulling the problem over in my head and I figured I could handle it as a post-processing step, so I implemented a decorator for GLTF loader that computes hashes of materials and keeps a cache, if a loaded model's material is found in the cache - it gets replaced by an already existing one. This has produced amazing results for me, number of compiled materials has dropped by as much as 50% in some cases, and number of used texture units has gone down too, as expected.

There are obvious limitations to such an approach, you assume all materials are Immutable basically, that's a pretty big assumption to make for some usecases.

for those who might want to do something similar, here's a code listing of my solution, for educational purposes:

import { Cache } from "../../../../core/Cache.js";
import { computeHashArray, computeHashFloat, computeHashIntegerArray } from "../../../../core/math/MathUtils.js";
import { computeStringHash } from "../../../../core/strings/StringUtils.js";


/**
 *
 * @param {Plane} plane
 * @returns {number}
 */
function planeHash(plane) {
    //TODO implement
    return 0;
}

/**
 *
 * @param {Plane} a
 * @param {Plane} b
 * @returns {boolean}
 */
function planesEqual(a, b) {
    return a.equals(b);
}


/**
 * @template T
 * @param {T[]} a
 * @param {T[]} b
 * @param {function(T,T):boolean} elementsEqual
 * @returns {boolean}
 */
function arraysEqual(a, b, elementsEqual) {
    if (a === b) {
        return true;
    }

    if (a === null || b === null || a === undefined || b === undefined) {
        return false
    }

    const l = a.length;
    if (l !== b.length) {
        return false;
    }

    for (let i = 0; i < l; i++) {
        const aE = a[i];
        const bE = b[i];

        if (!elementsEqual(aE, bE)) {
            return false;
        }
    }

    return true;
}

/**
 *
 * @param {Texture} texture
 * @returns {number}
 */
function textureHash(texture) {
    //TODO implement
    return 0;
}

/**
 *
 * @param {Image} a
 * @param {Image} b
 * @returns {boolean}
 */
function textureImagesEqual(a, b) {
    if (a instanceof Image && b instanceof Image) {
        //both are images
        if (a.src === b.src) {
            //same source
            return true;
        }
    }

    return false;
}

/**
 *
 * @param {Texture} a
 * @param {Texture} b
 * @returns {boolean}
 */
function texturesEqual(a, b) {
    if (a === b) {
        return true;
    }

    if (a === null || b === null || a === undefined || b === undefined) {
        return false
    }

    if (
        !textureImagesEqual(a.image, b.image)
        || a.mapping !== b.mapping
        || a.wrapS !== b.wrapS
        || a.wrapT !== b.wrapT
        || a.magFilter !== b.magFilter
        || a.minFilter !== b.minFilter
        || a.anisotropy !== b.anisotropy
        || a.format !== b.format
        || a.type !== b.type
        || !a.offset.equals(b.offset)
        || !a.repeat.equals(b.repeat)
        || !a.center.equals(b.center)
        || a.rotation !== b.rotation
        || a.generateMipmaps !== b.generateMipmaps
        || a.premultiplyAlpha !== b.premultiplyAlpha
        || a.flipY !== b.flipY
        || a.unpackAlignment !== b.unpackAlignment
        || a.encoding !== b.encoding
    ) {
        return false;
    }

    //TODO implement support for other texture types
    return true;
}

/**
 *
 * @param {Material} material
 * @returns {number}
 */
function materialHash(material) {
    let hash = 0;

    hash = computeHashIntegerArray(
        hash,
        computeHashFloat(material.alphaTest),
        material.blendDst,
        material.blendDstAlpha === null ? 0 : computeHashFloat(material.blendDstAlpha),
        material.blendEquation,
        material.blendEquationAlpha === null ? 0 : computeHashFloat(material.blendEquationAlpha),
        material.blending,
        material.blendSrc,
        material.blendSrcAlpha === null ? 0 : computeHashFloat(material.blendSrcAlpha),
        material.clipIntersection ? 0 : 1,
        material.clippingPlanes === null ? 0 : computeHashArray(material.clippingPlanes, planeHash),
        material.clipShadows ? 0 : 1,
        material.colorWrite ? 0 : 1,
        material.depthFunc,
        material.depthTest ? 0 : 1,
        material.depthWrite ? 0 : 1,
        material.fog ? 0 : 1,
        material.lights ? 0 : 1,
        computeHashFloat(material.opacity),
        material.polygonOffset ? 0 : 1,
        computeHashFloat(material.polygonOffsetFactor),
        computeHashFloat(material.polygonOffsetUnits),
        computeStringHash(material.precision),
        material.premultipliedAlpha ? 0 : 1,
        material.dithering ? 0 : 1,
        material.flatShading ? 0 : 1,
        material.side,
        material.transparent ? 0 : 1,
        computeStringHash(material.type),
        material.vertexColors,
        material.vertexTangents ? 0 : 1,
        material.visible ? 0 : 1,
    );

    if (material.isMeshStandardMaterial) {
        //TODO extend hash
    }

    return hash;
}


/**
 *
 * @param {Material|MeshStandardMaterial} a
 * @param {Material|MeshStandardMaterial} b
 * @returns {boolean}
 */
function materialEquals(a, b) {
    if (a.type !== b.type) {
        return false;
    }

    if (
        a.alphaTest !== b.alphaTest
        || a.blendDst !== b.blendDst
        || a.blendDstAlpha !== b.blendDstAlpha
        || a.blendEquation !== b.blendEquation
        || a.blendEquationAlpha !== b.blendEquationAlpha
        || a.blending !== b.blending
        || a.blendSrc !== b.blendSrc
        || a.blendSrcAlpha !== b.blendSrcAlpha
        || a.clipIntersection !== b.clipIntersection
        || !arraysEqual(a.clippingPlanes, b.clippingPlanes, planesEqual)
        || a.clipShadows !== b.clipShadows
        || a.colorWrite !== b.colorWrite
        || a.depthFunc !== b.depthFunc
        || a.depthTest !== b.depthTest
        || a.depthWrite !== b.depthWrite
        || a.fog !== b.fog
        || a.lights !== b.lights
        || a.opacity !== b.opacity
        || a.polygonOffset !== b.polygonOffset
        || a.polygonOffsetFactor !== b.polygonOffsetFactor
        || a.polygonOffsetUnits !== b.polygonOffsetUnits
        || a.precision !== b.precision
        || a.premultipliedAlpha !== b.premultipliedAlpha
        || a.dithering !== b.dithering
        || a.flatShading !== b.flatShading
        || a.side !== b.side
        || a.transparent !== b.transparent
        || a.vertexColors !== b.vertexColors
        || a.vertexTangents !== b.vertexTangents
        || a.visible !== b.visible
    ) {
        return false;
    }

    if (a.isMeshStandardMaterial) {
        if (
            !a.color.equals(b.color)
            || a.roughness !== b.roughness
            || a.metalness !== b.metalness
            || !texturesEqual(a.map, b.map)
            || !texturesEqual(a.lightMap, b.lightMap)
            || a.lightMapIntensity !== b.lightMapIntensity
            || !texturesEqual(a.aoMap, b.aoMap)
            || a.aoMapIntensity !== b.aoMapIntensity
            || !a.emissive.equals(b.emissive)
            || a.emissiveIntensity !== b.emissiveIntensity
            || !texturesEqual(a.emissiveMap, b.emissiveMap)
            || !texturesEqual(a.bumpMap, b.bumpMap)
            || a.bumpScale !== b.bumpScale
            || !texturesEqual(a.normalMap, b.normalMap)
            || a.normalMapType !== b.normalMapType
            || !a.normalScale.equals(b.normalScale)
            || !texturesEqual(a.displacementMap, b.displacementMap)
            || a.displacementScale !== b.displacementScale
            || a.displacementBias !== b.displacementBias
            || !texturesEqual(a.roughnessMap, b.roughnessMap)
            || !texturesEqual(a.metalnessMap, b.metalnessMap)
            || !texturesEqual(a.alphaMap, b.alphaMap)
            || !texturesEqual(a.envMap, b.envMap)
            || a.envMapIntensity !== b.envMapIntensity
            || a.refractionRatio !== b.refractionRatio
            || a.wireframe !== b.wireframe
            || a.wireframeLinewidth !== b.wireframeLinewidth
            || a.skinning !== b.skinning
            || a.morphTargets !== b.morphTargets
            || a.morphNormals !== b.morphNormals
        ) {
            return false;
        }
    } else {
        //TODO implement other material types
        return false;
    }

    return true;
}


export class StaticMaterialCache {
    constructor() {

        this.materialCache = new Cache({
            maxWeight: 1000,
            keyHashFunction: materialHash,
            keyEqualityFunction: materialEquals
        });

    }

    /**
     *
     * @param {Material} material
     * @returns {Material}
     */
    acquire(material) {
        const existingMaterial = this.materialCache.get(material);
        if (existingMaterial === null) {
            //doesn't exist, add
            this.materialCache.put(material, material);
            return material;
        } else {
            return existingMaterial;
        }
    }
}

[Usnul]

I believe that adding hash and equals methods to Texture and Material will help others with such solutions, and it may enable some simplifications inside the WebGLRednerer as a whole.