Ml5-library: New Model -- Conditional VAE

[WenheLI]

Nature of issue?

• [ ] Found a bug
• [ ] Existing feature enhancement
• [ ] New feature request
• [x] Other

New feature details:

@Derek-Wds and I have trained a model for Conditional VAE. And we have wrapped it with ml5. Here is what we have implemented so far as a workable ml5 version of CVAE.
https://github.com/WenheLI/ml5-library/blob/CVAE/src/CVAE/index.js

You can also view the cvae on top of tfjs.
http://cvae.steins.live/

Given the discussion on #302 , I have proposed a set of API that may be helpful for such generative like model to design.

let labels = ["top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle Boot"];

let cvae = ml5.CVAE('./model.json', labels, (err, res) => {
    res.generate("Dress", 0, 1, (err, ans) => {
         let ctx = document.getElementById('draw').getContext('2d');
         ctx.clearRect(0,0,300,150); //fresh
         let imgData = ctx.createImageData(28, 28); // width x height
         let data = imgData.data;
         for (let i = 0 ; i < 28*28*4; i+= 1)  data[i] = ans.bytes[i]; 
         ctx.putImageData(imgData, 0, 0);
    })
});

In my design, the result of the generate function should have 2 attributes, 1. bytes that contain an array of all the pixels; 2. a blob URL or blob object that contains the generated results.

{
    "image": Blob,
    "bytes": Array
}

The reason for me to have the blob is to help us simply convert any generate results to p5.Image by calling loadImg(blobLink) .

As for the model itself, we have done a pre-trained model on mnist and fashion-mnist and we are working on the quick draw model. Also, we will release a training code. Theoretically , you can use our training code to get any model you want.

[shiffman]

This is amazing work! We discussed this example a bit at ITP today. Two comments I have so far:

1. I think it might be nice to include the labels in model.json itself rather than have them passed around as a separate array? We do this for ml5.FeatureExtractor and ml5.KNNClassifier. We may have to build this into the python script for training as well.

2. Thinking about this more (and I am repeating a comment I just posted in #294), what if we detect it p5 is present and if so, wrap the data in a p5.Image? So we would return:

{
  "image": p5.Image,
  "blob": Blob,
  "bytes": Array
}

If not p5, then we'd skip image and it would be up to the end user to figure out what to with the data (DOM element or canvas context?).

Is there a better name for blob or bytes? Another option:

{
  "image": p5.Image,
  "src": Blob,
  "raw": Array
}

[WenheLI]

I am thinking of making a function from ml5 called exportToP5 or a function from p5js called loadImgFromBytes that allows users to get p5Image when they want, since returning a large nested object and TensorFlow.js might be memory-consuming.

[viztopia]

@WenheLI nice design! We can definitely do the same thing for DCGAN.

A few things to confirm:

1. I checked Zaid's PR, and found that src of the image DOM is used. It seems that the name "src" should be a good fit for all of us? @zaidalyafeai

2. The src will always be base64 PNG?

3. Are we using the p5.instance or the preLoad() to check if p5 exists?

4. Do we have any preferred formatting for the "bytes"/"raw"? For instance, will tf.Tensor.array() be good enough? @shiffman @joeyklee

[WenheLI]

@viztopia I am thinking that the src should be a Blob object or Blob URL since it will be quite easy to load the Blob to p5.image compared with base64.

As for checking p5, I used the below code to do so in previous project:

checkP5() {
        if (typeof window !== 'undefined' && window.p5 && window.p5.Image && typeof window.p5.Image === 'function') return true;
        throw new Error('p5 is not imported.');
    }

[shiffman]

@zaidalyafeai and I are currently discussing this on slack. Here are my current thoughts, let me know what you think!

For all models that return an image:

1. The callback function should come with a single argument that is a JavaScript object.

function gotResult(result) {
  const blob = result.blob;   // a JavaScript blob base64 encoded image
  const raw = result.raw;     // a JavaScript array with the raw data
  const image = result.image; // a p5.Image object
}

@WenheLI 's code above can be used to determine if p5 exists. If not, than image can be skipped. Perhaps we should have something in ml5/utils that is a p5 checker shared by all classes?

@viztopia I think it could also be useful to include a reference to the tensor as there are instances where you might want to pass the result directly to another model. But working with tensors is typically the domain on non-ml5 users. What about:

function gotResult(result) {
  const blob = result.blob;     // a JavaScript blob base64 encoded image
  const raw = result.raw;       // a JavaScript array with the raw data
  const tensor = result.tensor; // a Tensor with the raw data
  const image = result.image;   // a p5.Image object
}

We're doing a lot of extra work for the case when an ml5 user just wants to use the blob or the raw tensor, but that in my view is the point of ml5. The library is not the fastest / most efficient but it the easiest way to run a model and use its results.

[viztopia]

If we would have something in ml5/utils as a p5 checker, will there also be something that could convert raw data to p5.Image object and shared by all classes, e.g. array3DToP5Image?

[WenheLI]

@viztopia +1. I am thinking this kind of util functions are useful like arrayToP5Image, arrayToBlob. I can start a PR on this thing if it is needed!

[shiffman]

Yes, this is a great idea! For reference, we have https://github.com/ml5js/ml5-library/blob/development/src/utils/imageUtilities.js. I'm not sure if this would be a new file or the same one.

[WenheLI]

I am thinking we put the functions into a new file called p5Util.js might make the structure more clear?

[joeyklee]

Closing this issue now :) Many thanks + excellent work ya'll! Looking forward to having it in the next/upcoming release!