Flatbuffers: Any examples to deal with nested objects [all languages, FlatBuffers 1.12]

[am2222]

Hi,
Is there any example that shows how we can deal with nested objects? supposed we have this object:

table Tree {
  objects : [ubyte];
  nodes:[Tree];
}

As you can see each Tree can have a set of objects and a set of nodes with the type of Tree. I want to store a QuadTree object. So each node can have some children while, we want to store them we face

Uncaught Error: FlatBuffers: object serialization must not be nested.

error, I use this function to store tree

 function () {
        var tNodes = [];
        if(this.nodes.length){
            this.nodes.forEach(node => {

                var data= node.save();
                // console.log(treeOffset);
                tNodes.push(data[0]);
            });
        }
        var builder = new flatbuffers.Builder(0)

              //Generate array of objects
            const objectsArray = new Uint8Array(100000);
            const view = new DataView(objectsArray.buffer);
            const encoder = new TextEncoder();
            let offset = 0;


            for (let index = 0; index < this.objects.length; index++) {
                view.setUint16(offset, index, true)
                offset += 2
                const str = this.objects[index];
                const stringArray = encoder.encode(str);
                view.setUint32(offset, stringArray.length, true);
                offset += 4
                objectsArray.set(stringArray, offset)
                offset += stringArray.length;
            }

           var  objectsOffset = GeoIPFS.Tree.createObjectsVector(builder, objectsArray.slice(0, offset));
            var Tree = GeoIPFS.Tree;
            Tree.startTree(builder);
            Tree.addObjects(builder, objectsOffset);

            // Tree.createNodesVector(builder,tNodes);
            tNodes.forEach(n=>{
                Tree.createNodesVector(builder,n);
            })

            var treeOffset = Tree.endTree(builder);
            builder.finish(treeOffset);
             return [builder.asUint8Array(),treeOffset]

    }

Is there any method or workaround to deal with this problem?

[mustiikhalil]

Hey, saw this issue and wanted to clarify that FlatBuffers can't nest-objects. so you would at first build, your nodes, and add them to the array of offsets tNodes then you would build this offset vector by calling Tree.createNodesVector(builder,tNodes); I am not sure if js has a method where you can simply pass the array of nodes, and it returns an offset for you. then you will create the table

var offsets: [offsets] = []
offsets.append(createdNode)

var vecOffset = builder.createVector(offsets)
or
var vecOffset = Tree.createVector(builder, offsets)
Tree.start(builder)
Tree.addNodeVectorOffset(builder, vecOffset)
Tree.end(builder)

https://github.com/google/flatbuffers/blob/99aa1ef21dd9dc3f9d4fb0eb82f4b59d0bb5e4c5/samples/samplebinary.js#L48 you can see how they are adding the weapons in the sample

[am2222]

@mustiikhalil Thanks so much for your help!
should I pass the builder to all the children? or I can make a new builder per each child? Since each item has a set of objects I think having a builder object per each child it cleaner than passing it to all the children.

[mustiikhalil]

@am2222 all the objects are supposed to be encoded into the same buffer, since you want to create a Tree object.

[am2222]

@mustiikhalil so I have to pass builder to all the children right? in my case, since I have to also convert a set of objects that exists in each node in addition to the node children's I again get nested object error

[mustiikhalil]

@am2222 so the idea of the builder is that you can construct the buffer that will contain the data; from what I understood is that you want to create one buffer of type tree. The buffer itself needs to contain the data for you to be able to reference (access) the offset.
cause what's happening is the following:
1- You will be writing the child object new buffer each time you loop on the nodes. so if you have 64 nodes you will have 64 buffers.
[...., childObject]

2- You will start adding main builder, which won't be able to refer to the child buffer when you start reading from it. since the offset of
[...., 0, 1, 0, 0 ....] which would offset (256) which doesn't really exist in the main buffer which would mean that you might be out of bound for the array.

3- Make sure that strings are added before, the object, and the children are added before parents. I am not sure if you are using structs, but they have a different approach (you can see it in the sample file)

this.nodes.forEach(node => {
     // Saving it to a different builder
      var data= node.save();
      // console.log(treeOffset);
       tNodes.push(data[0]);
});
...
// Tree.createNodesVector(builder,tNodes);
  // you are trying to reference an offset in a different builder (buffer) here. 
    tNodes.forEach(n=>{
       Tree.createNodesVector(builder,n);
    })

plus, if you see the sample file linked for js earlier, you would see that they create the children first, the go up the chain to the parent object (since the parent object would only contain the offset of the actual child) I hope this clarifies things :D

[am2222]

@mustiikhalil Thanks so much for your detailed reply.

[am2222]

@mustiikhalil Hi,
I am currently using this method to save the tree

    save = function (builder) {
        if (!builder) {
            builder = new flatbuffers.Builder(0)
        }

        var treeOffsets = []
        this.nodes.forEach(element => {
            [treeOffset,b] = element.save(builder);
            treeOffsets.push(treeOffset)
            builder=b;
        });



        const objectsArray = new Uint8Array(100000);
        const view = new DataView(objectsArray.buffer);
        const encoder = new TextEncoder();
        let offset = 0;


        for (let index = 0; index < 4; index++) {
            view.setUint16(offset, index, true)
            offset += 2
            const str = "objectHash" + index;
            const stringArray = encoder.encode(str);
            view.setUint32(offset, stringArray.length, true);
            offset += 4
            objectsArray.set(stringArray, offset)
            offset += stringArray.length;
        }

        var objectsOffset = GeoIPFS.Tree.createObjectsVector(builder, objectsArray.slice(0, offset));
        var Tree = GeoIPFS.Tree;
        Tree.startTree(builder);
        Tree.addObjects(builder, objectsOffset);
        treeOffsets.forEach(element => {
            Tree.addNodes(builder,element);
        });
        var treeOffset = Tree.endTree(builder);
        return [treeOffset,builder];


    }

This is how I get the builder's byte array

                   var builder = new flatbuffers.Builder(0)
                    var xx = element.tree.save(builder);
                    vat treeOffset = xx[0]
                    builder.finish(treeOffset);
                    var bytes = builder.asUint8Array();

It seems to work and store data with no errors. But when I try to read it back to a tree object It returns tree.nodesLenght() equal to a negative number. Do you think it is a logical thing?

[mustiikhalil]

@am2222 well, there are a couple of things that you would want to look into:
1- The nodes offsets aren't being added properly. (I might not've explained it properly). You are adding the offsets one by one although they are a vector of objects. So what you would want to do is something like this

var objectsOffset = GeoIPFS.Tree.createObjectsVector(builder, objectsArray.slice(0, offset));        
var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets)
Tree.startTree(builder);
Tree.addObjects(builder, objectsOffset);
Tree.addNodes(builder, vectorTreeOffset);
var treeOffset = Tree.endTree(builder);

2- I am not sure how JS function work, with passing objects. Because you are passing the builder into function, but I am not sure if you are getting them back.
example:

// this is used in swift & builder is being sent by reference instead of copying the object
function save(_ b: inout FlatBuffersBuilder) {}

var builder = FlatBuffersBuilder()
var offset = element.save(&builder);
// this would make the save function write into the same builder we created 

[am2222]

@mustiikhalil Thanks for your reply,
using the method you provided I face FlatBuffers: object serialization must not be nested. error. I think my object is wrong and I cannot make such a structure.

table Tree {
  objects : [ubyte];
  nodes:[Tree];
}

In my case I am building treeOffsets as a simple array. Is this a right way to do so?

 var treeOffsets = []
        this.nodes.forEach(element => {
            [treeOffset,b] = element.save(builder);
            treeOffsets.push(treeOffset)
        });
if(treeOffsets.length>0){
            var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets);
            Tree.addNodes(builder,vectorTreeOffset);
        }

[mustiikhalil]

@am2222 you forgot to add the Tree.start()?

var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets)
Tree.startTree(builder);
Tree.addObjects(builder, objectsOffset);

and remember you need to call endTable, if you startTable

[am2222]

@mustiikhalil
sorry about incomplete example

Here is the full function. I am using Tree.start and Tree.endTree functions and also at the end I am using builder.finish() function at the end but it breaks at this line
var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets)

    HTree.prototype.save = function (builder) {
        if (!builder) {
            builder = new flatbuffers.Builder(0)
        }

        var treeOffsets = []
        this.nodes.forEach(element => {
            [treeOffset,b] = element.save(builder);
            treeOffsets.push(treeOffset)
            builder=b;
        });



        const objectsArray = new Uint8Array(100000);
        const view = new DataView(objectsArray.buffer);
        const encoder = new TextEncoder();
        let offset = 0;


        for (let index = 0; index < 4; index++) {
            view.setUint16(offset, index, true)
            offset += 2
            const str = "objectHash" + index;
            const stringArray = encoder.encode(str);
            view.setUint32(offset, stringArray.length, true);
            offset += 4
            objectsArray.set(stringArray, offset)
            offset += stringArray.length;
        }

        var objectsOffset = GeoIPFS.Tree.createObjectsVector(builder, objectsArray.slice(0, offset));


        var Tree = GeoIPFS.Tree;
        Tree.startTree(builder);
        Tree.addObjects(builder, objectsOffset);
        if(treeOffsets.length>0){
            var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets);
            Tree.addNodes(builder,vectorTreeOffset);
        }

        var treeOffset = Tree.endTree(builder);
        return [treeOffset,builder];


    }

[mustiikhalil]

@am2222 you see the example shows that you are trying to add the vector of nodes within the serialization of the object tree.

var objectsOffset = GeoIPFS.Tree.createObjectsVector(builder, objectsArray.slice(0, offset));        
var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets)
// add vectors into the buffer before using startTable, and endTable
Tree.startTree(builder);
// Offset of the vector of objects
Tree.addObjects(builder, objectsOffset);
// Offset of the vector of trees
Tree.addNodes(builder, vectorTreeOffset);
var treeOffset = Tree.endTree(builder);

 var Tree = GeoIPFS.Tree;
Tree.startTree(builder);
Tree.addObjects(builder, objectsOffset);

if(treeOffsets.length>0){
  // this shouldnt be here.
  var vectorTreeOffset = GeoIPFS.Tree.createNodesVector(builder, treeOffsets);
  Tree.addNodes(builder,vectorTreeOffset);
}

[am2222]

@mustiikhalil Thanks so much for your help, I was ignoring adding empty node array when tree does not have any nodes, now it works fine. Thanks so much

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.