Flatbuffers: Reverse engineering

[siebeneicher]

Hi,

for a project I have to reverse engineer a binary Flatbuffer. For Protobuf we have some useful tools like protocol --decode-raw. Any advice on how to create a scheme manually based on a given binary?

Thanks in advance, Markus

[aardappel]

A binary FlatBuffer is much harder to dump than a Protobuf, since you have no type information at all. You can decode the root table pretty easily (since you know its a table), but then decoding its fields will be hard. You'll have to guess wether a 32 bit value is an offset to a table/vector/string or a float/int just by guessing which is more likely, by looking if you assumed it was a table, wether it would have a valid vtable (all offsets inside the buffer, etc). I've thought about writing such a dumper, and its output would be more useful than a hex dump, but it would not be able to be very exact. Things like a vector of tables will be pretty hard to detect.

[aardappel]

A better way is to create a "registry" of all possible schemas you know about, and then use the file_identifier to find the schema when you come across a buffer. If the buffer contains no file_identifier, you are SOL though.

[siebeneicher]

Thanks for your feedback, I dont have any other schemas. So most likely I wont try it :(

[ghost]

aardappel, did you ever end up making that dumper? We have a lot of clients switching from protobuf to Flatbuffers and that would be extremely helpful even if it wasn’t exact.

[aardappel]

No, I haven't. Like I said, I think it would be minimally useful. It would be interesting to attempt, though.

I'd highly recommend your clients to at least put a file_identifier in all their schemas, that way you can recognize what type it is, and dump it in very readable JSON. You could use the functionality in registry.h, or even more lightweight is the buffer dumping in minireflect.h.

[d4l3k]

I recently needed to reverse engineer a flat buffer file so I thought I'd leave some hints here for anyone else who needs to try this. Hope it helps. Feel free to poke me and ask questions

Here's my final code for decoding the file: https://gist.github.com/d4l3k/ef2edb288608d2037abfd57e9fb138b9

There's a bunch of trial/error code in here so only some bits of it are useful but the helper methods for reading VTables/various data types are pretty generic.

Binary Format Description

Here's a single doc with all the descriptions of the format: https://github.com/dvidelabs/flatcc/blob/master/doc/binary-format.md

Finding all tables

You can figure out the rough number of tables/types by scanning through the whole file and finding all soffsets that point to a valid location within the file.

    for {
        position, err := r.ReadSOffset()
        if position >= 0 && position < int64(len(data)) {
            entry, err := r.ReadVTable(position)
            log.Printf("candidate %d: %#v", offset, entry)
        }
    }

Once you find all the tables + have the vtable it's pretty easy to just walk the tables and inspect each field to identify what it is.

Find known strings

You can use the strings command to list all strings in a flat buffer file. Once you find the strings you can find all uoffsets that point to the string and then walk back up the file until you find the matching soffset that points to the relevant vtable.

Incrementally decoding

I ended up incrementally walking the entire file bit by bit and alternated between calling r.PrintDebug() and decoding the parts of the tables.

func (r *Reader) ReadTable() error {
    log.Printf("loading table %d", r.Offset())
    vtableAddr, err := r.ReadSOffset()
    if err != nil {
        return err
    }
    vtable, err := r.ReadVTable(vtableAddr)
    if err != nil {
        return err
    }
    log.Printf("vtable %#v", vtable)
    //r.PrintDebug(100)

    if vtable.Position == 1866322 { // ptr to vector
        log.Printf("loading ptr to vector...")
        r.PrintDebug(128)
        vec, err := r.ReadOffset()
        if err != nil {
            return err
        }
        log.Printf("offset %d", vec)
        if err := r.Seek(vec); err != nil {
            return err
        }
        vecLen, err := r.ReadUint32()
        if err != nil {
            return err
        }
        log.Printf("vec length = %d", vecLen)
        var subtables []int64
        for i := 0; i < vecLen; i++ {
            subtable, err := r.ReadOffset()
            if err != nil {
                return err
            }
            log.Printf("subtable %d", subtable)
            if subtable >= r.Len() {
                log.Printf("invalid subtable hmm %d", subtable)
                if i == 0 {
                    continue
                } else {
                    return errors.Errorf("too many invalid")
                }
            }
            subtables = append(subtables, subtable)
        }
        subtables = subtables[1:]

        for _, subtable := range subtables {
            log.Printf("loading subtable %d", subtable)
            if err := r.Seek(subtable); err != nil {
                return err
            }
            if err := r.ReadTable(); err != nil {
                return err
            }
        }

        return nil
   }

This worked pretty well since the file/schema I was reading was fairly small.

Ex:

[4 0 0 0 0 2 0 0 1 0 0 0 1 0 0 0 1 0 0 0 218 143 227 255 28 0 0 0 4 0 0 0 15 0 0 0 116 97 115 107 95 119 105 112 101 114 95 112 111 111 108 0 0 0 0 0 254 143 227 255 32 0 0 0 4 0 0 0 19 0 0 0 105 110 99 101 112 116 105 111 110 95 52 97 46 99 111 110 99 97 116 0 0 0 0 0 38 144 227 255] |ڏ��task_wiper_pool���� inception_4a.concat&���|

• Large values like 204 134 227 255 likely point to a vtable in the file. Once you've identified the vtable for the entry you know how large each entry in the table is as well as where they start/end.
• Values like 04 00 00 00 are likely uoffsets since they're a multiple of 4.
• Values like 19 0 0 0 105 110 99 101 112 116 105 111 110 95 52 97 46 99 111 110 99 97 116 0 0 0 0 0 are likely a vector since there's a uint32 length and some number of bytes then padding at the end. If the item has padding it means a larger data type follows (in this case a soffset which is always aligned).