Flatbuffers: Dart produces flatbuffers which cannot be verified in C++ [C++, Dart, 1.9.0+]

[paulreimer]

Very odd behaviour when generating flatbuffers from Dart for consumption from C++ (anywhere, really). Certain combinations of field sizes produce binary data which C++ can verify, but other combinations (e.g. just add or remove a few characters from a string, or omit/add an extra field), and then the buffer does not verify anymore. Binary-wise -- valid or not -- the binaries generated from dart never seem to match the output from flatc, just upon a visual inspection with xxd.

So for example, a user name field set to "Foo Bar1" will not verify, but a user name of "Foo Bar" will. ("Foo Bar123" might be good again, who knows?!)

I made a reproducible test case (on my dev machine, but I originally ran into it on an embedded platform on the C++ side and cloud platform on the Dart side, but that is difficult to repro).

tree .
.
├── pubspec.yaml
├── repro.cpp
├── repro.dart
└── repro.fbs

cat pubspec.yaml:

name: repro

environment:
  sdk: '>=2.0.0-dev.55.0 <2.0.0'

dependencies:
  flat_buffers: ^1.9.0

cat repro.cpp:

#include <flatbuffers/flatbuffers.h>

#include <fstream>

#include "repro_generated.h"

auto main()
  -> int
{
  std::ifstream stream("repro.fb", std::ios::in | std::ios::binary);
  std::vector<uint8_t> repro_fb(
    (std::istreambuf_iterator<char>(stream)),
    std::istreambuf_iterator<char>()
  );

  flatbuffers::Verifier verifier(repro_fb.data(), repro_fb.size());

  if (verifier.VerifyBuffer<Repro::User>("repr"))
  {
    printf("valid user = true\n");
  }

  return 0;
}

cat repro.dart:

import "dart:io";
import "repro_repro_generated.dart" as Repro;

const String REPRO_FILE_IDENTIFIER = "repr";

void main() {
  final userBuilder = new Repro.UserObjectBuilder(
    name: "Foo Bar1",
    id: "123",
  );

  final userBytes = userBuilder.toBytes(REPRO_FILE_IDENTIFIER);
  if (userBytes == null) {
    throw ("Invalid (empty) User flatbuffer");
  }

  final filename = "repro.fb";
  new File(filename).writeAsBytes(userBytes).then((File file) {
    final user = new Repro.User(userBytes);
    print("got User: ${user}");
  });
}

cat repro.fbs:

namespace Repro;

table User
{
  name:string;
  id:string;
}

root_type User;
file_identifier "repr";
file_extension "fb";

To run it, do this:

pub get
flatc --cpp --dart -o . repro.fbs
c++ -std=c++14 repro.cpp -o repro
dart repro.dart && ./repro

The behaviour is erratic but if you add some characters on the end of the name field in the repro.dart file, the test will pass or fail based on the length (maybe it has to do with alignment but I didn't expect that for a string?)

[aardappel]

If it is caused by just a few bytes being added or removed, it sounds like the Dart implementation has an alignment problem of some sort.

[aardappel]

@dnfield can you have a look at this?

[dnfield]

I'll take a look.

I will say that both the Dart and the JS implementations would sometimes produce different binaries than the C++ implementation, but I hadn't run into issues verifying/decoding them. That said, I agree that this sounds like an alignment issue.

[dnfield]

The C++ code is able to correctly get the full user ID -

  auto user = Repro::GetUser(repro_fb.data());
  printf("%s %s\n", user->id()->c_str(), user->name()->c_str());

gives the expected result. But the verification fails as reported.

C++ code to write the same data gives four additional null bytes of padding at the end of the file for "Foo Bar1" (so a 52 byte file vs a 48 byte file from Dart). This is something missing in the Dart code, although it doesn't seem to impact readability (just verifiability). I'll look into it some more.

[paulreimer]

Oh sorry I should have mentioned that the fields do read OK when I receive a message (but it is from a service over the network so I would like to verify).

Also, good call on those extra 4 bytes. I tried hexedit'ing 4 00s onto the end of one of the failing repro.fb buffer files and it did verify now!

I'll note that even beyond the padding, the buffers are not binary identical as generated from C++ / flatc, the contents of the set of bytes (before what appears to be the string data from the fields) is slightly different. But if C++ verifies that's good enough for me!

[dnfield]

The C++ code ensures the String's null terminator will be contained within the buffer. Dart code was not doing that.

This wasn't causing any problems in reading (although I suppose it could have?), but it did cause problems in verifying (which was trying to make sure the string is null terminated properly).

Should have a PR for this shortly, but I'm noticing the tests don't seem to be running correctly right now and want to see what's going on with that.

[dnfield]

Also, it wasn't causing any problems if the String happened to be the right length to leave some space in the buffer (e.g. "Foo Bar" in your example) - but your example ("Foo Bar1") just had the String ending exactly at the end of the allocated space.

[paulreimer]

Awesome! Can't wait to check it out. Does publishing packages to pub happen when flatbuffers makes an official release, or could this specific issue maybe trigger a patch release for Dart only?

[dnfield]

@aardappel just has to run the publish script

[aardappel]

just ran publish.sh

[paulreimer]

Yes! Just confirmed that buffers generated from Dart in 1.9.2 verify successfully in C++ now!