Assimp: FR: Update FBX importer to support new version

[DmitriyYukhanov]

Hey, would be nice to see support for the newer FBX version (7.5).

[kimkulling]

Any special features you have in mind. Or do you only need to increment checked version of FBX?

[DmitriyYukhanov]

Just increment, if new format is fully backwards compatible, just to be able to read FBX exported from modern Autodesk applications (like AutoCAD 2017).

[ZappoMan]

According to the Autodesk site, the file format has changed...

http://help.autodesk.com/view/FBX/2016/ENU/?guid=__files_GUID_2ADB9BCE_15EA_485F_87C2_32D43F2D219D_htm

Updated the FBX file format to 7.5. This new file format is only compatible with the 2016 or later product versions that use 64-bit integers in the binary block headers for large file support (> 2GB). The new file format cannot be imported in 2015 or older product versions. You must set the correct export file format for an FBX file, if it needs to be opened in the 2015 or older product versions.

[ZappoMan]

Although our project doesn't use assimp, we have our own implementation of FBX parsing, I glanced over the assimp code to see if I could make a suggestion on how to support the 7.5 version files.

As far as I can tell from the handful of test files I've seen, the 7.5 version files have 64bit values for the per node: end_offset, prop_count, and prop_length...

I believe that you will want to adjust this file/function: /code/FBXBinaryTokenizer.cpp

// ------------------------------------------------------------------------------------------------
bool ReadScope(TokenList& output_tokens, const char* input, const char*& cursor, const char* end)
{
    // the first word contains the offset at which this block ends
    const uint32_t end_offset = ReadWord(input, cursor, end);

    // we may get 0 if reading reached the end of the file -
    // fbx files have a mysterious extra footer which I don't know
    // how to extract any information from, but at least it always
    // starts with a 0.
    if(!end_offset) {
        return false;
    }

    if(end_offset > Offset(input, end)) {
        TokenizeError("block offset is out of range",input, cursor);
    }
    else if(end_offset < Offset(input, cursor)) {
        TokenizeError("block offset is negative out of range",input, cursor);
    }

    // the second data word contains the number of properties in the scope
    const uint32_t prop_count = ReadWord(input, cursor, end);

    // the third data word contains the length of the property list
    const uint32_t prop_length = ReadWord(input, cursor, end);
...

Pre-7.5 files will use 32bit values for end_offset, prop_count, and prop_length... and post 7.5 files appear to use 64bit values for those.

Good luck!

[kimkulling]

Thanks a lot for the hint!

[VeganPower]

I already have the change done on my PC (hopefully my company will allow to merge them).
Basically at the beginning of the file after the magic word there is an int32_t of flags

const uint32_t flags = ReadWord(input, cursor, input + length);
const uint8_t padding_0 = ReadByte(input, cursor, input + length); // unused
const uint8_t padding_1 = ReadByte(input, cursor, input + length); // unused

and this is what I know about the flags.

enum Flag
{
   e_unknown_0 = 1 << 0,
   e_unknown_1 = 1 << 1,
   e_unknown_2 = 1 << 2,
   e_unknown_3 = 1 << 3,
   e_unknown_4 = 1 << 4,
   e_unknown_5 = 1 << 5,
   e_unknown_6 = 1 << 6,
   e_unknown_7 = 1 << 7,
   e_unknown_8 = 1 << 8,
   e_unknown_9 = 1 << 9,
   e_unknown_10 = 1 << 10,
   e_unknown_11 = 1 << 11,
   e_unknown_12 = 1 << 12,
   e_unknown_13 = 1 << 13,
   e_unknown_14 = 1 << 14,
   e_unknown_15 = 1 << 15,
   e_unknown_16 = 1 << 16,
   e_unknown_17 = 1 << 17,
   e_unknown_18 = 1 << 18,
   e_unknown_19 = 1 << 19,
   e_unknown_20 = 1 << 20,
   e_unknown_21 = 1 << 21,
   e_unknown_22 = 1 << 22,
   e_unknown_23 = 1 << 23,
   e_flag_field_size_64_bit = 1 << 24, // I hope is this one.
   e_unknown_25 = 1 << 25,
   e_unknown_26 = 1 << 26,
   e_unknown_27 = 1 << 27,
   e_unknown_28 = 1 << 28,
   e_unknown_29 = 1 << 29,
   e_unknown_30 = 1 << 30,
   e_unknown_31 = 1 << 31
};

Old header have 1A 00 4C 1D new header is 1A 00 E8 1C.
Seems to work fine with both format.

[kimkulling]

Could you prepare a pull-request for us? I am really busy with all these obj-stuff, which is so slow + all the maintain.stuff. Would be great!

[jpx]

Hi, I faced the same issue with a large file (> 3GB) and would be interested in seeing a pull-request for it. Thanks!

[VeganPower]

I'll do the change again at home and submit from home, so I don't have to ask permission to anyone.

[VeganPower]

Got authorization from my office (Sony is a bit slow sometimes) (^_^)/
https://github.com/assimp/assimp/pull/1042

[kimkulling]

The patch caused some trouble. I started to investigate. When older version are imported somethimes the DoubleWord rad will be used. This will cause a crash because the offset will be wrong for older versions.

[VeganPower]

I think the issue is that I really don't know the what that bit flags
really means. 😥
So probably? I took the wrong bit to recognise the 64 but version.

If you send me a file that cause the problem I can check.
Can we also make a small test case for this?

On 1 Dec 2016 19:14, "Kim Kulling" [e-mail] wrote:

The patch caused some trouble. I started to investigate. When older
version are imported somethimes the DoubleWord rad will be used. This will
cause a crash because the offset will be wrong for older versions.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/assimp/assimp/issues/988#issuecomment-264265244, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHLvRgwDjDjsUBMfpSUcFROdT9Vbe1DRks5rDxyNgaJpZM4Joq8_
.

[kimkulling]

Not wo worry :-)! We will find a flag! I am currently working on a test and
workaround!

Kim

Rosario [e-mail] schrieb am Do., 1. Dez. 2016 um
23:49 Uhr:

I think the issue is that I really don't know the what that bit flags
really means. 😥
So probably? I took the wrong bit to recognise the 64 but version.

If you send me a file that cause the problem I can check.
Can we also make a small test case for this?

On 1 Dec 2016 19:14, "Kim Kulling" [e-mail] wrote:

The patch caused some trouble. I started to investigate. When older
version are imported somethimes the DoubleWord rad will be used. This
will
cause a crash because the offset will be wrong for older versions.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/assimp/assimp/issues/988#issuecomment-264265244,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AHLvRgwDjDjsUBMfpSUcFROdT9Vbe1DRks5rDxyNgaJpZM4Joq8_

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/assimp/assimp/issues/988#issuecomment-264320048, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACNy1M4PMjUTn3fh-MH-0y5c15GE93lFks5rD08EgaJpZM4Joq8_
.

[Eirenliel]

1A 00 is just a part of the header (no one knows what does it mean and it is the same in every FBX) and 4C 1D and E8 1C is the VERSION NUMBER in little-endian (1d4c = 7500, 1ce8 = 7400).

The only differences between 7.4 and 7.5 in format are:

• end_offset, prop_count and prop_length are 8 byte-long in 7.5
• NULL-record at the end of a node is 25 zeroes in 7.5 instead of 13 in 7.4

Well, in format... There are ofc some changes in objects, but they are mostly compatible.

[VeganPower]

It's a bit weird that they added all that padding and fixed part. But having the version number make perfect sense. Thanks. @kimkulling do you want me to update the code or are you doing by your self?

[space2]

I also have a problem with version 7500 in an FBX file... although I have to admit, this is the first time working with FBX files ;-)
Anyway, based on the previous comments, I tried changing the 64bit detection code. Well, I still have some issues with the files, but at least it doesn't stop at the beginning, so I think this is a step in the right direction.

FBXBinaryTokenizer.cpp.patch.txt

[samizzo]

Has there been any progress on getting this in?

[VeganPower]

Checking the code it seems to correctly support 64bit headers. Are there more problems?

[samizzo]

Ah actually I just realised I'm out of date, woops! I'm using Assimp.Net and it ships with 3.1.1. I'll try the latest.