Cartographer_ros: What is the timestamp format in PBStream

[mjseabright]

Hi, I've managed to pull the trajectory out of a PBStream file but I cannot work out what the timestamps mean.

Exampe:
timestamp: 636533061005937350 pose { translation { x: 1.8459875994728516 y: -4.098475079319226 } rotation { z: -0.999340312949446 w: 0.036317198599333245 } } node_index: 1251
I would have thought the timestamps would corresponded to times in the bag file. So the example above would be somewhere around 1517709302.83 or Feb 04 2018 14:55:02.83.

I noticed a similar timestamp in https://github.com/googlecartographer/cartographer_ros/issues/460 so I don't think it's that I'm decoding it incorrectly.

What does this timestamp mean? What time does it correspond to?

Thank you very much for any insight you can provide.

[mjseabright]

A co-worker has pointed me in the right direction. The Protobuf documentation shows how the timestamp is formed.

The timestamps are relative to 0001-01-01T00:00:00Z rather than the Unix epoch. The units are 1/10000000 seconds.

[ojura]

Use FromUniversal/ToUniversal to convert from/to proto timestamps.

[woshidaye]

@mjseabright Hi, I want to extract data from pbstream. How do you extract trajectory from pbstream? I look forward to your reply.

[mjseabright]

@woshidaye I used this code. It is a little outdated now as I had to change all references of sparse_pose_graph to pose_graph. It may require more changes to be compatible with the latest version of Cartographer.

[woshidaye]

@mjseabright Is the code running through the 'python export_pose_graph.py example.pbstream' command? Why prompt error: 'ImportError: No module named cartographer.mapping' when running?
Is it wrong to run the order?

[mjseabright]

@woshidaye Yes, that is how you run it. It looks like you need to run the commands at the top of that script to generate the required python protobuf descriptor files.

sudo apt-get install protobuf-compiler
roscd cartographer && mkdir py_export
protoc --python_out ./py_export -I . ``find . -iname '*.proto'

The files this generates describe the pbstream format and how to interpret it. Copy the generated files (they will be in the py_export folder) to wherever you are running the export_pose_graph.py script from. That should solve the error you are getting.

[woshidaye]

@mjseabright I follow your steps, but there are still errors. When 'python export_pose_graph.py example.pbstream' is entered at the terminal, it still prompts errors 'ImportError: No module named cartographer.mapping'.

[mjseabright]

Try it with python3. python3 export_pose_graph.py example.pbstream

[woshidaye]

@mjseabright When I ran according to your command, a new error occurred :
File "export_pose_graph.py", line 5, in
import cartographer.mapping.proto.pose_graph_pb2
File "/home/sy/py_export/cartographer/mapping/proto/pose_graph_pb2.py", line 6, in
from google.protobuf import descriptor as _descriptor
ImportError: No module named 'google'
How do I modify it here? Thank you.

[mjseabright]

It's saying you're missing the google module that it is trying to get the protobuf module from.
Try:
pip3 install protobuf
pip3 install google
To get the required module(s).

[woshidaye]

@mjseabright Thank you for your answer. However, there was another error:
Traceback (most recent call last):
File "export_pose_graph.py", line 6, in
import cartographer.mapping.proto.serialization_pb2
File "/home/sy/py_export/cartographer/mapping/proto/serialization_pb2.py", line 16, in
from cartographer.mapping.proto import submap_pb2 as cartographer_dot_mapping_dot_proto_dot_submap__pb2
File "/home/sy/py_export/cartographer/mapping/proto/submap_pb2.py", line 15
from cartographer.mapping.proto.2d import grid_2d_pb2 as cartographer_dot_mapping_dot_proto_dot_2d_dot_grid__2d__pb2
^
SyntaxError: invalid syntax

Looking forward to your reply again. Thank you.

[fsblu]

@woshidaye Could you solve the last error?

[TSRyan]

@woshidaye @fsblu The invalid syntax error is due to the leading digit after the dot (...proto.2d) in

from cartographer.mapping.proto.2d import ...

You can get past it by putting in a leading underscore like this:

from cartographer.mapping.proto._2d import ...

You'll have to rename the associated folders and change it everywhere this happens. You'll also have to change references to SparsePoseGraph and sparse_pose_graph to PoseGraph and pose_graph, respectively, but once that's done, it should at least run.

Having done that, I'm still not sure it works correctly, though. When I try to run it on a test file I get:

$ ./export_pose_graph.py example.pbstream 
Found:  0  trajectories
0 0

If anyone can shed some light on this, I would appreciate it.

[Bitfroest]

@TSRyan
I guess all the data can be obtained using only the serialization because it already serializes the PoseGraph messages.

import cartographer.mapping.proto.serialization_pb2
...
def __init__(...):
    ...
    # create a new list
    self.serdatalist = []

def read_serialized_data():
    ...
    # add current message to list of messages in readable string format
    self.serdatalist.append(str(self.sd.__str__()))

available functions doc

at the end you can write the whole list or just the parts you need to a file.

[TSRyan]

Thanks, @Bitfroest. That will get me where I need to go.

[Bitfroest]

In addition to my previous post you can use the following code snipped to get the PoseGraph as dict

from google.protobuf.json_format import MessageToDict
...
        dict_obj = MessageToDict(self.sd)
        if 'poseGraph' in dict_obj:
            self.serdatalist.append(dict_obj)
        return True

Source: https://stackoverflow.com/questions/19734617/protobuf-to-json-in-python

In case someone else wants to convert the times here is my way.

Range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z.

The units are 1/10000000 seconds.

If you need to convert from those numbers to Unix time, substract 621355968000000000L, which is the Unix epoch expressed in ticks.

Here the python conversion with using the combined information.

>> import time
>> print( time.ctime( (636533061005937350 - 621355968000000000) / 10000000) )
Sun Feb  4 02:55:00 2018
1517709300.593735

Source: https://stackoverflow.com/questions/1832714/18-digit-timestamp

[artemiialessandrini]

MessageToDict is a great tool.

I assume import json could also solve this transformation without additional installation:
print (json.dumps(message))