Models: Is there a way to encode data in a TFRecord format using C++ instead of python?

[00krishna]

System information

• What is the top-level directory of the model you are using:

This question relates to both the official and research models.

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  Not applicable.

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Linux Ubuntu 16.04, but issue is cross-platform

• TensorFlow installed from (source or binary):
  TF version 1.11 from binary.

• TensorFlow version (use command below):

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
b'unknown' 1.12.0

• Bazel version (if compiling from source):
  NA, not using Bazel.

• CUDA/cuDNN version:
  CUDA 10. cuDNN 7.4

• GPU model and memory:
  NA - GTX 1080 Ti, 11GB memory.

• Exact command to reproduce:
  NA

  Describe the problem

This question points to an omission in the documentation. The question is whether there is a way to encode data in a TFRecord format using only C++, and not using the the python functions from tf.train. We know that there are python functions to encode data in TFRecord format, including the tf.train.Int64List() and tf.train.BytesList(). But are there similar function in C++? Are there any code samples of anyone doing something like this? Thanks for reviewing my question.

What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

What have I tried:
I have posted on multiple sites, but with no responses.

I posted this question on Stack Exchange:
https://stackoverflow.com/questions/53769229/tensorflow-can-i-create-a-tfrecords-file-using-c

I have posted this question on Reddit:
https://www.reddit.com/r/tensorflow/comments/a7jd0e/does_anyone_know_how_to_create_a_tfrecord_file/

I have posted this question on Quora:
https://www.quora.com/unanswered/Can-I-create-a-TensorFlow-TFRecord-file-from-C-instead-of-Python?__filter__=&__nsrc__=2&__snid3__=3644653184

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

NA

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[00krishna]

@tensorflowbutler Updated as requested.

[00krishna]

So the solution here is that the documentation for writing a TFRecord format file in C++ is in the original Google Protocol Buffers documentation. The link is:

https://developers.google.com/protocol-buffers/docs/cpptutorial#parsing-and-serialization

The problem here is twofold. First, the original protocol buffer documentation makes no reference to the tensorflow TFRecords format. So if you do a Google search on "TFRecord and C++," you will never find the right documentation.

Also, in the protocol buffer documentation, there is no reference to TFRecord format or that TFRecord format is the same as a protocol buffer--or at least is a subset of a protocol buffer. So again, if you search you will not find the right info.

So the solution would be to add the appropriate links in the Tensorflow documentation to the Protocol Buffer documentation. Also, the Protocol Buffer documentation should have an additional section talking about TFRecords and how this code examples for protocol buffers can also encode TFRecords files.

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.