Tensor2tensor: [feature request] im2markup or im2latex implementation in T2T

[bezigon]

I need a specific T2T example for the im2markup task by @harvardnlp (arXiv) or im2latex task by @untrix (arXiv).

Both has his own approach to solving the problem, it would be interesting to adopt both options.

BTW @harvardnlp also implemented in most universal OpenNMT variant.

[untrix]

Hi @bezigon,
I am the author of im2latex and am trying to understand your request. Are you asking for a new T2T 'model' and 'problem' for the image-to-latex task? Also, can you please let us know what you need this for? I will be happy to implement it if the T2T folks (@rsepassi ) accept this request.

[bezigon]

Hi @untrix! I'm newbie in deep learning, and I admire your work! Specifically in preprocessing and statistical analysis of training data.

To be honest, I would like to create a competitor of Mathpix :-)
But in general, I would like to get experience with translating images into structured text. Here are scopes of this algorithm:

• printed or handwritten mathematical formulas to latex
• musical notes
• html markup from painted web design
• etc.

It is very important to understand better the work of CNN + LSTM + Attention, play with hyperparameters to get a more lightweight model, with the ability to run on a mobile devices.

And I understand that I lack some knowledge, but no harm in trying.

[bezigon]

Why T2T? Because, I believe, T2T will become the supreme algorithm.

[untrix]

Hi @bezigon ,
It seems that you are intending to experiment with the image-to-latex model in order to learn and perhaps build an app. Sounds like a good idea. Feel free to contact me directly by email as needed.

Best