Tesseract: loop Infinity "Compute CTC targets failed!"

[xlight]

run lstmtraining --script_dir langdata --debug_interval 0 --U ../tessdata/Han.unicharset --model_output testOutput --continue_from ../impact_from_full/chi_sim.lstm --train_listfile traindata/caipiao.train_files.txt --target_error_rate 0.2 --learning_rate 10e-5 --net_spec '[1,0,0,1 Ct5,5,16 Mp3,3 Lfys64 Lfx128 Lrx128 Lfx384 O1c1]' --net_mode 192 --perfect_sample_delay 19
and it print:
`
Loaded file ../impact_from_full/chi_sim.lstm, unpacking...
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Continuing from ../impact_from_full/chi_sim.lstm
Loaded 1/1 pages (1-1) of document /data/docker-tesseract4/training/traindata/testTrain.real.exp431.lstmf

Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
`
and loop infinity print "Compute CTC targets failed!", no file wrote to path "testOutput"

lstmf file : here

[Shreeshrii]

You have combined the commands for two types of training ---

On one side you are trying to build upon an existing model ...

--U ../tessdata/Han.unicharset \
--continue_from ../impact_from_full/chi_sim.lstm \

and on the other you are specifying the full network spec like training from scratch.

--net_spec '[1,0,0,1 Ct5,5,16 Mp3,3 Lfys64 Lfx128 Lrx128 Lfx384 O1c1]' 

You have provided an lstmf file, but not the corresponding unicharset which should have been generated when the lstmf file was built.

[Shreeshrii]

Assuming you are training on ubuntu, follow the method below to test your setup. Then you can modify it with your training text, fonts, lstmf files etc.

Look in the /tmp/ directory to see the generated box/tiff pairs etc.

Edit the config file in ../langdata/chi_sim/chi_sim.config
and comment out the following line
# tessedit_load_sublangs chi_sim_vert

run the following commands from your tesseract-ocr directory, make sure that your relative directory structure matches the commands.

sudo apt-get install fonts-wqy-zenhei

fc-cache -vf

The above is to install a simplified chinese font and update font cache.

text2image --find_fonts \
--fonts_dir  /usr/share/fonts/ \
--text ../langdata/chi_sim/chi_sim.training_text \
--min_coverage .5  \
--outputbase ../langdata/chi_sim/chi_sim \
|& grep raw | sed -e 's/ :.*/" \\/g'  | sed -e 's/^/  "/' >../langdata/chi_sim/fontslistlocal.txt

(copy the font names from ../langdata/chi_sim/fontslistlocal.txt in the command below)

  training/tesstrain.sh \
--fonts_dir  /usr/share/fonts/ \
 --lang chi_sim   \
 --linedata_only \
 --noextract_font_properties \
 --exposures "0"    \
 --langdata_dir ../langdata \
 --tessdata_dir ../tessdata  \
 --output_dir ~/tesstutorial/chi_sim  \
 --fontlist   "WenQuanYi Zen Hei Medium" \
  "WenQuanYi Zen Hei Mono Medium" \
  "WenQuanYi Zen Hei Sharp Medium" 

  mkdir ~/tesstutorial/xlight

  combine_tessdata -e ../tessdata/chi_sim.traineddata \
   ~/tesstutorial/xlight/chi_sim.lstm

to replace a layer, use the following command

lstmtraining  \
 -U ~/tesstutorial/chi_sim/chi_sim.unicharset \
  --script_dir ../langdata \
  --debug_interval 0 \
  --continue_from ~/tesstutorial/xlight/chi_sim.lstm \
   --append_index 5 --net_spec '[Lfx256 O1c105]' \
  --learning_rate 10e-5 \
  --net_mode 192 \
  --train_mode 80 \
  --perfect_sample_delay 19 \
  --model_output ~/tesstutorial/xlight/chi_sim \
  --train_listfile ~/tesstutorial/chi_sim/chi_sim.training_files.txt \
  --target_error_rate 0.1

to finetune, use the following command

lstmtraining  \
  --debug_interval 0 \
  --continue_from ~/tesstutorial/xlight/chi_sim.lstm \
  --model_output ~/tesstutorial/xlight/chi_sim \
  --train_listfile ~/tesstutorial/chi_sim/chi_sim.training_files.txt \
  --target_error_rate 0.1

[Shreeshrii]

Here is the log of lstmtraining from the last command above

shree@ALL-IN-1-TOUCH:/mnt/c/Users/User/shree/tesseract-ocr$ lstmtraining  \
>  -U ~/tesstutorial/chi_sim/chi_sim.unicharset \
>   --script_dir ../langdata \
>   --debug_interval -1 \
>   --continue_from ~/tesstutorial/xlight/chi_sim.lstm \
>    --append_index 5 --net_spec '[Lfx256 O1c105]' \
>   --learning_rate 10e-5 \
>   --net_mode 192 \
>   --train_mode 80 \
>   --perfect_sample_delay 19 \
>   --model_output ~/tesstutorial/xlight/chi_sim \
>   --train_listfile ~/tesstutorial/chi_sim/chi_sim.training_files.txt \
>   --target_error_rate 0.01
Loaded file /home/shree/tesstutorial/xlight/chi_sim.lstm, unpacking...
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Continuing from /home/shree/tesstutorial/xlight/chi_sim.lstm
Mirror 』 of 『 is not in unicharset
Appending a new network to an old one!!Setting unichar properties
Setting properties for script Common
Setting properties for script Latin
Setting properties for script Han
Warning: given outputs 105 not equal to unicharset of 379.
Num outputs,weights in serial:
  Lfx256:256, 394240
  Fc379:379, 97403
Total weights = 491643
Built network:[1,0,0,1[C5,5Ft16]Mp3,3Lfys64Lfx128Lrx128Lfx256Fc379] from request [Lfx256 O1c105]
Training parameters:
  Debug interval = -1, weights = 0.1, learning rate = 0.0001, momentum=0.9
Loaded 280/280 pages (1-280) of document /home/shree/tesstutorial/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Medium.exp0.lstmf
Loaded 325/325 pages (1-325) of document /home/shree/tesstutorial/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Mono_Medium.exp0.lstmf
Loaded 281/281 pages (1-281) of document 

with --debug_interval 0 you will see console output similar to following:

At iteration 100/100/100, Mean rms=3.449%, delta=54.706%, char train=103.678%, word train=100%, skip ratio=0%,  New worst char error = 103.678 wrote checkpoint.

At iteration 200/200/200, Mean rms=3.329%, delta=51.386%, char train=101.839%, word train=100%, skip ratio=0%,  New worst char error = 101.839 wrote checkpoint.

At iteration 300/300/300, Mean rms=3.292%, delta=50.404%, char train=101.222%, word train=100%, skip ratio=0%,  New worst char error = 101.222 wrote checkpoint.

with --debug_interval -1 you will see console output similar to following:

File /tmp/tmp.KMwsKD065e/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Medium.exp0.lstmf line 176 :
Mean rms=3.291%, delta=50.383%, train=101.218%(100%), skip ratio=0%
Iteration 301: ALIGNED TRUTH : 威胁 意见俄语-精灵 51 目的 快报 茯苓用品 慷慨沓 条目 请马鞍山中心
Iteration 301: BEST OCR TEXT :
File /tmp/tmp.KMwsKD065e/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Mono_Medium.exp0.lstmf line 185 :
Mean rms=3.291%, delta=50.378%, train=101.214%(100%), skip ratio=0%
Iteration 302: ALIGNED TRUTH : CPU ~ 内部洼地继上传技术 妄想嫁网易中国法上饶楹联、商城9 兵马俑blog
Iteration 302: BEST OCR TEXT :

The BEST OCR TEXT will slowly start improving and get closer to the Aligned truth as training continues.

File /home/shree/tesstutorial/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Mono_Medium.exp0.lstmf line 303 :
Mean rms=2.96%, delta=41.413%, train=93.745%(99.404%), skip ratio=0%
Iteration 4811: ALIGNED TRUTH : 严洁猗大全局(濠 境内现栈蜻蜓点水 花园〉 、、ˇ癫芊个今天佰草集 Rights
Iteration 4811: BEST OCR TEXT : 世渗大全层源 土现构广 杖团    存介代天作 Rihts
File /tmp/tmp.KMwsKD065e/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Sharp_Medium.exp0.lstmf line 37 :
Mean rms=2.96%, delta=41.414%, train=93.741%(99.404%), skip ratio=0%
Iteration 4812: ALIGNED TRUTH : 糟蹋 提前心00类咽喉晕Utilization 发主持发现保障四16 式诠 Traditional 农业
Iteration 4812: BEST OCR TEXT : 踵 才00呀咿Utiiiatio 发扬现例陶同16 该 Traditioha 安世
File /tmp/tmp.KMwsKD065e/chi_sim/chi_sim.WenQuanYi_Zen_Hei_Medium.exp0.lstmf line 7 :
Mean rms=2.958%, delta=41.371%, train=93.686%(99.404%), skip ratio=0%
Iteration 4813: ALIGNED TRUTH : 错误京34 淅淅MP3恐慌 Medicine DEVELOPMENT 剂手机测绘屉阑尾绕住房的铅
Iteration 4813: BEST OCR TEXT : 锚该4 渗渊P edicine EELPNENT 刃杂结结价的钮

[xlight]

thanks for reply, and I will try your suggest today.

my lstmf provided above is not made by text2image. It's made by imagemagick base on a pic from camera.
and box file is written manually.
I am trying to fine tune base on chi_sim.lstm with some pictures from camera.

Is there some thing wrong that I made my pic by imagemagick?

[Shreeshrii]

For finetune the command is much shorter. However your lstmf file gives the CTC errors

$ combine_tessdata -e ../tessdata/chi_sim.traineddata \
>   ./chi_sim.lstm
Extracting tessdata components from ../tessdata/chi_sim.traineddata
Wrote ./chi_sim.lstm

$ lstmtraining  \
>   --debug_interval -1 \
>   --continue_from ./chi_sim.lstm \
>   --model_output ./testOutput  \
>   --train_listfile ./caipiao.train_files.txt \
>   --target_error_rate 0.01

Loaded file ./chi_sim.lstm, unpacking...
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Continuing from ./chi_sim.lstm
Loaded 1/1 pages (1-1) of document testTrain.real.exp431.lstmf
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!
Compute CTC targets failed!

I understand you used an image file taken as pic on camera and edited by imagemagick and wrote box file manually.

What dpi is your image?
Did you use makebox to create the box file and then edit it?
Did you add tab characters to mark end of line in box file?
How did you create an lstmf file from it?

[Shreeshrii]

Also see earlier discussion on https://github.com/tesseract-ocr/tesseract/issues/591

[xlight]

Bravo! It can be trained after add some tab .

---

What dpi is your image?
The dpi info in image meta is 72. The accurate dpi should be 260~300.
Did you use makebox to create the box file and then edit it?
I use jTessBoxEditorFX to create it.
Did you add tab characters to mark end of line in box file?
I did not edit the box file after saving it by jTessBoxEditorFX
How did you create an lstmf file from it?
I created it by tesseract xxx.tif xxx lstm.train

[Shreeshrii]

New version of jtessboxeditor has the option to mark EOL while editing box files.

Glad it works after adding the tabs.

[dzjin]

@Shreeshrii are the "Compute CTC targets failed!" message benign to fine-tuning? I have the same output as you do.

Also, to be _certain_, Tesseract 4 box files use tab character before \n at EOL?