Deeplabcut: Annotation method (skipping Step1 - using different image types or already extracted frames)

[arnitkun]

According to the readme I have tried to annotate some images, but the "format" that I seem to be getting differs from what I see in the demo csv files by a lot.

As I am using one file per folder, example from my Results.csv:

 ,Label,Area,Mean,Min,Max,X,Y,Slice
1,IMG_20180523_122904.jpg,0,15,15,15,1500.000,1638.000,1
2,IMG_20180523_122904.jpg,0,210,210,210,2160.000,1848.000,1
3,IMG_20180523_122904.jpg,0,15,15,15,2808.000,1740.000,1
4,IMG_20180523_122904.jpg,0,128,128,128,1182.000,1710.000,1
.......and so on

Although this might be something I should ask Fiji guys, I would still like to know how you did the annotations.( On a side note, I think the label thing is weird; why is it coming as image name when I have done that as numbers?)

Also, as Step 1,2,3,4 do not throw any error and generate the specific "separate part" .csv file,
running the train.py gives me with :

Config:
{'all_joints': [[0], [1], [2]],
 'all_joints_names': ['base', 'pivot', 'wrist'],
 'batch_size': 1,
 'crop': False,
 'crop_pad': 0,
 'dataset': '../../UnaugmentedDataSet_movingjuly26/moving_kinova95shuffle1.mat',
 'dataset_type': 'default',
 'display_iters': 1000,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '../../pretrained/resnet_v1_50.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 0.05,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'max_input_size': 1000,
 'mean_pixel': [123.68, 116.779, 103.939],
 'mirror': False,
 'multi_step': [[0.005, 10000],
                [0.02, 430000],
                [0.002, 730000],
                [0.001, 1030000]],
 'net_type': 'resnet_50',
 'num_joints': 3,
 'optimizer': 'sgd',
 'pos_dist_thresh': 17,
 'regularize': False,
 'save_iters': 50000,
 'scale_jitter_lo': 0.5,
 'scale_jitter_up': 1.5,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': './snapshot',
 'stride': 8.0,
 'use_gt_segm': False,
 'video': False,
 'video_batch': False,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
2018-06-26 10:31:07.888880: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-06-26 10:31:07.983375: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-06-26 10:31:07.983891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.8095
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 5.50GiB
2018-06-26 10:31:07.983939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "../../../train.py", line 47, in load_and_enqueue
    batch_np = dataset.next_batch()
  File "/root/deep3/DeepLabCut/pose-tensorflow/dataset/pose_dataset.py", line 158, in next_batch
    imidx, mirror = self.next_training_sample()
  File "/root/deep3/DeepLabCut/pose-tensorflow/dataset/pose_dataset.py", line 138, in next_training_sample
    self.curr_img = (self.curr_img + 1) % self.num_training_samples()
ZeroDivisionError: integer division or modulo by zero

EDIT: Also I annotated my images using Fiji and as specified.

[imagejan]

It seems like the example csv files in https://github.com/AlexEMG/DeepLabCut/tree/master/Generating_a_Training_Set/data-reaching/reachingvideo1 are actually _tab-separated_ files, not _comma-separated_ ones.

@Anitkrun can you try saving your results table from Fiji with an xls extension, and then renaming the file to csv? This way, Fiji (or ImageJ1, actually) saves your data tab-separated, as is apparently required for DeepLabCut...

[arnitkun]

why didn't I see such an obvious thing.
I'll create the annotations in .xls and then post the results. BTW, somehow .xls files DO appear beside the csv files in /data-<Task name>/<video-name>

[AlexEMG]

Dear Anitkrun,

I think it is because your images are not pngs. Everything else should be fine and the fix is straightforward (see below). Anyway, _I would recommend to always check the labels (output of Step3) before you proceed to training, just to makes sure the importing of all labels is correct._

@imagejan thanks for your suggestions!

The code does not rely on _png_'s, but they are "baked in" a little bit. See line 88 in Step2. If you use different files, then you need to update:

files = [
fn for fn in os.listdir(os.curdir)
if ("img" in fn and ".png" in fn and "_labelled" not in fn)
]

Also in Step3: line 84.

files = [
    fn for fn in os.listdir(os.curdir)
    if ("img" in fn and **".png"** in fn and "_labeled" not in fn)
]

• one can fix this by just exchanging the endings. I understand that several users start from Step2 (rather then 1 as they already have some frames that might be in a different format). As I said this is easy, but one needs to update those lines then.

I think in a future release, I will make the filetype a variable in myconfig (just like for videos, where one can select mp4 / avi in myconfig_analysis etc.). However, this would still need to be seamlessly integrated with step1.

Minor points:

• the code can import comma, or tab separated _csv_s, see https://github.com/AlexEMG/DeepLabCut/issues/10
• we use Fiji (rather than imageJ) and it seems that some of the elements in the csv file are different (although again, I don't think this is the problem).

Cheers,
Alexander

[AlexEMG]

"On a side note, I think the label thing is weird; why is it coming as image name when I have done that as numbers?)"

That might be ImageJ? But that is not bad, could be used for importing!

[arnitkun]

would recommend to always check the labels (output of Step3)

I did ran Step3....py, there were no issues.

Aren't fiji and imagej almost the same thing?

[imagejan]

Aren't fiji and imagej almost the same thing?

Yes, absolutely. When I referred to "ImageJ1" that is saving your data, I meant to say that it is the legacy layer in Fiji (which contains both ImageJ1 and ImageJ2) that's responsible for saving results tables.

The behavior of saving as CSV is no different between Fiji and a "vanilla" ImageJ 1.x installation.
In that sense the statement "we use Fiji (rather than imageJ)" is irrelevant here (and you both seem to use Fiji anyways).

What might however cause differences is the current settings in Edit > Options > ..., so in order to guarantee reproducible behavior across systems, you both should try with Edit > Options > Reset... to make sure you don't have any personalized settings active.

[AlexEMG]

Agreed: but comma separated or tab separated etc. should all work. (Is that true for you @imagejan?) I think jpg was the problem.

@imagejan : Do you know what settings are necessary to always also export the image address? [seems like you know Fiji well]. That could make the importing easier...

[AlexEMG]

In the latest push, I added a variable in the configuration file called imagetype=".png" that simplifies the use of other image formats than png. Just change the ending to whatever format you are using, when you start with step 2 and want to import other image types.

If you are skipping step1 (i.e. don't use our png file extractor), then you need to do several things (to make sure that the csvs from Fiji and the images are correctly imported in Step2 and subsequently processed in Step3 & 4.)

A) change the file-ending in the myconfig file to the image format you are using: imagetype=".png".
B) make sure that the folder structure is like after "our step1", i.e. a create a folder with "data-TASKNAME" that contains several subfolders each containing images + annotation csv files
C) Ordering. It is key that the order of the Fiji annotation files (csv) and the python loading of the images will match! For this purpose we suggest you either carefully check that manually, or just use a scheme like e.g.:
img0000.jpg
img0001.jpg
...
img9999.jpg
etc. to be sure! (make sure the number of digits is constant, i.e. pad img1.jpg to img0001.jpg).

Then everything should work! However, make sure you get labeled images after Step3 that match what you annotated in Fiji, before you proceed to training DeepLabCut!

P.S. Why should "img" be a subset of the filename? - Consider line 88 of https://github.com/AlexEMG/DeepLabCut/blob/master/Generating_a_Training_Set/Step2_ConvertingLabels2DataFrame.py

fn for fn in os.listdir(os.curdir)
if ("img" in fn and imagetype in fn and "_labelled" not in fn)
]

this is why. The same import is done in Step3. Perhaps in the next push we plan to make this more general by removing this affix. However, this is the current naming convention.

[imagejan]

Sorry for not replying earlier.

@AlexEMG wrote:

Do you know what settings are necessary to always also export the image address?

There is the Display label option in Analyze > Set Measurements...; and there are options in Edit > Options > Input/Output... that define if row and/or column headers are saved.