Keras: Audio Classification on different dataset

[henry090]

Hi, I just want to try an audio classification task on different dataset but get an error while training.
Could not figure out what should be changed. Could you help with that?

Epoch 1/10
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  DataLossError: Attempted to pad to a smaller size than the input element.
     [[node IteratorGetNext_22 (defined at /Library/Frameworks/R.framework/Versions/3.5/Resources/library/keras/python/kerastools/generator.py:7)  = IteratorGetNext[output_shapes=[[?,98,257,?], [?,?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator_22)]]

Caused by op u'IteratorGetNext_22', defined at:
  File "/Library/Frameworks/R.framework/Versions/3.5/Resources/library/keras/python/kerastools/generator.py", line 7, in dataset_generator
    batch = iter.get_next()
  File "/Users/turgutabdullayev/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 421, in get_next
    name=name)), self._output_types,
  File "/Users/turgutabdullayev/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2069, in iterator_get_next
    output_shapes=output_shapes, name
In addition: Warning messages:
1: In normalizePath(path) : path[1]="": No such file or directory
2: In normalizePath(path) : path[1]="": No such file or directory

[skeydan]

Hi, please provide the exact code and dataset you're using (if possible, a small sample from the dataset that would produce the error).

[henry090]

Hi, please provide the exact code and dataset you're using (if possible, a small sample from the dataset that would produce the error).

The audio file is on my GitHub page,
And code is here

[skeydan]

To make sure I'm running the same files as you, could you indicate which should be the actual paths used?

The script says

df <- data_frame(
  fname = files, 
  class = fname %>% str_extract("cats_dogs/train/.*/") %>% 
    str_replace_all("cats_dogs/train", "") %>%
    str_replace_all("/", ""),
  class_id = class %>% as.factor() %>% as.integer() - 1L
)

but the directories I get from the download are cats and dogs, with raw .wav files inside.

[henry090]

You are absolutely right. Sorry for that, I just shared a test (cause it is small) folder on GitHub and forgot to change path within script... Could you create a new folder and place them there, like:

files <- fs::dir_ls(
  path = "/Users/turgutabdullayev/Downloads/**new_folder**", 
  recursive = TRUE, 
  glob = "*.wav"
)


df <- data_frame(
  fname = files, 
  class = fname %>% str_extract("**new_folder**/.*/") %>% 
    str_replace_all(**"new_folder"**, "") %>%
    str_replace_all("/", ""),
  class_id = class %>% as.factor() %>% as.integer() - 1L
)

[skeydan]

thanks, I see!

Well, from "printf-debugging"

print(spectrogram)

I see it fails on the 3rd file (for me), and somehow it's the value of n_chunks in

dataset_padded_batch(batch_size, list(shape(n_chunks, fft_size, NULL), shape(NULL)))

that doesn't work for this file.
If you change n_chunks to NULL, you get rid of this but the model isn't happy any more:

 ValueError: Error when checking input: expected conv2d_41_input to have shape (98, 257, 1) but got array with shape (377, 257, 1)

I think you should dig deeper into that whole __spectrogram__ thing and see what's going on (keep us posted too :-))

[henry090]

I think you should dig deeper into that whole spectrogram thing and see what's going on (keep us posted too :-))

After some days I could figure out many things =)

n_chunks              # We should not touch this

Instead, we should change the window_stride_ms = 10 to window_stride_ms = 20 _(I tested it on different dataset from here)._ So, for cats and dogs it should be the same strategy.

library(tfdatasets)

audio_ops <- tf$contrib$framework$python$ops$audio_ops

data_generator <- function(df, batch_size, shuffle = TRUE, 
                           window_size_ms = 28, window_stride_ms = 20)

That's it =)

This is for those who want to test audio class-n on different dataset:

_Change that value to different numbers_ in order to get the same shape here:

model %>%  
  layer_conv_2d(input_shape = c(49, 257, 1), 
                filters = 32, kernel_size = c(3,3), activation = 'relu') %>% 

# Otherwise you will get an error

ValueError: Error when checking input: expected conv2d_41_input to have shape (98, 257, 1) but got array with shape (377, 257, 1)

I got this shape by the following:

window_size_ms <- 28
window_stride_ms <- 20
window_size <- as.integer(16000*window_size_ms/1000)
stride <- as.integer(16000*window_stride_ms/1000)
fft_size <- as.integer(2^trunc(log(window_size, 2)) + 1)
n_chunks <- length(seq(window_size/2, 16000 - window_size/2, stride))

If you change n_chunks to NULL

and you can test batch structure like this:

sess <- tf$Session()
batch <- next_batch(ds_train) #error
str(sess$run(batch))

BTW, everytime you should get the same shape.

[skeydan]

Great you were able to sort it out and make it work!

[I'd also like to look a bit into this when I get to it - which probably won't happen immediately - so I'll let it open for now.]

[henry090]

I'd also like to look a bit into this

Please share your solution later because I am facing errors again.

[skeydan]

Sure! It may take a bit before I get to it though.

[dfalbel]

I am guessing the problem is probably due to some audio files have different sizes, and consequently their spectrograms have different dimensions.

I think there are 2 main options:

• Parametrizing the spectrogram creation on the audio lenght.
• Reshaping the spectrogram before getting it into the model. (there are some tf ops to reshape images, and you can consider the spectrogram like an image.)

[henry090]

Hi, I just took code snippet from this blog and would like to know whether it is the right way to solve this issue.

data_generator <- function(df, batch_size, shuffle = TRUE, 
                           window_size_ms = 2, window_stride_ms = 30) {

  #window_size <- as.integer(16000*window_size_ms/1000)
  #stride <- as.integer(16000*window_stride_ms/1000)
  #fft_size <- as.integer(2^trunc(log(window_size, 2)) + 1)
  #n_chunks <- length(seq(window_size/2, 16000 - window_size/2, stride))

  ds <- tensor_slices_dataset(df)

  if (shuffle) 
    ds <- ds %>% dataset_shuffle(buffer_size = 10)  

  ds <- ds %>%
    dataset_map(function(obs) {

      # decoding wav files

      #audio_binary <- tf$read_file(tf$reshape(obs$fname, shape = list()))

      audio_binary = tf$read_file(obs$fname)

      wav <- audio_ops$decode_wav(audio_binary, desired_channels = 1)

      # create the spectrogram
      spectrogram <- audio_ops$audio_spectrogram(
        wav$audio, 
        window_size = window_size, 
        stride = stride,
        magnitude_squared = TRUE
      )

      #spectrogram <- tf$log(tf$abs(spectrogram) + 0.01)
      #spectrogram <- tf$transpose(spectrogram, perm = c(1L, 2L, 0L))

      #brightness <- tf$placeholder(tf$float32, shape=list())
      brightness<-100
      mul <- tf$multiply(spectrogram, brightness)
      # Normalize pixels
      min_const<-tf$constant(255)
      minimum <- tf$minimum(mul, min_const)
      # Expand dims so we get the proper shape
      expand_dims <- tf$expand_dims(minimum, -1L)

      # Resize the spectrogram to input size of the model
      resize <- tf$image$resize_bilinear(expand_dims, c(128L, 128L))

      # Remove the trailing dimension
      squeeze <- tf$squeeze(resize, 0L)

      # Tensorflow spectrogram has time along y axis and frequencies along x axis
      # so we fix that
      flip <- tf$image$flip_left_right(squeeze)
      transpose <- tf$image$transpose_image(flip)

      # Convert image to 3 channels, it's still a grayscale image however
      grayscale <- tf$image$grayscale_to_rgb(transpose)

      # Cast to uint8 and encode as png
      cast <- tf$cast(grayscale, tf$uint8)
      png <- tf$image$encode_png(cast)
      # transform the class_id into a one-hot encoded vector
      response <- tf$one_hot(obs$class_id, 2L)

      list(png, response)
    }) %>%
    dataset_repeat()

  ds <- ds %>% 
    dataset_padded_batch(batch_size, list(shape(), shape(NULL)))

  ds
}

After that, train dataset looks like this:

List of 2
 $ :List of 10
  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..$ :�PNG


  ..- attr(*, "dim")= int 10
 $ : num [1:10, 1:2] 1 1 0 0 1 1 0 0 0 0 ...

[dfalbel]

You shouldn't return a PNG encoded vector to the model. I think you could return the result from
squeeze instead.

[henry090]

Yes, it works now :-) Thank you so much!

[skeydan]

I'm closing this as it was solved (planning on a short background-explaining post some time later this year :-))

[birbritto]

I'd also like to look a bit into this

Please share your solution later because I am facing errors again.

I am having the same issue, it seems that the error seems to happen in this bit:

ds %>%
dataset_padded_batch(
batch_size = batch_size,
padded_shapes = list(tf$stack(list(
n_periods, n_coefs,-1L
)),
tf$constant(-1L, shape = shape(1L))),
drop_remainder = TRUE
)

[birbritto]

I read in this post it could be due to the shape of the padded shapes

[birbritto]

@mrry