Neural-style: Hours to work?

Created on 1 Mar 2016 · 16Comments · Source: jcjohnson/neural-style

I think I've hit a bottleneck here.

I'm using Ubuntu 15.10, no CUDA drivers for me in nVidia's site. When I try to run on a simple image (512x512px), I've waited hours (and I'm talking about 4 - 5 hours) to START the iterations...

When it started iterating, after about 2 hours, it was still on iteration 3 / 1000. Then, I quit.

My CPU is an Intel(R) Core(TM) i7-4700MQ. I have 16gb RAM. What's possibly wrong?

Source

mauricioszabo

Most helpful comment

For ubuntu 16.04; I solved this by installing openBLAS. Seems that the default one that comes with ubuntu only supports single threaded computation.

git clone https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS
make CC=clang
sudo make PREFIX=/usr/local install

Then reinstall torch and go!

Bubbad on 20 Nov 2016

👍2

All 16 comments

On my computer with an i5-4690k, the iterations begin immediately and take about 2-3 seconds each, for context.

Maurdekye on 1 Mar 2016

That doesn't sound right - are you sure you passed -print_iter 1 to make sure it's printing?

Are you maybe running out of system memory and swapping to disk? That would kill performance.

On my i7-4790k, iterations take just a couple seconds each in CPU mode.

jcjohnson on 1 Mar 2016

-print_iter 1 is being passed, and it's not swapping - I don't even have a swap space configured.
I think it's a problem with Ubuntu 15.10, because using a VM like mentioned in this post: http://blog.gruffdavies.com/tag/neural-style/, things are MUCH faster...

mauricioszabo on 1 Mar 2016

I'm having a generals slowness in my vm even with a 14.04:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty

16 cores, 16gb ram, 6+ hours to get one image https://i.imgur.com/FNYqu3c.jpg (time+ is cumulative, but that was time to the first iteration)

if I have to take a guess, it's certain cloud providers removing simd instruction from the cpu but!

mine does advertise all the sse goodies:

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 29
model name      : Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz
stepping        : 1
microcode       : 0x29
cpu MHz         : 2393.890
cache size      : 12288 KB
physical id     : 30
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 30
initial apicid  : 30
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 x2apic hypervisor lahf_lm dtherm
bogomips        : 4787.78
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

still they may be broken/emulated down the chain, no real way to know

is there a way to profile it and help pinpoint the issue? I wouldn't mind to leave one instance running with full debug info if needed be.

trying this now https://groups.google.com/forum/#!msg/torch7/F9M2AHMxZP0/r72qXgoopv8J

LorenzoBoccaccia on 1 Mar 2016

I can confirm that it takes ages to start even the iterations process on my 14.04 i7-3630qm

tried default 512px image size. Went sleep, woke up in the morning with 50 iterations done. Only one core at time was using. For now i'm trying to get CUDA working and test it further with and without GPU.

Edit: Results with CPU: Itterations started in seconds. One iteration takes about 3.2s with default image size. Changes between setups:

installed OpenMP
installed Ubuntu via WUBI (before it was via unetbootin)
(installed cutorch/cunn - all cores were used for the first time - due luarocks? )

jezkovivoci on 1 Mar 2016

👍1

Hello,

I also had the same issue with my VM on ubuntu server 14.04. I got an i7 4720HQ 2,6 Ghz and 8 GB of RAM.
During a night, the vm could only make ~250 iterations (512 px).
I was thinking that it was normal because of the non-use of the gpu and the virtualization.
In fact, I didn't allocate enough memory and core, I allow 7 cores instead of 4 and 6GB ram instead of 4 and now I make 1000 iterations in one hour and a half.

Hope it could help anyone in any way.

pampaf on 3 Mar 2016

Neural-style requires a lot of memory. 4G is too little when using VGG19 network. There are alternative networks which work in less memory but not so easy to get good results.

I would suggest using top to monitor how much memory is being used while running neural-style. If you can open another terminal session, that is. Then it is easy to see when you hit the memory limit. I do it routinely.

br,
Hannu

pampaf [email protected] kirjoitti 3.3.2016 kello 11.02:

Hello,

I also had the same issue with my VM on ubuntu server 14.04. I got an i7 4720HQ 2,6 Ghz and 8 GB of RAM.
During a night, the vm could only make ~250 iterations (512 px).
I was thinking that it was normal because of the non-use of the gpu and the virtualization.
In fact, I didn't allocate enough memory and core, I allow 7 cores instead of 4 and 6GB ram instead of 4 and now I make 1000 iterations in one hour and a half.

Hope it could help anyone in any way.

—
Reply to this email directly or view it on GitHub.

htoyryla on 3 Mar 2016

I managed to get over some issue by compiling openblas, but it's still slower by magnitude of what other reports.

it does seem related by using too many threads. (strace shows loads of time wasted in kernel mode yelding()) will try with just four of them, but testing is slow as it takes loads to rebuild all the stuff properly

for reference here's a vmstat of it running

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
19  0      0  10367     34   1661    0    0    10    23   32   83  3  4 92  1  0
16  0      0  10367     34   1661    0    0     0     0 4531  178  8 92  0  0  0
16  0      0  10368     34   1661    0    0     0     0 4310  158  7 93  0  0  0
17  0      0  10368     34   1661    0    0     0     0 3796  148  7 93  0  0  0
16  0      0  10369     34   1661    0    0     0     0 4938  141  9 91  0  0  0
16  0      0  10369     34   1661    0    0     0     0 4921  254 32 50 18  0  0
16  0      0  10369     34   1661    0    0     0     0 3463  330 28 51 21  0  0
16  1      0  10369     34   1661    0    0     0     0 3726  113  8 92  0  0  0

there seems to be very little work done

LorenzoBoccaccia on 3 Mar 2016

Currently running it on VirtualBox. Gave the VM two processors, 5272 MB memory (as much as it would let me), and a 64-bit Ubuntu install. Running with -gpu -1 because I can't seem to figure out how to make it use my GPU (AMD Radeon R9 M275, if it matters - ticked all the acceleration boxes in VirtualBox settings but it's not working). When I ran it about a month ago with those settings, it would finish an image (with one style) in about two hours.

Trying it today... it was taking about a minute to a minute and a half per iteration when it was at around iteration 60, and is only getting slower. It's been about two hours and it's currently at iteration 76. My hardware hasn't changed, and neither has the VM. It's possible that there have been driver updates, and I can't quite remember if I was running Win 10 or Win 8.1 a month ago (on 10 now).

My host machine's Task Manager is saying VirtualBox is using barely over 100 MB of memory, not counting anything that might be marked as "Service Host" etc.. One particular VirtualBox Manager process is pretty much peaking my hard-drive read-write, though.

Any ideas on any of:

How to speed it up?
Why it got so much slower?
How to get it to use the GPU?

I can provide more logs or error messages or w/e as necessary, but didn't want to clutter this space up with a wall of stats if they wouldn't actually help.

qwertystop on 3 Mar 2016

The use of the gpu through a VM on windows is not supported by
VirtualBox (never read anything useful about it), so keep going on CPUmode.
Maybe you should try with more processors on your vm (I use seven
processors on my configuration).

Le 03/03/2016 21:41, qwertystop a écrit :

Currently running it on VirtualBox. Gave the VM two processors, 5272
MB memory (as much as it would let me), and a 64-bit Ubuntu install.
Running with -gpu -1 because I can't seem to figure out how to make it
use my GPU (AMD Radeon R9 M275, if it matters - ticked all the
acceleration boxes in VirtualBox settings but it's not working). When
I ran it about a month ago with those settings, it would finish an
image (with one style) in about two hours.

Trying it today... it was taking about a minute to a minute and a half
per iteration when it was at around iteration 60, and is only getting
slower. It's been about two hours and it's currently at iteration 76.
My hardware hasn't changed, and neither has the VM. It's possible that
there have been driver updates, and I can't quite remember if I was
running Win 10 or Win 8.1 a month ago (on 10 now).

My host machine's Task Manager is saying VirtualBox is using barely
over 100 MB of memory, not counting anything that might be marked as
"Service Host" etc.. One particular VirtualBox Manager process is
pretty much peaking my hard-drive read-write, though.

Any ideas on any of:

How to speed it up?

Why it got so much slower?

How to get it to use the GPU?

I can provide more logs or error messages or w/e as necessary, but
didn't want to clutter this space up with a wall of stats if they
wouldn't actually help.

—
Reply to this email directly or view it on GitHub
https://github.com/jcjohnson/neural-style/issues/142#issuecomment-191952963.

pampaf on 4 Mar 2016

@jezkovivoci: What did you do to install OpenMP? I read that it should be preinstalled with current versions of gcc/g++ (which I have).
I installed cutorch and cunn, but still only one core is used. How can I get it to use more cores?

EDIT: Apparently I didn't have OpenBLAS installed, which turned out to be kind of a pain to install but once it was, I reinstalled torch/nn/cutorch/cunn and then all cores were utilized!

Koaieus on 21 Apr 2016

I've encountered the same problem and I'm wondering if any of you found a solution since the last post. My linux knowledge is limited so I'm slow at finding a solution myself.