Theano: Global optimization to move gpu graph to the GPU.
Our current mechanism to move a graph to the GPU make too much validation and extra work. In one cases, I saw it take 30% of the optimization time.
We can make for the new GPU back-end a global optimizer that will do one full pass on the graph to move it to the GPU. Then let the current code finish the move (for the back-ward pass in case of op in the middle of the graph isn't implemented on the GPU).
The global optimizer we make should reuse current local optimizer to do the move. It would do something like:
topo = io_toposort(fgraph.inputs, fgraph.outputs)
for n in topo:
find as done by the Equilibrium optimizer, the subset of local opt that can apply on n and apply them.
topo = io_toposort(fgraph.inputs, fgraph.outputs)
Todo a back-ward pass on the graph.
for n in topo[::-1]:
find as done by the Equilibrium optimizer, the subset of local opt that can apply on n and apply them.
validate the change after the loop.
This new global opt should be registered in gpu_seqopt at step 0.5 (after InputToGpuOptimizer, before gpu_optimizer)
nouiz
All 12 comments
I saw @local_optimizer([]) - register it into cannonicalize without this useless pattern.
https://searchcode.com/codesearch/view/15851265/
@local_optimizer([])
def local_cut_gpu_host_gpu(node):
if tensor.opt.opt.check_chain(node, gpu_from_host, host_from_gpu):
return [node.inputs[0].owner.inputs[0]]
if tensor.opt.opt.check_chain(node, host_from_gpu, gpu_from_host):
return [node.inputs[0].owner.inputs[0]]
return False
gpu_cut_copies.register('cut_gpu_host_transfers', local_cut_gpu_host_gpu,
'fast_run', 'inplace', 'gpu')
gpu_cut_copies.register('cut_gpu_constant_transfers',
tensor.opt.constant_folding,
'fast_run', 'gpu')
register it into canonicalize to allow other optimization to work without
botering with this useless pattern.
optdb['canonicalize'].register('local_cut_gpu_host_gpu',
local_cut_gpu_host_gpu, 'fast_run', 'gpu')
Should the topo=io_toposort(fgraphs.inputs, fgraph,outputs)
be associated to this cannonicalization to make a pattern that is kept, while dumping the other useless patterns?
MERCI
AJB
ajboulay
on 17 Mar 2016
I don't understand the question. That particular optimization isn't
bottleneck to my knowledge.
On Wed, Mar 16, 2016 at 8:17 PM, aj boulay [email protected] wrote:
I saw @local_optimizer([]) - register it into cannonicalize without this
useless pattern.
https://searchcode.com/codesearch/view/15851265/@local_optimizer([])
def local_cut_gpu_host_gpu(node):
if tensor.opt.opt.check_chain(node, gpu_from_host, host_from_gpu):
return [node.inputs[0].owner.inputs[0]]
if tensor.opt.opt.check_chain(node, host_from_gpu, gpu_from_host):
return [node.inputs[0].owner.inputs[0]]
return False
gpu_cut_copies.register('cut_gpu_host_transfers', local_cut_gpu_host_gpu,
'fast_run', 'inplace', 'gpu')
gpu_cut_copies.register('cut_gpu_constant_transfers',
tensor.opt.constant_folding,
'fast_run', 'gpu')register it into canonicalize to allow other optimization to work without
botering with this useless pattern.
optdb['canonicalize'].register('local_cut_gpu_host_gpu',
local_cut_gpu_host_gpu, 'fast_run', 'gpu')Should the topo=io_toposort(fgraphs.inputs, fgraph,outputs)
be associated to this cannonicalization to make a pattern that is kept,
while dumping the other useless patterns?
MERCI
AJB—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4201#issuecomment-197619826
nouiz
on 19 Mar 2016
@nouiz A small confirmation.Is backward-pass done to verify that the op in the middle of the graph is implemented on the GPU or not ?
Sentient07
on 24 Mar 2016
The back-ward pass is needed only when some nodes in the middle of the
graph don't have a GPU implementation. Without it, all computation after
that node would be on CPU.
This part should be implemented only on the new gpu back-end under
sandbox/gpuarray/opt.py. The optimization have been refactored to have less
copied with small twist code. This will also prevent the creation of some
extra node that would happen with the old back-end.
On Thu, Mar 24, 2016 at 7:41 AM, Ramana Subramanyam <
[email protected]> wrote:
@nouiz https://github.com/nouiz A small confirmation.Is backward-pass
done to verify that the op in the middle of the graph is implemented on the
GPU or not ?—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4201#issuecomment-200800326
nouiz
on 24 Mar 2016
@nouiz : I'm a little confused.
The global optimizer we make should reuse current local optimizer to do the move.
Do I need to make a new global optimizer or use the current optimizer to move all the nodes to GPU at once ?
Can I make a check if all the nodes in the graph has GPU implementation, and then make the move, instead of making the move and doing a back-pass ?
Could you please explain what this [1] method does ?
Should the new Global Optimizer implement what [2] does, but for all the nodes in the entire graph at once ? If not could you please explain how different is our desired result from [2] ?
The back-ward pass is needed only when some nodes in the middle of the
graph don't have a GPU implementation.
From my understanding of this, i need to make the move to GPU, check if a node doesn't have GPU implementation. If a node does not have GPU implementation, ignore that and move the nodes after that to the GPU and keep doing this until all the nodes have been either moved or ignored(because no GPU implementation exists). Am I correct ?
[1] https://github.com/Theano/Theano/blob/master/theano/gpuarray/opt.py#L112
[2] https://github.com/Theano/Theano/blob/master/theano/gpuarray/opt.py#L172
Sentient07
on 19 May 2016
Do I need to make a new global optimizer or use the current optimizer to move all the nodes to GPU at once ?
I understood this now. If i am using local optimizer, the optimization is applied only to a single ApplyNode. Which is not the case with global optimizer. Am i right ?
Sentient07
on 20 May 2016
On Thu, May 19, 2016 at 5:06 PM, Ramana Subramanyam <
[email protected]> wrote:
@nouiz https://github.com/nouiz : I'm a little confused.
The global optimizer we make should reuse current local optimizer to do
the move.Do I need to make a new global optimizer or use the current optimizer to
move all the nodes to GPU at once ?You need to make a global optimizer, but it should reuse the local
optimizer to lift individual node.Can I make a check if all the nodes in the graph has GPU implementation,
and then make the move, instead of making the move and doing a back-pass ?
Could you please explain what this [1] method does ?[1] is a wrapper that help lift the nodes. If you look at local_gpuaalloc,
you see that local_gpuaalloc only move a node to the GPU. the op_lifter
class contain the logic if we need to move the node to the GPU or not.Should the new Global Optimizer implement what [2] does, but for all the
nodes in the entire graph at once ? If not could you please explain how
different is our desired result from [2] ?
You should not make as [2]. [2] cause many intermediate states and this
slow down the optimization phase. We don't want that. [2] and the logic
around cause that we move 1 node at a time and this is costly. We want to
move multiple nodes at a time.
The old logic is that we move 1 node at a time to the GPU. We move them
when 1 inputs or 1 outputs is on the GPU (see op_lifter for the exact
definition). For this to work InputToGpuOptimizer introduce transfert on
the inputs. It transfer a graph like:
a+b
to
host_from_gpu(gpu_from_host(a)) + host_from_gpu(gpu_from_host(b))
This allow the optimization to detect that the input of + is was on the GPU
and trigger the move of + to the GPU with this graph:
host_from_gpu(gpu_from_host(a) + gpu_from_host(b))
Here we see that the + now work on 2 inputs that are on the GPU.
The new algo should work more like (this is pseudo-code, I'm missing
probably stuff)
- We build a graph in parallel to the current graph and to help build the
new graph we keep a mapping of the node between them in a dict. - We init a loop like:
mapping = {}
for i in fgraph.inputs:
mapping[i] = GpuFromHost(None)(i)
for node in old_graph.toposort():
move_to_gpu = Use the op_lifter condition to move to the GPU with a
small modification, if the inputs and outputs is an input of the graph,
consider this as if it was on the GPU.
if move_to_gpu:
#call the method wrapped in op_lifter to move that node to the
graph.
#Most of the time this call should return a new op. For now if this
isn't the case, don't move that op to the GPU.
new_op = call(node)
new_outputs = new_op(*[mapping[i] for i in node.inputs])
for old_o, new_o in zip(node.outputs, new_outputs):
mapping[old_o] = new_o
else:
for o in node.outputs:
mapping[o] = o
The back-ward pass is needed only when some nodes in the middle of the
graph don't have a GPU implementation.From my understanding of this, i need to make the move to GPU, check if a
node doesn't have GPU implementation. If a node does not have GPU
implementation, ignore that and move the nodes after that to the GPU and
keep doing this until all the nodes have been either moved or
ignored(because no GPU implementation exists). Am I correct ?This is a different idea to what I had. I'm not sure the best one. If you
don't understand what I descrive bellow, you can do that and we'll see
where this goes. It won't be complicated to modify this version to that I
have in mind.
Historically, we always moved computation to the GPU when some condition
indicate us that it would be good to do the computation on the GPU, like it
lower transfer as the inputs or outputs are on the GPU. But in some cases,
this cause us problems that some compute intensive node aren't on the GPU
and it would have been better to move them there. This can happen if those
nodes are surounded of nodes that aren't implemented on the GPU. But this
don't happen frequently.
So what I had in mind is what you describe with an extra condition: that
the condition in op_lifter are met (what the change that inputs and outputs
to the graph are considered as being on the GPU).
It is probably easier to start without that extra condition. It would make
sure the logic to move is working correctly and we can add this extra
condition later.
[1]
https://github.com/Theano/Theano/blob/master/theano/gpuarray/opt.py#L112
[2]
https://github.com/Theano/Theano/blob/master/theano/gpuarray/opt.py#L172—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4201#issuecomment-220447047
nouiz
on 20 May 2016
On Fri, May 20, 2016 at 8:35 AM, Ramana Subramanyam <
[email protected]> wrote:
Do I need to make a new global optimizer or use the current optimizer to
move all the nodes to GPU at once ?I understood this now. If i am using local optimizer, the optimization is
applied only to a single ApplyNode. Am i right ?exact.
nouiz
on 20 May 2016
This can happen if those nodes are surrounded of nodes that aren't implemented on the GPU. But this don't happen frequently.
In backpass, we restart the transfer of Nodes to GPU that have been implemented(if the transfer stops after coming across a Node that hasn't been implemented). So, even if a compute intensive node is surrounded by a node that isn't implemented on GPU, how would it affect ?
Sentient07
on 20 May 2016
The back-ward pass is only needed with the extra condition. If you move all node that are implemented on the GPU to the GPU, it isn't needed.
The back-ward pass is needed when the input of a node isn't implemented on the GPU, but the node is and the clients of that node is also implemented on the GPU. Mostly, it is useful for node that have 1 inputs in the case this inputs come from a node that isn't implemented on the GPU.
In ML, this can happen if the cost isn't implemented on the GPU. Without the backward phase, the grad will be completly on the GPU.
nouiz
on 20 May 2016
@nouiz Your initial comment was pretty concrete. I understand now. Thanks.
Sentient07
on 20 May 2016
This was done in the new back-end.
nouiz
on 10 Nov 2016
Related issues
grayfall
·
3Comments
prapo550
·
9Comments
subzerofun
·
8Comments
asimkievich
·
7Comments
QuantumLiu
·
4Comments
Most helpful comment
On Thu, May 19, 2016 at 5:06 PM, Ramana Subramanyam <
[email protected]> wrote:
You should not make as [2]. [2] cause many intermediate states and this
slow down the optimization phase. We don't want that. [2] and the logic
around cause that we move 1 node at a time and this is costly. We want to
move multiple nodes at a time.
The old logic is that we move 1 node at a time to the GPU. We move them
when 1 inputs or 1 outputs is on the GPU (see op_lifter for the exact
definition). For this to work InputToGpuOptimizer introduce transfert on
the inputs. It transfer a graph like:
a+b
to
host_from_gpu(gpu_from_host(a)) + host_from_gpu(gpu_from_host(b))
This allow the optimization to detect that the input of + is was on the GPU
and trigger the move of + to the GPU with this graph:
host_from_gpu(gpu_from_host(a) + gpu_from_host(b))
Here we see that the + now work on 2 inputs that are on the GPU.
The new algo should work more like (this is pseudo-code, I'm missing
probably stuff)
new graph we keep a mapping of the node between them in a dict.
mapping = {}
for i in fgraph.inputs:
mapping[i] = GpuFromHost(None)(i)
for node in old_graph.toposort():
move_to_gpu = Use the op_lifter condition to move to the GPU with a
small modification, if the inputs and outputs is an input of the graph,
consider this as if it was on the GPU.
if move_to_gpu:
#call the method wrapped in op_lifter to move that node to the
graph.
#Most of the time this call should return a new op. For now if this
isn't the case, don't move that op to the GPU.
new_op = call(node)
new_outputs = new_op(*[mapping[i] for i in node.inputs])
for old_o, new_o in zip(node.outputs, new_outputs):
mapping[old_o] = new_o
else:
for o in node.outputs:
mapping[o] = o
Historically, we always moved computation to the GPU when some condition
indicate us that it would be good to do the computation on the GPU, like it
lower transfer as the inputs or outputs are on the GPU. But in some cases,
this cause us problems that some compute intensive node aren't on the GPU
and it would have been better to move them there. This can happen if those
nodes are surounded of nodes that aren't implemented on the GPU. But this
don't happen frequently.
So what I had in mind is what you describe with an extra condition: that
the condition in op_lifter are met (what the change that inputs and outputs
to the graph are considered as being on the GPU).
It is probably easier to start without that extra condition. It would make
sure the logic to move is working correctly and we can add this extra
condition later.