Numba: Feature request. Iterating by dimension over ND array

[elfnor]

I'm very new to numba, but the error message requests an issue report so:

minimal code

import numpy as np

from numba.decorators import jit

@jit(nopython=True)
def bug_test(pos1):
    for xyz in pos1:
        pass

    return

p1 = np.random.uniform(-10, 10, (20, 3))
bug_test(p1)

Traceback

Traceback (most recent call last):
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/errors.py", line 491, in new_error_context
    yield
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 216, in lower_block
    self.lower_inst(inst)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 265, in lower_inst
    val = self.lower_assign(ty, inst)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 411, in lower_assign
    return self.lower_expr(ty, value)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 767, in lower_expr
    res = impl(self.builder, (castval,))
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/targets/base.py", line 1043, in __call__
    return self._imp(self._context, builder, self._sig, args)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/targets/imputils.py", line 306, in wrapper
    _IternextResult(context, builder, pairobj))
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/targets/arrayobj.py", line 287, in iternext_array
    raise NotImplementedError("iterating over %dD array" % arrayty.ndim)
NotImplementedError: iterating over 2D array

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/elfnor/elfdisk/gits/add_mesh_space_tree/tests_2018/numba_bug.py", line 13, in <module>
    bug_test(p1)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args
    raise e
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile
    cres = self._compiler.compile(args, return_type)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py", line 83, in compile
    pipeline_class=self.pipeline_class)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 871, in compile_extra
    return pipeline.compile_extra(func)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 365, in compile_extra
    return self._compile_bytecode()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 802, in _compile_bytecode
    return self._compile_core()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 789, in _compile_core
    res = pm.run(self.status)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 251, in run
    raise patched_exception
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 243, in run
    stage()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 676, in stage_nopython_backend
    self._backend(lowerfn, objectmode=False)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 626, in _backend
    lowered = lowerfn()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 613, in backend_nopython_mode
    self.flags)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/compiler.py", line 990, in native_lowering_stage
    lower.lower()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 135, in lower
    self.lower_normal_function(self.fndesc)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 176, in lower_normal_function
    entry_block_tail = self.lower_function_body()
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 201, in lower_function_body
    self.lower_block(block)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/lowering.py", line 216, in lower_block
    self.lower_inst(inst)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/errors.py", line 499, in new_error_context
    six.reraise(type(newerr), newerr, tb)
  File "/media/elfnor/elfdisk/anaconda3/lib/python3.6/site-packages/numba/six.py", line 659, in reraise
    raise value
numba.errors.LoweringError: Failed at nopython (nopython mode backend)
iterating over 2D array

File "../../../../../../home/elfnor/elfdisk/gits/add_mesh_space_tree/tests_2018/numba_bug.py", line 7:
def bug_test(pos1):
    for xyz in pos1:
    ^
[1] During: lowering "$6.2 = iternext(value=$phi6.1)" at /home/elfnor/elfdisk/gits/add_mesh_space_tree/tests_2018/numba_bug.py (7)
-------------------------------------------------------------------------------
This should not have happened, a problem has occurred in Numba's internals.

Please report the error message and traceback, along with a minimal reproducer
at: https://github.com/numba/numba/issues/new

If you need help writing a minimal reproducer please see:
http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

If more help is needed please feel free to speak to the Numba core developers
directly at: https://gitter.im/numba/numba

Thanks in advance for your help in improving Numba!

[jllanfranchi]

You can iterate over each index of the array:

@jit(nopython=True)
def bug_test(pos1):
    for i in range(pos1.shape[0]):
        for j in range(pos1.shape[1]):
            xyz = pos1[i, j]

p1 = np.random.uniform(-10, 10, (20, 3))
bug_test(p1)

or if you don't care about the shape of the array and want to treat any-dimensioned array as a "flat" array (which seems to be how you want to work with it), you can use the flat attribute of a Numpy array within Numba as an iterator (or you can index into it, but using as an iterator as below is more concise and conforms better to your example):

@jit(nopython=True)
def bug_test(pos1):
    for xyz in pos1.flat:
        pass

p1 = np.random.uniform(-10, 10, (20, 3))
bug_test(p1)

[elfnor]

Thanks, I found what matches what I need (iterating by row) is

import numpy as np
from numba.decorators import jit

@jit(nopython=True)
def bug_test(pos1):
    for i in range(pos1.shape[0]):
        xyz = pos1[i, :]
        # do stuff with row xyz
    return

p1 = np.random.uniform(-10, 10, (20, 3))
bug_test(p1)

[stuartarchibald]

Thanks for the report @elfnor, this is unfortunately not supported yet, but it sounds like you have a work around, thanks for assisting @jllanfranchi . I've changed the title and made this a feature request.

[jllanfranchi]

You're both welcome, @elfnor and @stuartarchibald but no need to thank me, as I help out of pure self-interest: The more people who use Numba and the more time the core devs have to spend on developing it further, the better it'll be for me ;)