Python-docx: feature: BlockItemContainer.iter_block_items()

[danmilon]

I'd like to iterate over the elements of they document as they appear in it. For example if there is a paragraph a table and then a paragraph again, I want to get them in that order. AFAIK currently there are two properties on Document, paragraphs and tables but have no notion of ordering between them.

[pmagsino]

This is a feature that would be useful for data mining. My use case is such that the primary data to be extracted are within tables. The related secondary data are from paragraphs that are either precede or straddle the table.

[scanny]

This workaround should work for anyone who can't wait for the Document.iter_block_items() feature to be implemented. I haven't tested it, so please provide feedback if it gives any trouble or you get it to work.

It can accept either a document or a table cell for its _parent_ argument.

from docx.api import Document
from docx.oxml.table import CT_Tbl
from docx.oxml.text import CT_P
from docx.table import _Cell, Table
from docx.text import Paragraph

def iter_block_items(parent):
    """
    Yield each paragraph and table child within *parent*, in document order.
    Each returned value is an instance of either Table or Paragraph.
    """
    if isinstance(parent, Document):
        parent_elm = parent._document_part.body._body
    elif isinstance(parent, _Cell):
        parent_elm = parent._tc
    else:
        raise ValueError("something's not right")

    for child in parent_elm.iterchildren():
        if isinstance(child, CT_P):
            yield Paragraph(child)
        elif isinstance(child, CT_Tbl):
            yield Table(child)

[pmagsino]

Awesome. The workaround works great for my use case. Thanks again.

[scanny]

Glad to hear it Paul :)

I'll leave this issue open as the feature request.

[ghost]

Had to make these changes to the code to get the function to work

if isinstance(child, CT_P):
    yield Paragraph(child,parent_elm)
elif isinstance(child, CT_Tbl):
    yield Table(child,parent_elm)

[scanny]

I think None would probably be better than parent_elm. The parent parameter which was added to the Paragraph and Table constructors since this issue opened expects the parent proxy object like _Body or (table)_Cell, not the lxml parent element (e.g. <w:body>).

These are only used when an upward reference is required, such as when inserting a picture, so depending on the use case, using None might work well enough to get the job done. Using parent and making sure it was a reference to _Body or _Cell would be better.

In any case, this hack is due for a proper solution once I can get back to it. Been very busy on python-pptx just lately getting chart functionality going there :)

UPDATE:
On later reflection, it became clear the new parameter should simply be parent as provided as an original call argument to iter_block_items. An updated full version is a couple comments down.

[ghost]

Thanks for the help.I am also interested to know how would you go about this function,more specifically how would you want to handle inline images,charts and mathematical equations when they come in the text.I am thinking of just returning the xml in case of charts or equations and returning the image in case there is an image in the run.

[scanny]

Well, a solution for the general case would yield a proxy object (e.g. Paragraph, Table) for each element encountered so the developer could operate on the object without having to go down to the XML level. This gets a little tricky because there are a surprisingly large array of types that can possibly appear within a block context or inline context and not nearly all of them have proxy objects yet. Things like a <w:del> and <w:ins> element that have to do with the revision tracking, for example.

One solution would be to return a proxy object when you could and then a generic NotImplementedObject or something when no suitable proxy class existed for the item.

Note also that there are two main contexts one might want to iterate over, a block context and an inline context. An element like the <w:body> element of a document part contains block-level objects like Paragraph and Table. A Paragraph itself is an inline context and contains things like Run, and inline pictures, hyperlinks, etc.

This issue was originally about block-level items, but a corresponding method for iterating over inline objects would also be handy.

[scanny]

An updated snippet that should do the trick and is consistent with the latest internals would look like this. I haven't had time to test it, so if it gives you trouble let me know and I'll help fix :)

from docx.document import Document
from docx.oxml.table import CT_Tbl
from docx.oxml.text.paragraph import CT_P
from docx.table import _Cell, Table
from docx.text.paragraph import Paragraph

def iter_block_items(parent):
    """
    Yield each paragraph and table child within *parent*, in document order.
    Each returned value is an instance of either Table or Paragraph. *parent*
    would most commonly be a reference to a main Document object, but
    also works for a _Cell object, which itself can contain paragraphs and tables.
    """
    if isinstance(parent, Document):
        parent_elm = parent._document_part.body._body
    elif isinstance(parent, _Cell):
        parent_elm = parent._tc
    else:
        raise ValueError("something's not right")

    for child in parent_elm.iterchildren():
        if isinstance(child, CT_P):
            yield Paragraph(child, parent)
        elif isinstance(child, CT_Tbl):
            yield Table(child, parent)

[danmilon]

@scanny, yes, this works perfectly. Do you want me to wrap this up, and send in a PR?

[scanny]

The tests will be the key outstanding components for this one. If you want to take a crack at it, by all means :)

[cez81]

Is there a way of doing this after the recent changes to docx.Document?

[scanny]

Not yet; this feature is still in the backlog. The last release focused on styles support.

[scanny]

Oh, I think I misinterpreted your question. Some of the imports have to change due to recent refactoring:

from docx.document import Document
from docx.oxml.text.paragraph import CT_P
from docx.text.paragraph import Paragraph

Is that what you were asking @cez81 ?

I've updated the example.

[cez81]

Yes I think it was. Unfortunately I can't get it to work tho... I get an error creating the Document instance
"TypeError: init() missing 1 required positional argument: 'part'". I'm guessing it has to do with the first line importing the wrong Document class?

from docx.document import Document
from docx.oxml.text.paragraph import CT_P
from docx.text.paragraph import Paragraph

def iter_block_items(parent):
    """
    Yield each paragraph and table child within *parent*, in document order.
    Each returned value is an instance of either Table or Paragraph. *parent*
    would most commonly be a reference to a main Document object, but
    also works for a _Cell object, which itself can contain paragraphs and tables.
    """
    if isinstance(parent, Document):
        parent_elm = parent._document_part.body._body
    elif isinstance(parent, _Cell):
        parent_elm = parent._tc
    else:
        raise ValueError("something's not right")

    for child in parent_elm.iterchildren():
        if isinstance(child, CT_P):
            yield Paragraph(child, parent)
        elif isinstance(child, CT_Tbl):
            yield Table(child, parent)


doc = Document('test.docx')
for block in iter_block_items(doc):
    print(block.text)

[scanny]

Ah, right. If you do this that should fix your case where you have them both in the same module and need to use both the docx.document.Document class and the docx.Document factory function:

import docx

doc = docx.Document('test.docx')
for block in iter_block_items(doc):
    print(block.text)

[pdelsante]

Hi, I think @cez81 is right: there seems to be something more that changed in your code lately. To make your example work again with 0.8.5 I had to change it like this:

from docx.document import Document
from docx.oxml.table import CT_Tbl
from docx.oxml.text.paragraph import CT_P
from docx.table import _Cell, Table
from docx.text.paragraph import Paragraph


def iter_block_items(parent):
    """
    Yield each paragraph and table child within *parent*, in document order.
    Each returned value is an instance of either Table or Paragraph. *parent*
    would most commonly be a reference to a main Document object, but
    also works for a _Cell object, which itself can contain paragraphs and tables.
    """
    if isinstance(parent, Document):
        parent_elm = parent.element
    elif isinstance(parent, _Cell):
        parent_elm = parent._tc
    else:
        raise ValueError("something's not right")

    for child in parent_elm.iterchildren():
        if isinstance(child, CT_P):
            yield Paragraph(child, parent)
        elif isinstance(child, CT_Tbl):
            yield Table(child, parent)

In particular, I had to change the following line:

        parent_elm = parent._document_part.body._body

to this:

        parent_elm = parent.element

[scanny]

Ah, yes, I see what you're saying. The way to get a reference to to the <w:body> element changed as well. I think what you want is this though:

if isinstance(parent, Document):
    parent_elm = parent.element._body

... because parent.element is the <w:document> element if I'm reading the code correctly.

Apologies I don't have time to test this right now, but hope that helps. Such are the wages of workaround functions because they rely on internals that aren't guaranteed to be stable between releases.

[cez81]

Ok got it working now! Changed it to:

if isinstance(parent, Document):
    parent_elm = parent.element.body

Thanks for the help both of you!

[nguyen1110935]

Hi @scanny

your iter_block_items() parse paragraph and table in docx file.
I'm a beginner in python.
Could you please update to parse all headings (Heading 1, Heading 2,..etc), paragraphs and tables.

Thank you.

[emishaikh]

In my case, i have double column in my paragraph , i wanted to extract that also as well as any image if there in paragraph.
if i am using
for child in parent_elm.iterchildren(): if isinstance(child, CT_P): yield Paragraph(child, parent) elif isinstance(child, CT_Tbl): yield Table(child, parent)

It is taking Double column text as a paragraph. Please help me to find double column also in my paragraph

[igorsavinkin]

how to get text then from yielded Table object?

[alfiyafaisy]

Hi @scanny ,
I tried your code, but Im facing a problem in the section mentioned below.

for block in iter_block_items(doc):
    print(block.text)

Here Im getting the error _"AttributeError: 'Table' object has no attribute 'text'"_ .
Can anyone please help me to solve this issue?
Thanks in advance.

[alfiyafaisy]

how to get text then from yielded Table object?

Hey @igorsavinkin , this worked for me.

for block in iter_block_items(doc):
    if isinstance(block, Table):
        for row in block.rows:
            row_data = []
            for cell in row.cells:
                for paragraph in cell.paragraphs:
                    row_data.append(paragraph.text)
            print("\t".join(row_data)

[igorsavinkin]

This works to me:
def iter_block_items(parent):
if isinstance(parent, Document):
parent_elm = parent.element.body
elif isinstance(parent, _Cell):
parent_elm = parent._tc
else:
raise ValueError("something's not right")

for child in parent_elm.iterchildren():
    if isinstance(child, CT_P):
        yield Paragraph(child, parent)
    elif isinstance(child, CT_Tbl):
        # yeild paragraphs from table cells
        # Note, it works for single level table (not nested tables)
        table = Table(child, parent)
        for row in table.rows:
            for cell in row.cells:
                for paragraph in cell.paragraphs:
                    yield paragraph

On Wed, Nov 7, 2018 at 6:29 AM alfiyafaisy [e-mail] wrote:

how to get text then from yielded Table object?

Hey @igorsavinkin https://github.com/igorsavinkin , this worked for me.

for block in iter_block_items(doc):
if isinstance(block, Table):
for row in block.rows:
row_data = []
for cell in row.cells:
for paragraph in cell.paragraphs:
row_data.append(paragraph.text)
print("\t".join(row_data)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/python-openxml/python-docx/issues/40#issuecomment-436501231,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE_LhbDpRjnDDnP3iuorZCrthgqhNW12ks5usmGigaJpZM4B1eh3
.

--
Igor Savinkin, ([phone]
Skype: igorsavinkin
http://ergonotes.com/
http://scraping.pro/
http://about.me/ http://about.me/igorsavinkin
Riga, EU.
In Him.

[shulinwz]

nested tables should be easy to handle with recursion

def iter_block_items(parent):
    if isinstance(parent, Document):
        parent_elm = parent.element.body
    elif isinstance(parent, _Cell):
        parent_elm = parent._tc
    else:
        raise ValueError("something's not right")

    for child in parent_elm.iterchildren():
        if isinstance(child, CT_P):
            yield Paragraph(child, parent)
        elif isinstance(child, CT_Tbl):
            table = Table(child, parent)
            for row in table.rows:
                for cell in row.cells:
                    yield from iter_block_items(cell)

[romran]

Hello everyone,
It helped me a lot when parsing docx files.
May you tell, if it is possible to refactor this function and also find InlineShapes ?

[Slowhalfframe]

Hello everyone.
I have a question: how to use Python to read pictures in word order？

[lxj0276]

I want to know how to read pictures or charts like function "iter_block_items"

[Slowhalfframe]

I want to know how to read pictures or charts like function "iter_block_items"

def read_item_block(parent):
'''
顺序读取wordneir
:param parent: 文档
:return: p/t
'''
if isinstance(parent, _Document):
parent_elm = parent.element.body
elif isinstance(parent, _Cell):
parent_elm = parent._tc
elif isinstance(parent, _Row):
parent_elm = parent._tr
else:
raise ValueError("something's not right")
for child in parent_elm.iterchildren():
if isinstance(child, CT_P):
count = 1
count_flase = 0
res = Paragraph(child, parent)
if res.text != '':
yield (res,count_flase)
else:
try:
# 试着去取内联元素
from xml.dom.minidom import parseString
DOMTree = parseString(child.xml)
data = DOMTree.documentElement
nodelist = data.getElementsByTagName('pic:blipFill')
print('nodelist'9,nodelist)
if len(nodelist) < 1:
yield (res,count_flase)
else:
yield (res, count)
except Exception as e:
print(''9,e)
yield (res,count_flase)
elif isinstance(child, CT_Tbl):
yield (Table(child, parent),)

This is how I read pictures.

[devanshugupta]

Having this error in your code:

Traceback (most recent call last):
File "C:/Users/home/PycharmProjects/Sentiment_analysis/yup.py", line 46, in
for cell in row.cells:
File "C:\Users\home\Desktop\devu\venv\lib\site-packages\docx\table.py", line 401, in cells
return tuple(self.table.row_cells(self._index))
File "C:\Users\home\Desktop\devu\venv\lib\site-packages\docx\table.py", line 106, in row_cells
return self._cells[start:end]
File "C:\Users\home\Desktop\devu\venv\lib\site-packages\docx\table.py", line 173, in _cells
cells.append(cells[-col_count])
IndexError: list index out of range

[Labyrins]

Ok got it working now! Changed it to:

if isinstance(parent, Document):
    parent_elm = parent.element.body

Thanks for the help both of you!

Thank you. this works to me!

[div1996]

how to read paragraph,table,shapes all in one place....Kindly Help ASAP