Dj-stripe: InvoiceItem.api_retrieve() always fails

[jleclanche]

InvoiceItem is .. weird, to say the least.

There's this whole thing going on:

    @classmethod
    def _stripe_object_to_invoice_items(cls, target_cls, data, invoice):
        lines = data.get("lines")
        if not lines:
            return []

        invoiceitems = []
        for line in lines.get("data", []):
            if invoice.stripe_id:
                save = True
                line.setdefault("invoice", invoice.stripe_id)

                if line.get("type") == "subscription":
                    # Lines for subscriptions need to be keyed based on invoice and
                    # subscription, because their id is *just* the subscription
                    # when received from Stripe. This means that future updates to
                    # a subscription will change previously saved invoices - Doing
                    # the composite key avoids this.
                    if not line["id"].startswith(invoice.stripe_id):
                        line["id"] = "{invoice_id}-{subscription_id}".format(
                            invoice_id=invoice.stripe_id,
                            subscription_id=line["id"])
            else:
                # Don't save invoice items for ephemeral invoices
                save = False

            line.setdefault("customer", invoice.customer.stripe_id)
            line.setdefault("date", int(dateformat.format(invoice.date, 'U')))

            item, _ = target_cls._get_or_create_from_stripe_object(
                line, refetch=False, save=save)
            invoiceitems.append(item)

        return invoiceitems

So, the stripe_id of InvoiceItem is, for subscriptions at least, never correct.
I don't even know what's happening here. Stripe doesn't seem to be using InvoiceItem by itself. The only InvoiceItem I get when doing InvoiceItem.list() is Unused time on $plan (with 100% off) after 09 Apr 2017, which is an item which begins with ii_.

I think dj-stripe might be mistaking InvoiceItem for SubscriptionItem. Or, well, I'm not sure. I don't know what's going on. Anyone more familiar with the system?

[jleclanche]

Okay, I see what's going on.

How it's actually laid out: Invoice.lines is a list of line_item. Some of these line items are of type invoiceitem and can be retrieved with InvoiceItem.retrieve(id=...). The rest are of type subscription and do not seem to be retrievable without their invoice.

dj-stripe reuses the InvoiceItem object for both types. This makes subscription line items unretrievable.

So first of all: What the actual fuck is that API? :|

Second: We don't store subscription_item (we don't even have a SubscriptionItem model). I don't fully understand why it's separate from the line item; it looks like it only has a plan and a quantity, both of which are present on the line item. Legacy stuff?

Third: Our abstraction is definitely wrong here, but on the other hand invoiceitem is virtually identical to subscription lineitem. Oh and guess what: A subscription_item on an invoiceitem type line item is not requestable! I'll file a support ticket.

[kavdev]

@jleclanche yeah it's a damn mess. A long time ago, it was a lot simpler. We can probably get rid of most of this when we implement multi-plan subscriptions

[kavdev]

@jleclanche is this still relevant?

[jleclanche]

Possibly… Stripe changed a lot about the II interface.

[jleclanche]

Okay, looks like Stripe has cleaned up the invoice item API indeed. I have an old invoice (in_1GsovGEU4JBg99IS63skU8Ps) where I can reproduce this issue, but with a different result:

in_1GsovGEU4JBg99IS63skU8Ps-il_1GsovGEU4JBg99ISHpSYhvCa
Traceback (most recent call last):
  File "<console>", line 2, in <module>
  File "/home/adys/src/dj-stripe/djstripe/models/base.py", line 164, in api_retrieve
    return self.stripe_class.retrieve(
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/api_resources/abstract/api_resource.py", line 12, in retrieve
    instance.refresh()
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/api_resources/abstract/api_resource.py", line 16, in refresh
    self.refresh_from(self.request("get", self.instance_url()))
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/stripe_object.py", line 243, in request
    response, api_key = requestor.request(method, url, params, headers)
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/api_requestor.py", line 122, in request
    resp = self.interpret_response(rbody, rcode, rheaders)
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/api_requestor.py", line 373, in interpret_response
    self.handle_error_response(rbody, rcode, resp.data, rheaders)
  File "/home/adys/.cache/pypoetry/virtualenvs/djstripe-example-project-mGtDSX_z-py3.8/lib/python3.8/site-packages/stripe/api_requestor.py", line 152, in handle_error_response
    raise err
stripe.error.InvalidRequestError: Request req_RIbDsBTsWMPu1L: Old `id` values cannot be used to specify an invoice line item in this API version. Please use the current value of the `id` field of the invoice line item object.

[jleclanche]

In db7c7dd30a8853aec3477c3064fb5cf3c887fdde, a new management command has been implemented to migrate the old InvoiceItem IDs to the new format.

Why isn't this part of database migrations?

1. Because it's potentially expensive to run for databases with large amounts of InvoiceItems
2. It's mostly harmless, but may lead to weird data inconsistencies
3. The management command does it more safely, by actively retrieving and re-syncing the data from upstream for every InvoiceItem that uses the old format, one at a time. This means the command can fail / be interrupted, and be resumed at a later time (safely, too; it's okay to only run it for a portion of the data).