Sylius: [RFC] Order state machine

[pjedrzejewski]

Hello!

Current states and state machine of Order is pretty confusing. This RFC aims to solve following problems:

• Clarifying all states that we have on the order;
• Making the checkout process cleaner;
• Making it easier to identify what is cart and what is order;

I will be talking about the following states and field:

• Order.state
• Order.checkoutState
• Order.shippingState
• Order.paymentState
• Order.completedAt

We currently use Order.completedAt to decide whether it is a Cart or Order. That's annoying. As suggested in another PR/issue, I think we should rename it to orderCheckoutCompletedAt and use it for information purposes when exactly the checkout has been completed.

Checkout state

Now, to the states. Let's start from checkoutState, which is pretty clear and reworked already in #3918. I will paste the configuration overview for reference:

winzou_state_machine:
    sylius_order_checkout:
        class: %sylius.model.order.class%
        property_path: checkoutState
        graph: sylius_order_checkout
        state_machine_class: %sylius.state_machine.class%
        states:
            cart: ~
            started: ~
            addressed: ~
            shipping_selected: ~
            payment_selected: ~
            completed: ~
        transitions:
            start:
                from: [cart]
                to: started
            address:
                from: [started]
                to: addressed
            readdress:
                from: [payment_selected, shipping_selected, addressed]
                to: started
            select_shipping:
                from: [addressed]
                to: shipping_selected
            reselect_shipping:
                from: [payment_selected, shipping_selected]
                to: addressed
            select_payment:
                from: [shipping_selected]
                to: payment_selected
            reselect_payment:
                from: [payment_selected]
                to: shipping_selected
            complete:
                from: [payment_selected]
                to: completed

This is pretty much self-explanatory and covers the whole checkout process. If you customize it, you will need to rework this state-machine, but this gives us clean way of executing callbacks, recalculations, etc.

State

Next state is Order.state, this one is a bit tricky, but it should represent the overall state of the order and also we should use it to identify if Order is a cart. Here is the state machine I propose as initial implementation:

winzou_state_machine:
    sylius_order:
        class: %sylius.model.order.class%
        property_path: state
        graph: sylius_order
        state_machine_class: %sylius.state_machine.class%
        states:
            cart: ~
            new: ~
            abandoned: ~
            cancelled: ~
            closed: ~
        transitions:
            create:
                from: [cart]
                to: new 
            abandon:
                from: [new]
                to: abandoned
            cancel:
                from: [new]
                to: cancelled
            retry:
                from: [cancelled, abandoned]
                to: new
            close:
                from: [new]
                to: closed

Explanation of the states:

• cart - This is default state of the order and we use it to differentiate between Orders and Carts. Basically, everything that is not in state cart is an Order that should show up in the admin.
• new - Order goes into this state as soon as the checkout is completed. "New" orders appears in the backend.
• abandoned - If someone completes the checkout, but does not pay after X time, order goes into abandoned state, inventory is released, etc.
• cancelled - Cancelled via admin.
• closed - Final state of the order. When the order is paid and shipped, it becomes "closed". You can't modify it and that's end of the story.

Transitions are self-explanatory. Later we want to give the ability to "retry" an abandoned/cancelled order, that's why I added this transition.

Payment state

This one represents the status of all payments on the order. We already partially support multiple payments per order, that's why I want to have this separate state.

winzou_state_machine:
    sylius_order_shipping:
        class: %sylius.model.order.class%
        property_path: paymentState
        graph: sylius_order_payment
        state_machine_class: %sylius.state_machine.class%
        states:
            cart: ~
            awaiting_payment: ~
            partially_paid: ~
            cancelled: ~
            paid: ~
        transitions:
            request:
                from: [cart]
                to: awaiting_payment
            partially_pay:
                from: [awaiting_payment]
                to: partially_paid
            cancel:
                from: [awaiting_payment]
                to: cancelled
            pay:
                from: [awaiting_payment, partially_paid]
                to: paid

This one is very rough, I am sure we will need more transitions or even maybe states. I am unsure about the partially_paid state, perhaps we should have just one amount_due state or something like that.
One thing I want to avoid is discussing these longer than until next weekend because we really need to clean this up and get rid of remaining event listeners to regain control of the whole order processing.

Shipping state

Sylius Order can have multiple shipments too, so we need to know what is the overall status of the order.

winzou_state_machine:
    sylius_order_shipping:
        class: %sylius.model.order.class%
        property_path: shippingState 
        graph: sylius_order_shipping
        state_machine_class: %sylius.state_machine.class%
        states:
            cart: ~
            abandoned: ~
            new: ~
            ready: ~
            cancelled: ~
            partially_shipped: ~
            shipped: ~
        transitions:
            request:
                from: [cart]
                to: new
            abandon:
                from: [new]
                to: abandoned
            prepare:
                from: [new, pending]
                to: ready
            cancel:
                from: [new, pending]
                to: cancelled
            partially_ship:
                from: [ready, pending]
                to: partially_shipped
            ship:
                from: [ready, partially_shipped]
                to: shipped

I am not entirely sure about this one, so just throwing it here. I think we could safely skip partially_shipped state and transitions for now, because Sylius does not handle multiple shipments now. Except in the checkout.

Summary

Some of these are already there, but not properly used or not in use at all. Let's brainstorm a bit and finally put this state machine into action!

[peteward]

• Checkout State :+1: Very easy to customise anyway and there's nothing contentious there.
• Payment State :+1: Makes sense to me.
• State
  
  
  
  • What's the use-case for completing checkout but then not paying for it?
  
  
  • I would have thought abandoned is more a term for people who don't make it to completing checkout.
  
  
  • Is an order really ever closed? It's very difficult to call an order closed if you support returns/refunds. The only way that I could think of an order being closed is if it's fully refunded or after X number of days when refund possibilities have expired. I know Sylius doesn't support refunds yet but this naming would become confusing as soon as it does. I think something like shipped or fulfilled is the better state than closed.
  
  

We merged order.state and order.shippingState and only use order.state, because of the above reason that the shipping becomes a core part of the order lifecycle and it became difficult to separate their meaning without contention.

We have quite a long state machine definition, but it reflects a long chain of events around the order that includes a fraud review process where orders are approved or rejected, and subsequently transmitted to our stock system. There's also some sylius legacy stuff around locked carts in there:

        states:
            payment_unknown: ~
            cart:            ~
            cart_locked:     ~
            pending:         ~
            released:        ~
            abandoned:       ~
            confirmed:       ~
            rejected:        ~
            cancelled:       ~
            approved:        ~
            transmitted:     ~
            packed:          ~
            shipped:         ~
            returned:        ~
        transitions:
            payment_unknown:
                from: [cart, cart_locked, pending, released, confirmed]
                to:   payment_unknown
            lock_cart:
                from: [cart]
                to:   cart_locked
            unlock_cart:
                from: [cart_locked]
                to:   cart
            abandon:
                from: [cart, pending]
                to:   abandoned
            create:
                from: [cart]
                to:   pending
            release:
                from: [pending]
                to:   released
            confirm:
                from: [cart, pending, released]
                to:   confirmed
            approve:
                from: [confirmed]
                to:   approved
            reject:
                from: [confirmed]
                to:   rejected
            transmit:
                from: [approved]
                to:   transmitted
            cancel:
                # NOTE: When we change to Merret we might need to remove transmitted from here.
                from: [confirmed, approved, transmitted]
                to:   cancelled
            pack:
                from: [transmitted]
                to:   packed
            ship:
                from: [confirmed, packed]
                to:   shipped
            return:
                from: [shipped]
                to:   returned

[michalmarcinkowski]

• Checkout state :+1: - as for completedAt field I would rename it to simply checkoutCompletedAt
• Payment state :+1:
• Shipping state
  
  
  
  • IMO there shouldn't be cart state, what does it mean cart in terms of shipping? Maybe we should have none state until the order qualifies for shipping (e.g. is paid) and then new? Same for abandoned, the shipping can be abandoned? When the order is abandoned the shipping should be cancelled. I would suggest:
  
  

winzou_state_machine:
    sylius_order_shipping:
        class: %sylius.model.order.class%
        property_path: shippingState 
        graph: sylius_order_shipping
        state_machine_class: %sylius.state_machine.class%
        states:
            none: ~
            new: ~
            ready: ~
            cancelled: ~
            partially_shipped: ~
            shipped: ~
        transitions:
            request:
                from: [none]
                to: new
            prepare:
                from: [new]
                to: ready
            cancel:
                from: [new, ready]
                to: cancelled
            partially_ship:
                from: [ready]
                to: partially_shipped
            ship:
                from: [ready, partially_shipped]
                to: shipped

• Order state - it should represent the overall state of the order, but shouldn't represent any of payment or shipping state (it's redundant).

winzou_state_machine:
    sylius_order:
        class: %sylius.model.order.class%
        property_path: state
        graph: sylius_order
        state_machine_class: %sylius.state_machine.class%
        states:
            cart: ~
            new: ~
            pending: ~
            abandoned: ~
            cancelled: ~
            completed: ~
        transitions:
            create:
                from: [cart]
                to: new
            process:
                from: [new]
                to: pending
            abandon:
                from: [new]
                to: abandoned
            cancel:
                from: [new, pending]
                to: cancelled
            complete:
                from: [pending]
                to: completed

Let me explain the differences:

• new - After the checkout is completed
• pending - the order is ready to be processed, e.g. if someone has fraud review process, there can be other states before pending e.g. verified, approved etc. The default implementation can be that when the order is paid, then it becomes pending.
• completed - the order is 'done'. The default implementation can represent 'done' as paid and shipped.

Then when someone supports returns there can be two more states after completed:

• partially_returned - only few items were returned (not everything)
• returned - the whole order was returned.

This way we have pretty flexible flow and do not have any redundant states.

[peteward]

I'm not sure about none as a state, isn't new sufficient until the shippingState is of relevance?

For order state I think new not being the first state is misleading. Shouldn't it be checkoutComplete or similar? Likewise pending sounds like the order is being held or waiting for something. Perhaps processing is better? since you've called the transition process anyway? What action does the process transition represent in real world terms? I'm wondering if there's even a need for new and pending|processing because a post-checkout order should immediately be in a state of being processed?

I'm still not a fan of complete for future reasons, I still think fulfilled is better.

Also presumably an order can be abandoned from cart state too?

[michalmarcinkowski]

Also not sure about none as shipping state, but my idea was that when the shippingState becomes new it is to be processed (waiting for being packed) and none means that the order is e.g. waiting for payment, so we should hold with packing.

We could skip it and in order to get orders that needs to be packed we can query by e.g. paymentState: paid and shippingState: new, but it will be simpler and more generic if we agree that new represents shipping that needs to be processed. You could have different criteria for marking the shipping as ready to be packed, if you have fraud detection you could query by orderState: confirmed and shippingState: new etc.

Maybe we it would be better to have new as the first state and then have another state that will indicate the order is awaiting for being packed?

:+1: for renaming pending to processing.
:+1: for renaming complete to fulfilled

The cart state is necessary since the order becomes a real order only after the checkout is completed. In the backed you will never see the cart state of the order, so it's more like internal indicator.

• The new order state is for carts that changed into real order. This state indicates the order was placed, but before we go into processing we could for example wait for payment or wait for fraud detection process to complete etc.
• The order changes from new to processing when the order is ready to be processed. In default implementation the order could go into processing after the payment state becomes paid.
• The processing state indicates that someone needs to take care of that order e.g. the order needs to be shipped.

Abandoned cart is not that simple since you need to decide when the cart is considered abandoned. I can get back to my cart after two weeks and does it mean I abandoned cart or postponed my shoppings? Do you find it useful to have abandoned cart?

[peteward]

If you were to follow the none idea then shouldn't paymentState also start at none for consistency? Because payment isn't possible at the very start of a cart journey either. Does there need to be a consistent _unavailable_ or none state for all?

As an idea it makes sense to me though.

The problem with the order.state naming is because cart == order, so while new makes sense for a New Order, it's not a New Cart, so there's scope for confusion. placed? checkedOut?

I'm not sure about abandoned baskets for us yet. It is important for both cleaning up the DB and for statistics (although an external provider currently handles our abandonment analytics and re-engagement campaigns). Carts that are known only by session are not really possible to recover after session expires, whereas customer carts can be reclaimed for a much longer period. On our legacy site we have something like 2 weeks expiry on abandoned carts for customers.

Again, it's more the clean-up of the table that's currently important with over 1.5 million orders, but it is a process that needs some thought...

[michalmarcinkowski]

Indeed we should have none or preferably better named state on payment also.

The new state will only be valid for order, so the New Order is when the checkout is completed (basically when the cart is no longer a cart, it is transformed to an order). You will not be able to have like 'New Cart'. This way in the backend we will not display any cart in the orders section.

Abandoned baskets is not an easy topic and IMO it deserves a separate RFC :) We will be happy to discuss this topic once you will have some more thoughts about this process in your company.

[peteward]

Whilst we're in state machines...

I've just seen that the InventoryUnit state machine has a state called returned, I would recommend renaming this as restocked since this state seems to represent the unit's stock status.

It also means there would be no confusion against a returned state which could also be used in a shipping state machine (or possibly a separate returns state machine if you really wanted to separate this flow, we've kept it in the shipping state machine).

In our case, a customer can successfully return an item but it could either be restocked or discarded depending on the condition (faulty goods are obviously not going back into stock).

[pjedrzejewski]

@peteward I will open separate RFC about all other states, wanted to start with the most important for Order&Cart. Good point about restocked! :+1:

[peteward]

no problem, just had to write it down before I forgot :smile:

[gabiudrescu]

I have one question about this RFC: do you guys have to use 3rd party systems such as ERPs or WMS that have to keep in the loop during order lifetime?

For example: When I place an order on a Sylius instance, the warehouse team should see my order in their WMS in order to pick the goods from our shelves and prepare the package. Then, before closing the package, the ERP should issue an invoice for all the picked goods.

If there is someone having to deal with this kind of architecture, I'd be interested to know more about the status used in Sylius for these moments. Is it "prepare" from Shipping?

Notes:
Just to make sure we're on the same page:

• ERP = Enterprise resource planning
• WMS = warehouse management system

---

Regarding closed, I'm with @peteward on this page. I'd go for "shipped" when the order is paid and shipped and "closed" after the vesting period ends and the customer cannot return the merchandise.

When you use affiliate marketing channels, it's very important to know when the order is "closed" in the acceptance of my above definition so you can pay commissions - then and only then. Otherwise, you are liable to commission fraud: place an order, get a commission, get paid and then return the merchandise.

[peteward]

@gabiudrescu we have a WMS but not really an ERP as such.

Once an order has completed checkout it is held for fraud review.
Only when it is approved by our fraud provider is it transmitted to our WMS for warehouse picking.
We provide the admin interface for packing and shipping too so have the full lifecycle in our platform, but we need to communicate orders for stock allocation.

However we will be moving to a new WMS later this year which will handle much more of the fulfilment flow, telling us by CSV when an order has been shipped.

Our state flow will not really change much (if at all), we'll simply call the state transitions in sequence when we receive status updates.

It sounds a bit different from your situation though.

[gabiudrescu]

well, my description above is the "desired" implementation, not the current one. I'll go into specifics in private, if you want to know more.

Anyway, I got what I wanted from the description you mentioned above regarding fraud. In our case, "fraud" review its optional if the user doesn't have automated validation credentials. In this case, an operator is going to call him and ask for additional papers.

otherwise, his order will be shipped without further questions.

[mingxing]

And we try to update the status of payment or shipment, e.g refund, the payment_state of order has to been update manually, why not update with state-machine.. Did you meet the issue?

[koemeet]

I noticed that the state of an Order is always on new. Is it on purpose that there is no callback to set the state to fulfilled when the whole order is fulfilled (all shipments are shipped)?

[michalmarcinkowski]

@steffenbrem we got this feature scheduled for the coming week. The order will be marked as fulfilled when all payments will be paid and all shipments shipped 😉

[michalmarcinkowski]

Closing as order states has been already established.