Aframe: Brainstorming <a-state-machine />

[donmccurdy]

Just writing up some API ideas here.

---

Developers try to use <a-animation/> to transition between states (do X after Y), and of course it doesn't (and shouldn't) do any of that. In Unity that would be managed by a state machine, for which we currently have no construct.

But state machines are naturally declarative. They can be cyclical graphs, which the DOM can't, but let's try:

<a-scene>
  <a-entity gltf-model="..."
            state-machine="#npc-state;"
            animation-mixer="clips: none;"
            proximity-emitter__far="filter: #player; distance: 20; event: far;"
            proximity-emitter__near="filter: #player; distance: 1; event: near;"
            path-finding="target: none;"
            attack-ability="target: none;"></a-entity>

  <a-state-machine id="npc-state">
      <a-state name="idle"
               default
               enter-set="property: animation-mixer.clips; value: idle_*;"
               transition__walk="on: farstart; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="walk"
               enter-set__1="property: animation-mixer.clips; value: walk_*;"
               enter-set__2="property: path-finding.target; value: event.target;"
               exit-set="property: path-finding.target; value: none;"
               transition__idle="on: farstop; state: idle;"
               transition__attack="on: nearstart; state: attack;"
               transition__walk="on: nearstop; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="attack"
               enter-set__1="property: animation-mixer.clips; value: attack_*;"
               enter-set__2="property: attack-ability.target; value: event.target;"
               exit-set="property: attack-ability.target; value: none;"
               transition__walk="on: nearstop; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="dead"
               set-value="property: animation-mixer.clips; value: dead_*;"
      </a-state>
  </a-state-machine>

</a-scene>

The syntax above represents the following graph:

A very similar pattern would apply to conversational bots, quest progress, puzzle rooms, and more. So I think this solves a generalizable problem.

Concerns here:

1. Triggering on events alone is maybe not enough. See things like health <= 20. I think that syntax would be helpful, but parsing it might be a chore.
2. Verbose. We could split out separate lists of <a-state/>s and a list <a-transition/>s, such that transition to dead wouldn't have to be repeated, and states could just list their transitions as transitions="towalk, toattack, todead", but is that harder to read?
3. Every transition is triggered by an event, and that event needs to contain parameters for the next state. E.g. target for the walk and attack states. That feels like a weird coupling, maybe?
4. Perhaps all of the enter-set / exit-set stuff should not be in the state machine at all. The state machine only handles transitions, and logic for behavior in each state lives in components attached directly to the entity:

```html

[ngokevin]

I have a blog post going out tmr about state machines and state management and game state in aframe that's relevant. We can have a look and compare ideas

[donmccurdy]

Did some prototyping. There's one show-stopper in the syntax above, which is that the state machine is a sibling of the entity it's assigned to, as though multiple entities were going to share that state machine. That doesn't work at all, because the state machine code needs to know which thing it's operating on, having multiple entities is complicated, and yeah. So with some revision:

 <a-entity gltf-model="..."
           state-machine="#npc-state;"
           animation-mixer="clips: none;"
           proximity-emitter__far="filter: #player; distance: 20; event: far;"
           proximity-emitter__near="filter: #player; distance: 1; event: near;"
           path-finding="target: none;"
           attack-ability="target: none;"
           state-machine="initial: idle">
      <a-state name="idle"
               start-set="property: animation-mixer.clips; value: idle_*;"
               transition__walk="on: farstart; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="walk"
               start-set__1="property: animation-mixer.clips; value: walk_*;"
               start-set__2="property: path-finding.target; value: event.target;"
               end-set="property: path-finding.target; value: none;"
               transition__idle="on: farstop; state: idle;"
               transition__attack="on: nearstart; state: attack;"
               transition__walk="on: nearstop; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="attack"
               start-set__1="property: animation-mixer.clips; value: attack_*;"
               start-set__2="property: attack-ability.target; value: event.target;"
               end-set="property: attack-ability.target; value: none;"
               transition__walk="on: nearstop; state: walk;"
               transition__dead="on: healthchange; where: health <= 0; state: dead;"
      </a-state>
      <a-state name="dead"
               start-set="property: animation-mixer.clips; value: dead_*;"
      </a-state>
</a-entity>

tl;dr it will be good to test more alternatives and compare. I quite like this. Per-entity state seems like a necessity, actually.

But — as you point out in the article — the web has done really well with application-wide state, too. We certainly need some form of application/scene-wide state. This approach isn't incompatible with that. It just means for a given entity the application only knows what state it's in, not the internals of which components do what in that state. So each entity declares its own reducer, I think? Redux still confuses me...

[ngokevin]

Redux takes time to grasp which is why I have sort of have a more consumable form in the second approach of the article without all the jargon.

Entity-specific state can still be contained within the single global store, and we could select or bind with its corresponding piece (subtree) in the global store. In web apps you often have lists of fetched objects in state that get binded or assigned to respective pieces of the DOM

I like that the general approach contrasts with finite states where each entity is in a single discrete state at a time. In the Redux-y approach, app state is a single tree of data, and any change to that tree represents an entirely new state. So rather than represent entities as separate states, we modify data about the entity and the application can react how it needs. More flexible since we don't try to pigeonhole all the possibilities of states into single strings.

Perhaps I can refine the article to clear up my ideas with diagrams

[donmccurdy]

Entity-specific state can still be contained within the single global store, and we could select or bind with its corresponding piece (subtree) in the global store. In web apps you often have lists of fetched objects in state that get binded or assigned to respective pieces of the DOM.

I agree with that, yes, a global store is helpful. But I'm not sure how an entity or component would bind to its subtree without some sort of subtree-specific syntax, e.g. what I've written above.

...we don't try to pigeonhole all the possibilities of states into single strings.

I should clarify here, the state machine is very much not meant represent the entire state of the entity, because state won't be that discrete. For example, in addition to idle | walk | attack | dead states, an entity might have inventory: manaPotion; health: 5/10; mana: 8/10;. Those stats wouldn't be in the state machine, but do affect what the entity does when it's in, say, attack state.

So whether or not there is a state machine for animation, sound, and behavior modes, there is still a separate need for overall state management like you are proposing. My syntax is just declaratively adding a reducer to the Redux store, bound to a subtree.

The one point I don't see in the article is that IMO entity-level state is important. You may have animations and sounds associated with each state, but the animation-mixer and sound components shouldn't know about that, and shouldn't have currentSound: run; currentAnimation: run as separate global state entries. State should say the entity isRunning, and components can handle that accordingly.

[ngokevin]

Posted on Slack:

Kevin: I see states like idle/follow/attack could be boolean indicators that are pieces of the overall entity’s entire state (idle: false, follow: false, attack: true, health: 50} rather than discrete. and the entity can react anytime any piece of that changes

...

But I'm not sure how an entity or component would bind to its subtree without some sort of subtree-specific syntax, e.g. what I've written above.

A separate "smart" component could help bind to a subtree, yeah with syntax. Could be declarative like state.entities.0. Or just a more ad-hoc component that reaches into the tree.

The one point I don't see in the article is that IMO entity-level state is important.

Entity-level state could be stored as a piece of the tree. At least in Redux, it'd be common to have specific reducers to manage individual entities or types of entities which would effectively give it its own state, but just still contained within the global store. If the state is trivial, then yeah we could use an approach like existing entity state API.

As you said, what we have in mind might be complementary, so looking forward to seeing your experiments!

[hammerandtongs]

Just digging into this and redux thanks to this bug and the blog post (thanks!).

So I'd like to suggest that there should definitely be discrete state machines, with their own state object, below the level of scene.

I anticipate wanting a tag like<activity/>with a free standing game or interaction that users might approach and interact with at a level below the whole scene. It might even have a collision field associated with it to start and stop it as you approach etc.

[ngokevin]

Will close as not actionable.

There's https://www.npmjs.com/package/aframe-state-component being used in production on large projects and it's ergonomic.

And https://www.npmjs.com/package/aframe-animation-timeline-component

Continue to come up with good stuff, but this'll all happen outside core for a while.