Trinitycore: Core/Creature: The way texts are handled cannot be correct

[offl]

Description:

I noticed recently how much stuff we are duplicating because our scripts, conditions and many other things are assigned to object entry. This may be easy, quite safe but at the same time it is wrong because we are creating tons of duplicates. There are currently 17k texts in creature_text and the estimated amount only for first 3 expansions is 20k+ while total count of all broadcast texts is less than 35k and they includes all gossip texts and not only creature texts.

Blizzard assigns scripts to gossip menu entry, quest entry, gameobject entry(seems like) and to creature spawn group and not to creature entry. Not sure about gossips, quests and gameobjects but in case of creatures they all are with free ids so one script can be used by multiple creatures.
They created really small amount of pickpocket and skinning loot templates(few hundreds) and assigned them to various creatures. So do we, mostly.
So why should we think they assigns texts to creature entry?

Expected behaviour:

What I'm suggesting is create creature_text_list. Basically the same concept.

INSERT INTO `creature_text_list` (`ListID`,`GroupID`,`ID`,`Text`,`Type`,`Language`,`Probability`,`Emote`,`Duration`,`Sound`,`BroadcastTextId`,`TextRange`,`comment`) VALUES
(4550,0,0,"INTRUDERS! KILL THEM!",12,1,100,0,0,0,27473,0,"Conquest Hold Windrager"),
(4550,0,1,"ALLIANCE FILTH! DIE! DIE!",12,1,100,0,0,0,27474,0,"Conquest Hold Windrager"),
(4550,0,2,"Take their heads! For the Horde!",12,1,100,0,0,0,27475,0,"Conquest Hold Windrager");

There's no more CreatureID, instead we are using ListID. This means you can use multiple different lists for one creature without creating 200 rows in total for all creatures who uses those lists. There are tons of commonly used texts and probably we created few thousands duplicates of them.

Moreover you no longer need to add texts to secondary creatures. For example if male creature and female creature are supposed to talk between each other, currently we need to create 2 entries in creature_text. If 10 creatures are involved in major event, they all should have an entry in creature_text.
With new table all we will need is create one list for all creatures involved in one event.
This is mostly why I started to think about it. Broadcast texts are ordered by their appearance in the game, mostly. So why you need to crack them between all creatures involved in an event if you can just use one list for all of them? It looks way much easier to build texts for all those events simply because they are already ordered correctly, all you need is just add GroupID, emotes, sounds and other stuff.

Mostly only one creature controls event. Even if one of creatures should talk from its own script, it doesn't matter, it can use text from the same list.

In SAI all you need is add both ListID and GroupID to make creature talk. Same for core scripts. Since we can't convert all texts from one table to another in a minute, we need to support both tables.

What do you think, maybe someone discovered something better?

Branch(es): both
TC rev. hash/commit: 8b2fd345e7b0581e74add60db5410e1ae109c5fd

[Ovahlord]

Actually blizzard probably uses broadcast texts only as seen in some text script effect spells so instead of duplicating data in creature_text, we should go for plain broadcast texts. Emotes of course will need manual scripting then but that's just how it goes.

[offl]

Yeah, I was thinking about that initially but we have to add too many things: sounds, type, language(language in broadcast_text is wrong in some cases but only because it was changed in Cata or MoP), text range and probability(probability is not chance, it's needed in some cases to make creatures use one text from the same group more often).
And grouping is still a problem. Some creatures uses 80 texts with different type.
Emotes in broadcast_text in case of creature texts mostly in all cases are wrong or there are no emotes at all(wonder why). Sounds are correct but they didn't added sounds to all texts.
And we can't use text from broadcast_text in some cases because the text was changed in Cata.

Sounds are correct but they didn't added sounds to all texts.

Those spells sometimes contains also sound entry https://github.com/TrinityCore/TrinityCore/issues/25683#issuecomment-735798410 so my best guess they didn't added sounds in classic, TBC or even WotLK.
Just compare:

SELECT * FROM broadcast_text WHERE soundentriesid

Even if they uses just broadcast text directly from table, their talk action supports all those parameters. But grouping is still a problem.
Since broadcast_text supports texts for different systems(gameobjects, gossip_menu_option, npc_text), I think they have their own creature_text.

Or they uses the same structure as in npc_text. But again, one single group can contain 80 texts while npc_text contains only 8. That table will be too much big.

[offl]

Ok, I have crazy theory about all that stuff now. There are no emotes and sounds in broadcast_text for creature_text for first 3 expansions, mostly. They added sounds to last WotLK dungeons and raids.

SELECT * FROM broadcast_text a LEFT JOIN creature_text b ON a.id=b.broadcasttextid WHERE soundentriesid=sound AND soundentriesid !=0

Same with emotes.

SELECT * FROM broadcast_text WHERE EmotesID !=0

Emotes in broadcast_text in case of creature texts mostly in all cases are wrong or there are no emotes at all(wonder why).

I checked EmoteID1 - EmoteID3, seems like those emotes are valid only for npc_text. In case of creature_text I have no idea what they means. For creature_text only EmotesID is valid.
Look, again last WotLK dungeons and how many matches.

SELECT * FROM broadcast_text a LEFT JOIN creature_text b ON a.id=b.broadcasttextid WHERE emotesid=emote AND emotesid !=0

Also they added sounds in broadcast_text for Zul'Aman bosses in Cata. But in general there are only few creatures with sounds, especially in classic and TBC.

--

Ok, let's check sniffs.

This is MoP without sound in broadcast_text but with emote.

    ID  LanguageID  Text                     Text1                    EmoteID1  EmoteID2  EmoteID3  EmoteDelay1  EmoteDelay2  EmoteDelay3  SoundEntriesID  EmotesID   Flags  VerifiedBuild  
------  ----------  -----------------------  -----------------------  --------  --------  --------  -----------  -----------  -----------  --------------  --------  ------  ---------------
 60979           0  Away, you fiends! Help!  Away, you fiends! Help!         0         0         0            0            0            0               0        20       1            18019

```text
ServerToClient: SMSG_EMOTE (0x0987) Length: 12 ConnIdx: 2 Time: 07/13/2014 20:24:10.719 Number: 203692
Emote ID: 20 (OneShotBeg)
GUID: Full: 0xF130FEF700001B9F Type: Creature Entry: 65271 Low: 7071

ServerToClient: SMSG_PLAY_OBJECT_SOUND (0x1443) Length: 18 ConnIdx: 2 Time: 07/13/2014 20:24:10.719 Number: 203693
Sound Id: 29809
Guid 1: Full: 0xF130FEF700001B9F Type: Creature Entry: 65271 Low: 7071
Guid 2: Full: 0xF130FEF700001B9F Type: Creature Entry: 65271 Low: 7071

ServerToClient: SMSG_CHAT (0x1A9A) Length: 70 ConnIdx: 1 Time: 07/13/2014 20:24:10.922 Number: 203696
Show only in bubble: False
SlashCmd: 14 (MonsterYell)
Text: Away, you fiends! Help!
Sender Name: Mayor Lin
SenderGUID: Full: 0xF130FEF700001B9F Type: Creature Entry: 65271 Low: 7071

</details>

This is too MoP with both emote and sound.

<details>

```text
    ID  LanguageID  Text                                Text1   EmoteID1  EmoteID2  EmoteID3  EmoteDelay1  EmoteDelay2  EmoteDelay3  SoundEntriesID  EmotesID   Flags  VerifiedBuild  
------  ----------  ----------------------------------  ------  --------  --------  --------  -----------  -----------  -----------  --------------  --------  ------  ---------------
 70787           0  Dis be some kinda spawnin chamber.                 0         0         0            0            0            0           34534         1       1            18019

```text
ServerToClient: SMSG_PLAY_SOUND (0x102A) Length: 11 ConnIdx: 2 Time: 07/13/2014 20:39:22.704 Number: 240533
Sound Id: 34534
Guid: Full: 0xF1310756000001BE Type: Creature Entry: 67414 Low: 446

ServerToClient: SMSG_EMOTE (0x0987) Length: 12 ConnIdx: 2 Time: 07/13/2014 20:39:22.704 Number: 240534
Emote ID: 1 (OneShotTalk)
GUID: Full: 0xF1310756000001BE Type: Creature Entry: 67414 Low: 446

ServerToClient: SMSG_CHAT (0x1A9A) Length: 73 ConnIdx: 2 Time: 07/13/2014 20:39:22.719 Number: 240535
Show only in bubble: False
SlashCmd: 12 (MonsterSay)
Text: Dis be some kinda spawnin chamber.
Sender Name: Vol'jin
SenderGUID: Full: 0xF1310756000001BE Type: Creature Entry: 67414 Low: 446

--

And another one

ServerToClient: SMSG_PLAY_SOUND (0x102A) Length: 11 ConnIdx: 2 Time: 07/13/2014 20:36:52.829 Number: 230993
Sound Id: 34359
Guid: Full: 0xF13108C6000001A2 Type: Creature Entry: 67782 Low: 418

ServerToClient: SMSG_EMOTE (0x0987) Length: 12 ConnIdx: 2 Time: 07/13/2014 20:36:52.829 Number: 230994
Emote ID: 22 (OneShotShot)
GUID: Full: 0xF13108C6000001A2 Type: Creature Entry: 67782 Low: 418

ServerToClient: SMSG_CHAT (0x1A9A) Length: 101 ConnIdx: 2 Time: 07/13/2014 20:36:52.844 Number: 231004
Show only in bubble: False
SlashCmd: 14 (MonsterYell)
Text: They're busting up our boat!
Sender Name: Grizzle Gearslip
SenderGUID: Full: 0xF13108C6000001A2 Type: Creature Entry: 67782 Low: 418

</details>

Look at the delays between packets in second & third case. Personally I never seen something like that in TBC or WotLK sniffs.
So if broadcast_text doesn't contain sound, there is 200 ms delay between sound+emote packets and text packet.
And there are almost no delay if broadcast_text contains sound entry (second case & third case).

--

Let's check some TBC cases.

No emote and sound of course. Again that 200 ms delay between sound+emote packets and text packet.

<details>

```text
    ID  LanguageID  Text                                                  Text1   EmoteID1  EmoteID2  EmoteID3  EmoteDelay1  EmoteDelay2  EmoteDelay3  SoundEntriesID  EmotesID   Flags  VerifiedBuild  
------  ----------  ----------------------------------------------------  ------  --------  --------  --------  -----------  -----------  -----------  --------------  --------  ------  ---------------
 16089           0  Goodbye, Taretha. I will never forget your kindness.                 0         0         0            0            0            0               0         0       1            18019

```text
ServerToClient: SMSG_PLAY_OBJECT_SOUND (0x2635) Length: 20 ConnIdx: 2 Time: 06/14/2012 22:15:54.283 Number: 99639
Sound Id: 10472
GUID: Full: 0xF13045D400011AB0 Type: Creature Entry: 17876 Low: 72368
GUID 2: Full: 0xF13045D400011AB0 Type: Creature Entry: 17876 Low: 72368

ServerToClient: SMSG_EMOTE (0x0A34) Length: 12 ConnIdx: 2 Time: 06/14/2012 22:15:54.283 Number: 99640
Emote ID: 1 (OneShotTalk)
GUID: Full: 0xF13045D400011AB0 Type: Creature Entry: 17876 Low: 72368

ServerToClient: SMSG_CHAT (0x2026) Length: 106 ConnIdx: 1 Time: 06/14/2012 22:15:54.517 Number: 99643
Type: 12 (MonsterSay)
Language: 0 (Universal)
GUID: Full: 0xF13045D400011AB0 Type: Creature Entry: 17876 Low: 72368
Constant time: 0
Name Length: 7
Name: Thrall
Receiver GUID: Full: 0xF130492300013B5D Type: Creature Entry: 18723 Low: 80733
Receiver Name Length: 8
Receiver Name: Erozion
Text Length: 53
Text: Goodbye, Taretha. I will never forget your kindness.
Chat Tag: 32 (32)

</details>

Again TBC. No emote at all(aggro text).

<details>

```text
    ID  LanguageID  Text               Text1   EmoteID1  EmoteID2  EmoteID3  EmoteDelay1  EmoteDelay2  EmoteDelay3  SoundEntriesID  EmotesID   Flags  VerifiedBuild  
------  ----------  -----------------  ------  --------  --------  --------  -----------  -----------  -----------  --------------  --------  ------  ---------------
 17861           0  I will break you!                 0         0         0            0            0            0               0         0       1            18019

```text
ServerToClient: SMSG_PLAY_SOUND (0x2134) Length: 12 ConnIdx: 2 Time: 05/31/2012 20:50:23.779 Number: 76127
Sound Id: 10476
GUID: Full: 0xF130492B00003ECA Type: Creature Entry: 18731 Low: 16074

ServerToClient: SMSG_CHAT (0x2026) Length: 71 ConnIdx: 1 Time: 05/31/2012 20:50:23.997 Number: 76131
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF130492B00003ECA Type: Creature Entry: 18731 Low: 16074
Constant time: 0
Name Length: 19
Name: Ambassador Hellmaw
Text Length: 18
Text: I will break you!
Chat Tag: 0 (None)

</details>

More TBC, with and without milliseconds. Same boss.

<details>

```text
ServerToClient: SMSG_CHAT (0x0096) Length: 83 ConnIdx: 0 Time: 03/13/2010 01:15:30.000 Number: 80889
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF1300047A809261C Type: Creature Entry: 18344 Low: 599580
Constant time: 0
Name Length: 21
Name: Nexus-Prince Shaffar
Text Length: 28
Text: I must bid you... farewell.
Chat Tag: 0 (None)

ServerToClient: SMSG_PLAY_SOUND (0x02D2) Length: 4 ConnIdx: 0 Time: 03/13/2010 01:15:30.000 Number: 80890
Sound Id: 10546

--

ServerToClient: SMSG_PLAY_SOUND (0x2134) Length: 12 ConnIdx: 2 Time: 05/31/2012 20:27:36.352 Number: 40113
Sound Id: 10546
GUID: Full: 0xF13047A80000720F Type: Creature Entry: 18344 Low: 29199

ServerToClient: SMSG_CHAT (0x2026) Length: 83 ConnIdx: 1 Time: 05/31/2012 20:27:36.570 Number: 40125
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF13047A80000720F Type: Creature Entry: 18344 Low: 29199
Constant time: 0
Name Length: 21
Name: Nexus-Prince Shaffar
Text Length: 28
Text: I must bid you... farewell.
Chat Tag: 0 (None)

More with same boss.

ServerToClient: SMSG_PLAY_SOUND (0x02D2) Length: 4 ConnIdx: 0 Time: 07/06/2010 20:53:36.000 Number: 1482577
Sound Id: 10557

ServerToClient: SMSG_CHAT (0x0096) Length: 149 ConnIdx: 0 Time: 07/06/2010 20:53:36.000 Number: 1482580
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF13000482926540E Type: Creature Entry: 18473 Low: 2511886
Constant time: 0
Name Length: 17
Name: Talon King Ikiss
Text Length: 98
Text: Ra-ak! Trinkets, yes. Pretty trinkets! Ak! Power, great power. Ra-kaw! Power in trinkets! Ak-caw!
Chat Tag: 0 (None)

--

ServerToClient: SMSG_PLAY_SOUND (0x2134) Length: 12 ConnIdx: 2 Time: 05/31/2012 20:39:30.025 Number: 61158
Sound Id: 10557
GUID: Full: 0xF130482900004C13 Type: Creature Entry: 18473 Low: 19475

ServerToClient: SMSG_CHAT (0x2026) Length: 149 ConnIdx: 1 Time: 05/31/2012 20:39:30.213 Number: 61160
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF130482900004C13 Type: Creature Entry: 18473 Low: 19475
Constant time: 0
Name Length: 17
Name: Talon King Ikiss
Text Length: 98
Text: Ra-ak! Trinkets, yes. Pretty trinkets! Ak! Power, great power. Ra-kaw! Power in trinkets! Ak-caw!
Chat Tag: 0 (None)

Here's what I think about it. They are delayed because are handled by different systems and called from different tables or directly from scripts. And I have no idea so far what it means for us :DDDDDDDDDDDDDDD

[offl]

And here my theory breaks, just checked Nalorakk. Pretty sure sounds & emotes were added in Cata when the instance was redone but packets are still delayed. So it doesn't matter if sound presents in broadcast_text or not, the delay will be still here.

ServerToClient: SMSG_PLAY_SOUND (0x2134) Length: 12 ConnIdx: 2 Time: 08/24/2012 15:31:25.244 Number: 66504
Sound Id: 12066
GUID: Full: 0xF1305C1800000E79 Type: Creature Entry: 23576 Low: 3705

ServerToClient: SMSG_EMOTE (0x0A34) Length: 12 ConnIdx: 2 Time: 08/24/2012 15:31:25.244 Number: 66505
Emote ID: 51 (OneShotSpellCast)
GUID: Full: 0xF1305C1800000E79 Type: Creature Entry: 23576 Low: 3705

ServerToClient: SMSG_CHAT (0x2026) Length: 87 ConnIdx: 1 Time: 08/24/2012 15:31:25.431 Number: 66506
Type: 14 (MonsterYell)
Language: 0 (Universal)
GUID: Full: 0xF1305C1800000E79 Type: Creature Entry: 23576 Low: 3705
Constant time: 0
Name Length: 9
Name: Nalorakk
Text Length: 44
Text: Get da move on, guards! It be killin' time!
Chat Tag: 0 (None)

Also small delay still means packets were send not at the same time. It doesn't really matter how small delay is.
Anyway now we know that at some point they started to handle emotes and sounds directly in broadcast_text table. They didn't converted emotes and sounds from old system to their new system so we have only less than 1% of all emotes and less than 2% of all sounds in 335.
The point still stands, they are handled separately but the question is how exactly they are handled and most important what else is handled in that system (type, text range, probability?) and should we do something like that or not if we already know that they created something new in 3.3.0.

[illfated]

Looks like you have taken a deep dive into this subject. Remember to come up for air now and then. 😉

[offl]

I did, there is a small delay between my comments

[illfated]

Pun accepted and appreciated. Thank you. 😄