Skript: Duplicated variables on variables.csv

[YellowZaki]

I have this skript:

on block mine:
add 1 to {blockgoal}

When I stopped the server and I oppened the variables.csv, there were millions of blockgoal variables like this:

blockgoal, long, 0000000000050954 |  
blockgoal, long, 0000000000050955 |  
blockgoal, long, 0000000000050956 |  
blockgoal, long, 0000000000050957 |  
blockgoal, long, 0000000000050958 |  
blockgoal, long, 0000000000050959 |  
blockgoal, long, 000000000005095A |  
blockgoal, long, 000000000005095B |  
blockgoal, long, 000000000005095C |  
blockgoal, long, 000000000005095D |  
blockgoal, long, 000000000005095E |  
blockgoal, long, 000000000005095F |  
blockgoal, long, 0000000000050960 |  
blockgoal, long, 0000000000050961 |  
blockgoal, long, 0000000000050962 |  
blockgoal, long, 0000000000050963 |  
blockgoal, long, 0000000000050964 |  
blockgoal, long, 0000000000050965 |  

I've realised this happen with all my other variables, how can I solve it?

[Pikachu920]

You shouldn't open the variables file. Deleting it manually is not a good idea, and neither is deleting the entire file unless you want to lose all your variables.

[YellowZaki]

Well, It is not my first skript, I have a lot of them. And now I have
realised that any variable gets overwritten on the variables.csv.

El lun., 18 dic. 2017 20:24, hazycraft [e-mail] escribió:

@Pikachu920 https://github.com/pikachu920
deleting manually works fine of course you lose all the variables that is
what this guy is asking...
look at his script it's clearly his first script I highly doubt he has
important variables.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bensku/Skript/issues/1020#issuecomment-352531781, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AM-EYUvlyF_Giq01AQyMjMJXASQ21jcUks5tBrvegaJpZM4RErIJ
.

[Pikachu920]

Or just leave it because it's fine

[Syst3ms]

@hazycraft You're very hazy about how variables.csv works

Sorry for bad pun, it's just too appropriate.

[TheLimeGlass]

Skript does have some duplicates some times, and it also contains some un-needed null values for some reason when they're never going to be used again.

Also if you delete a world and you have location variables related to that world, Skript errors when loading. I feel like that should be addressed as that's a main issue.

[TheBentoBox]

What Skript/server version did this occur on, by the way?

[bensku]

Skript does not rewrite variables.csv after every change for performance reasons. It has all sorts of weird side effects, but that is what we have to live with unless significant changes are done to variable system.

[TrademarkTM]

The variables.csv should be rewritten upon restart.
It keeps duplicating more and more null variables, and there's no way to clear it unless you delete the file.

The variable system could have something like skript-yaml, where you set the variable, but need another expression to save it in the file.

[TrademarkTM]

You used temporary variables in the example, so they will never be saved anyway.
What I am saying is that the variables are duplicated in the file, not in RAM. It gets bigger and bigger in size, for no reason.

echat::click::prize, string,   80103220676F6C64656E206170706C653A31
echat::click::winid,   string, 80084F594245554C4D57 |   |  
echat::click:time,   double, 41D6969B444BE76D |   |  
echat::click::winid,   null, |   |   |   |  
echat::click:time,   null, |   |   |   |  
echat::click::prize,   long, 00000000000003F5 |   |  
echat::click::winid,   string, 80085446524951485053 |   |  
echat::click:time,   double, 41D6969C4816F9DB |   |  
echat::click::winid,   null, |   |   |   |  
echat::click:time,   null, |   |   |   |  
echat::click::prize,   long, 00000000000004B5 |   |  
echat::click::winid,   string, 8008444B4C49534B5550 |   |  
echat::click:time,   double, 41D6969C4B384189 |   |  
echat::click::winid,   null, |   |   |   |  
echat::click:time,   null, |   |   |   |  

As you can see, the same thing gets repeated everytime I delete it. And it's not really deleted at any point, just stays there to increase the file size.

[bi0qaw]

How variable saving works

When you load Skript it reads all variables from the database. If a variable is multiple times in the database it will only keep the last value it finds. Every time you change a variable it will write the updated value in a new line to the database. This is good because it prevents variables from being lost when your server crashes. The tradeoff is that you get a new entry in your database every time a variable is modified. So your database will keep growing as long as the server is running. Once you stop the server it will wipe your database and completely rewrite it from scratch with all the variables that are currently saved in memory. Note that this does NOT happen if you just reload Skript with /skript reload all!

How you should use variables

• Use temporary variables if possible.
• Change the variable pattern in your config.sk file to: (?!-).*. All variables that start with a hyphen (example: {-var}) are no longer written to the database! You should use these variables for everything that does not have to persist over server restarts.
• Only use variables that are saved to database where it is absolutely necessary!
• Variables that should be saved but are not of very high importance can be cached and only saved when needed.
• Use multiple databases
• Restart your server from time to time!

TLDR

There is a reason why variables behave the way they do.

[bensku]

@bi0qaw is correct. However, CSV format is not particularly efficient for journaling like this. Actually, it is quite terrible for database usage. Obvious solution would be implementing better variable storage.

[Syst3ms]

Let me guess, super low priority ?

[bensku]

Yeah, sadly it is.

[bi0qaw]

Other database solutions are only faster when querying is required. The only real problem right now is that the files keep growing in size until you restart the server. But I think we could just add a timer that occasionally calls saveVariables for all flatfile storages. Don't think it would break anything as long as it is properly locked/unlocked.