V: Memory consumption or leakage issue in write()

[kingofoz]

*V version: 0.1.12
*OS: Win7 x64

What did you do?
import os

fn main() {
f := os.open_append('./file.txt') or {
return
}
for {
f.write('hello V')
}
f.close()
}

What did you expect to see?
file.txt is written.

What did you see instead?
Run the code for 1 minute, maybe your win7 desktop will freeze. Sometimes nothing is written to file.txt.
I create file.txt firstly, then run the code.

[Glitchfix]

There is nothing wrong with it and the text does exist in your file it might be you are not able to view it and it seems blank.
It is just a logical fault that you are running it infinitely. (sort of suicide)
Point is that it works because it generated a text file of 691MB.
If you see there is change in size of the file and you see nothing when you open it.
It is probably because your text editor fails to load it.

[kingofoz]

hi @Glitchfix,
Sometimes the file size is zero. Another problem is desktop freezing.

[justicesuh]

Can you try doing f.write('V') without the for loop

The for loop runs infinitely and the file never closes

[kingofoz]

hi @justicesuh
Looks f.write('V') works well without the loop.
Indeed, the for loop runs infinitely. But I can change the infinite loop to a finite loop which has a very big times, I can still reproduce the problem - desktop freezing.
I feel it looks some system buffer is full..

[Glitchfix]

hello Vhello Vhello Vhello Vhello Vhello Vhello Vhello Vhello Vhello V

I got this output in a loop that doesn't run infinitely so maybe it's something weird on your end.

[justicesuh]

@Glitchfix what OS are you running?

@kingofoz Can you post your updated code? If the system is running out of memory there isn't really much that can be done. If you compile with the -debug flag the generated C code will remain. Could you also post that so we can be sure it isn't a C issue.

[Glitchfix]

@justicesuh ubuntu 18.04 and 16.04

[elimisteve]

for {
    f.write('hello V')
}

This is an infinite loop!

[kingofoz]

Hi @justicesuh I have no update to my code in this bug report. I have tested my code on a Mac and it works well. So please focus on Windows.
Actually the infinite loop is not a problem, I can change it to a 999999999999 times loop and still reproduce the desktop freezing problem on Windows 7.

[kingofoz]

Actually I think stdout buffer maybe full or something like that.

[justicesuh]

I meant could you post your updated for loop. Could you also try a much smaller number, like 100.

999999999999 is a really big number, so it could be an integer overflow in the for loop, a file buffer overflow, or a RAM issue. (Not entirely sure but the file buffer might be stored in RAM until it is flushed and closed)

[kingofoz]

Hi @justicesuh
Please see the code:
import os

fn main() {
f := os.open_append('./file.txt') or {
return
}
for i := 0; i < 100; i++ {
f.write('hello V')
}
f.close()
}
I change the loop to 100 times, it works pretty well. No desktop freezing.
When I was using the infinite loop, I also opened task manager of windows to see the memory. Actually the memory was not full. But after about 1 minute, the desktop would freeze. Please check this issue.

[stappersg]

@kingofoz your request is unreasonable. Infinite loops do cause freezes.
If you don't want freezes, don't use infinite loops.

@ those who have the power to close this issue, please close it. ( Yes, King of Oz, that includes you )

[kingofoz]

I will try it with a finite loop ASAP. I feel some buffer is full, this causes the problem. Can we fix this? I mean, add some flushing mechanism or something like that.

[kingofoz]

Please see the C code with an infinite loop:

include

include

int main()
{
FILE *pFile = fopen("file.txt", "w");
while(1)
{
fwrite ("hello V", 7, 1000, pFile);
}
fclose(pFile);
return 0;
}

It is much similar with the V code I posted, but the C code will never lead to desktop freezing.

[justicesuh]

Can you post the relevant generated C code? If you compile with the -debug flag the generated C code won't be deleted

[kingofoz]

Hi @justicesuh I use "v -debug test.v" to compile my code, but seems no C code preserved. Do I do something wrong?

[justicesuh]

The preserved C code will be in ~/.vlang/

[kingofoz]

test.zip
Hi @justicesuh Please take a look at the attached file.

[justicesuh]

Unsure what the issue is. I just opened a PR to add a flush method to see if that rectifies the problem.

V uses C.fputs under the hood as opposed to C.fwrite. I can run the V code on Windows 10 without any problems so may be a Windows 7 problem. The freezing still makes me think its a RAM or buffer issue.

[kingofoz]

Yes, I see fputs. Glad to see your test result on win10. Maybe I can try fputs in C code on Win7.

[kingofoz]

Hi @justicesuh I think I find something new.
First, I wrote a fputs version in C code and didn't reproduce the desktop freezing issue.
Secondly, I find the exe of V code quickly consumes all of my memory(just 4GB) that leads to the desktop freezing.
Third, the exe of C code just consumes 400kb memory.
So this may be a memory consumption or leakage issue in write() of V.

[kingofoz]

Please see os__File_write() function in preserved C code in ~/.vlang/:
void os__File_write(os__File f, string s) {
string ss= string_clone( s ) ;
fputs ( string_cstr( ss ) , f .cfile ) ;
}
If I change it to:
void os__File_write(os__File f, string s) {
//string ss= string_clone( s ) ;
fputs ( string_cstr( s ) , f .cfile ) ;
}
Then compile it, my change would help some, but AFTER one minute, my memory is still run out.

[justicesuh]

I believe strings in V are dynamically allocated. So when you call f.write('hello V'),mallocis called internally.malloc` is called within the infinite loop, leading to no more memory. Declaring the string before the loop should fix the issue.

string s := 'hello V'
f := ... // open file
for {
    f.write(s)
}
s.free() // you can also free the string
f.close()

Let me know if that fixes the issue.

The string_clone in write isn't necessary

[medvednikov]

I'll fix this memory issue tomorrow.

for {
f.write('hello V')
}
f.close()
}

What did you expect to see?
file.txt is written.

You can't expect that since you are running write() in an infinite loop :)

[kingofoz]

Hi @justicesuh I tried your code, moved the string definition out of the loop, it didn't help. That should be some memory leakage issues somewhere.
Hi @medvednikov I know the infinite loop is not reasonable. You see the C code works well, I just hope the V code also work. Glad to see V becomes better and better.

[kingofoz]

Hi @medvednikov Have you fixed the issue?

[Rendims]

Doesn't appear to be fixed yet.

The cstr() function on strings allocates a cloned copy of the string that is never freed in write or any other function. write also makes its own clone that isn't freed.

The write leak can be fixed by replacing write in os.v with this code.

pub fn (f File) write(s string) {
    cs := s.cstr()
    C.fputs(cs, f.cfile)
    free(cs)
}

open_append and most other functions calling cstr or clone have a similar leaks.
Why does cstr allocate a copy instead of just returning the pointer (s.str)?

With that change the infinite loop can run forever without consuming all RAM until it freeze or crash.
But the rest of vlib is still leaky and I'm not sure how @medvednikov plan to handle memory management in V.

[kingofoz]

Hi @Rendims Thanks for your fix! It works pretty well and just consumes 420k memory. Please also see #1151 I think @medvednikov will have a good plan to handle the memory issue. :-)

[stappersg]

Thanks for your fix! It works pretty well and just consumes 420k memory.

What is the correspondending merge request?

[medvednikov]

Fixed. Your program now consumes a constant 300 KB on my machine.

[kingofoz]

Glad to see the official fix. 👍 I have tested it and the memory leakage issue is indeed fixed. It consumes a constant 710kb memory on my Win7 machine.