Cidram: Real time ajax based log viewer

[Dibbyo456]

Sometimes we wanna know what's happening to the firewall in real-time without using terminal or view the logs in a text viewer. I am pretty sure many of us will find it useful.

Steps:

1) Create a file name ajax-log.php then add this lines:

````php

$all_logs = preg_split( '#\n\s\n#', file_get_contents( end( glob( __DIR__ . '/vault/logs/.txt' ) ) ) );

// Get only last 20.
foreach ( array_slice( $all_logs, -21 ) as $single_log ) {
echo '

'. $single_log .'

';
}
````

2) Change /vault/logs/*.txt to your appropriate structure. I save log as .txt in the /vault/logs directory. You should change it according to your structure.

3) Create file name ajax-log-view.html and add this lines:

````html

JavaScript is required.

````

That's it. Now if you open ajax-log-view.html on browser, you should able to see logs in real time.

Having something like this into cidram's core would be awesome.

[DanielRuf]

Hi @Dibbyo456,

Cool idea. But I see a few things that can be optimized, especially regarding performance and scalability.

[Maikuolan]

Definitely a cool idea. :-)

One possible improvement (in terms of performance): If we're only interested in seeing n entries at a time, instead of reading in the entire file all at once (performance for which could vary, depending on the size of the file), it might be better to read a small chunk of the file, processing one chunk at a time, before moving onto the next chunk, so that we can check the total number of entries we've read thus far during processing, and cease reading any chunks once we've finished processing n entries. Easiest way to do that, I think, would be to build a generator, which reads the chunks, and then use that generator as the argument for the foreach there.

[Dibbyo456]

Oh yeah, you're right. And why do you think I have posted it here? 😁
I want you to take this idea and make it more robust, scalable and have it included into cidram core (optional).

[naveen17797]

I don't know whether this approach is correct, but when writing logs if we make a count of how much data has been written to a log file and store that data in log_data.json file in to a suitable data format like { "log_file":3, "log_apache":3 } , when the client checks for the changes, it can check the .json file, so we will exactly know how much lines we need to show in real time, then we check the logview_api.php to get those lines and display it to user and logview_api.php will reset the log_data.json file, by this approach we read the changes only once, we dont have to open the file per every request of client, what do you think @DanielRuf @Maikuolan @Dibbyo456 , will this lead to any concurrency issues? i think flock should do fine , what are your thoughts?

[Maikuolan]

will this lead to any concurrency issues? i think flock should do fine , what are your thoughts?

The idea is good. But, I can't really deny that, due to previous experiences (e.g., #120, #128, etc), any time I see things like "concurrency", "flock", and "should do fine" in the same sentence, I'm always going to feel at least slightly uneasy, no matter the context or whatever else was written. :P

But, in this particular context, seeing as we're effectively looking at building a live feed for log entries, and that live feed should presumably persist for as long as the live feed is open, and then terminate once the live feed is closed (and we'll presumably want to be able to have different live feeds of the same data being able to coexist without interfering with each other), it'd probably be better to just store that information at the client-side, I think. Maybe even just in the active DOM, in a hidden field or something like that. That data could then be sent along with the async request (i.e., client says "I want the data contained by x file, but no earlier than y; please respond with the data, and tell me which is the latest entry you're sending to me, z, which I can use to update my y"; server says "here's your data and z"), kind of like a cargo and manifest design pattern (not the actual name of any pattern; I just made up the name; not sure if such a pattern actually exists somewhere though, maybe with a different name; but, it expresses the intended idea).

[Maikuolan]

Related: #140.