Data.table: Rolling regression

[waynelapierre]

Just curious when will the rolling regression functionalities be added to data.table. I am totally using data.table for my data wrangling tasks and really hope I can do rolling regression in data.table.

[jangorecki]

Yes, it is going to be added, someday, as mentioned in https://github.com/Rdatatable/data.table/issues/2778
There is a long list of rolling functions to implement, efficient implementation of those is usually tricky, even more tricky in plain C. Rolling regression will be a little bit different because currently rolling functions takes atomic vector on input. You are welcome to propose an API for a rolling regression.

[waynelapierre]

I am currently using the package rollRegres for rolling regressions, maybe you could borrow some ideas from it?

[jangorecki]

I just found it and it looks very neat, pretty lightweight. It also seems to be feature rich.

[MichaelChirico]

have you tried frollapply?

as for an optimized rolling regression, if I'm not mistaken there is a formula from linear algebra for updating a linear regression when adding/subtracting one observation, but I'm not sure what linear algebra facilities are available in standard C libraries

[jangorecki]

@MichaelChirico frollapply won't yet work for regression, see https://github.com/Rdatatable/data.table/issues/2778#issuecomment-546805235, still waiting for feedback. When supported in frollapply, it will be much slower than rollRegres, but at least it will very flexible and a little bit more lightweight.

[smingerson]

The rollRegres package has been removed from CRAN for misrepresentation of authorship.

[jangorecki]

GitHub version already fixes that so probably sooner or later will come back to CRAN.

[smingerson]

Great! I swear I looked for a Github page before mentioning it, but I simply must be blind. Found it now.

[waynelapierre]

I believe the rolling regression function is highly demanded by finance guys like me. I hope to see this functionality added soon. Fingers crossed...

[waynelapierre]

By the way, the r package roll also has a roll_lm function. Maybe you could borrow some ideas from it.

[jangorecki]

I briefly went through https://cran.r-project.org/web/packages/rollRegres/vignettes/Comparisons.html and I am now not sure if we really want to have it in data.table. I agree it is highly demanded feature, but not sure if it is really necessary to have it in DT. @mattdowle what is your opinion on that?

[mattdowle]

I haven't clicked any links or looked at any details, but yes if I understand correctly, if rollRegress can be used with data.table, then that's ideal and there's no need to build it into data.table.
@waynelapierre You stated you're already using rollRegress. Is the problem merely its removal from CRAN, or were there any other problems/inefficiencies using it together with data.table? If the later, please provide more detail by way of example code and benchmarks.

[MichaelChirico]

Seems rollRegress is back on CRAN. My understanding is it's using some pretty smart stuff already to do the rolling regression efficiently.

Unless there's something specific & substantial to be gained by including this level of sophistication in data.table directly, I would push to add FRs & bugfixes to rollRegress instead.

[jangorecki]

Closing this FR. So far we are not really convinced to have that in scope. I remember I had to convince Matt to have even rolling mean :) We can always re-open in future, so feedback on that FR is still welcome here in this closed issue.