Server: Need ctime of files after uploads

[nickvergessen]

Ref https://github.com/nextcloud/files_retention/issues/23

Steps

1. Set up retention to delete files after 14 days.
2. Sync or upload a file from last year
3. Run retention cron job

Expected

The file remains on the instance for 14 days

Actually

The file is immediately deleted, because the mtime of the file is 2016.

Problem

The problem is that since some time we use the original mtime of the file when writing the entry into the database instead of "current time". I don't know when that changed, but it's a bit annoying. Same happens with the "recent files" filter. It will not list such files, although you just uploaded them.

So we need some kind of timestamp when the file was first seen/discovered by nextcloud.

@icewind1991 you have any idea why and when this was changed?
Also cc @rullzer for the client, @MorrisJobke for the customer related issue and @schiessle as per yesterday.

[KB7777]

Wow, huge problem.

[icewind1991]

Maybe storage_mtime

[js94x]

Unfortunately, no. The column _storage_mtime_ contains the same value as _mtime_.

After an upload over the web gui the original (old) mtime is set too.

OS: Windows 10
Browser: Firefox 51.0.1 (32-Bit)
Nextcloud: 11.0.1

[rullzer]

The client does this by design. Because if you edit something while you have no connection you don't want to see a changed time on the web wich differes (also you will most likely redownload the file then).

From my POV keeping the time of the file is correct. We should just also keep track of the uploaded time.

[icewind1991]

https://github.com/nextcloud/server/pull/3822 mostly fixes this imo

[nickvergessen]

Well it doesn't really help with the retention app, the behaviour is still "undesired/unintentional".

[icewind1991]

I would argue that with retention you do want the original mtime from the sync client.

If retention is set to 14 days, I expect it to trigger 14 days after I create the file, regardless of the fact that the client was offline for a week.

[nickvergessen]

Well but when you created the file four weeks ago and just put it into the sync folder just now. You don't want it to be deleted in the night when retention triggers, but 2 weeks after upload?

[rapITler]

Hi,

we facing the same issue. The uploads should get automatically deleted after 14 days after upload.
But some of our users reported that their files were dropped much earlier than expected.
After some tests and researches we found that nextcloud triggers on the mtime instead of the upload time.

Hopefully it will be fixed soon because this is a showstopper for our planned rollout.

[nickvergessen]

We really need to look into this.

[js94x]

I´ve seen that the milestone was modified from 12 to 13 a month ago. Is there already a commit which changes the behavior with ctime?

[rapITler]

Damn really bad news for us!
We postponed the rollout in our company to wait on NC12 only for that fix :(.

[maxmeyer]

Any progress on that one?

[marinofaggiana]

Any progress on that one?

[jospoortvliet]

@rapITler and others: if this is a issue for an enterprise PoC/deployment, contact our sales team, we can help either find an alternative solution or speed up fixing this.

[MorrisJobke]

@icewind1991 Mind to have a look into collecting the cdate as well?

[MorrisJobke]

ref https://github.com/icewind1991/SearchDAV/issues/3

[rullzer]

The trouble here is that in order to collect it we'd need to extend the filecache table.

What we could do:

1. Add a new column ctime and make it null by default
2. Add the logic to store ctime
3. When retrieving a file if the ctime is null set it to the value of mtime

This should slowly make sure it is set for all elements. However keep in mind that adding a (even nullable) column can take significant time on certain setups.

Of course we could also add an extra occ command that does this automatically. But I would not run this in a repair step by default. It could take ages.

[icewind1991]

Adding the column in an occ command is only possible if done beforehand

[rullzer]

sure. I meant the copying of the mtime to the ctime ;)

[tuxis-ie]

I have customers asking for dav:creationdate as well, which would also require fixing ctime.

Adding the column make take some time, but it's a usefull addition. Filling the ctime field could be done lazy. If a file is changed, set ctime to old mtime and be done with it.

Does the sync-client send ctimes?

[marinofaggiana]

iOS send the creation date with the http header field "X-OC-Mtime"

[tuxis-ie]

That feels incorrect..

[rullzer]

@marinofaggiana that is not right. Lets assume I have a file that was created 4 years ago. But modified today. I upload it. Now if you send the creation time in the X-OC-MTime header it will be shown as 4 years old.

[rullzer]

@tuxis-ie if you need this for customers I suggest you contact Nextcloud GmbH at https://nextcloud.com/contact/

[marinofaggiana]

@rullzer I have a my photo taken 4 years ago, if upload my album now the foto is create now not 4 year ago ... the use of X-OC-MTime it correct in sync with date creation photo / video ONLY not for all files ... ;-)

But the new field ctime is URGENT !!! PLEASE !!!

[rullzer]

Just something that comes to mind. This is not ctime. But rather uploadtime.

ctime in unix systems is not the time of creation. It is the change time (so when an attribute of the inode was changed).

Or we call it fully creationtime but not ctime as that is just confusing ;)

[rullzer]

Ok I just ran some tests. On my system with roughly 230k rows in the filecache table adding an extra default null column takes roughly 3 minutes.

So what I would propose:

1. In the 13.0.2 add already a occ command to add the column
2. In the webupdater also check the number of files in the filecache table and if this is to much also suggest to upgrade via cli
3. During the upgrade to 14 we can then just check if the table already exists.

Then regarding handling of files:

1. We add a cachewrapper that checks if the creation time is set. If not set it to the value of mtime.
2. We add a background job that in batches of 1000 or so copies the mtime to the creationtime column
   
   
   
   • If ajax cron is used we just run one of those and then reschedule
   
   
   • if real cron is used we just keep on iteration until they are all gone
   
   

Of course newly created files get the creation time set to the current time.

@icewind1991 did I miss anything?

[icewind1991]

Makes sense, if we also document it properly and notify the large customers then it should be fine

[rusher1992]

Hello everyone,

Can anyone say when that's gonna change?

[jospoortvliet]

@rusher1992 that depends on someone doing the work. You can volunteer, or wait until somebody else does it.... Roeland and the others probably solved the problem for our customer, so this isn't on their priority list. Sorry.

[marinofaggiana]

toc toc .... any news here = :)

[nickvergessen]

Its on the map for 16

[marinofaggiana]

@tobiasKaminsky @camilasan this is very important for we ...

[tobiasKaminsky]

I am a bit confused.
What is the purpose? Client should pass the correct change time to server? Or the correct creation time?

[marinofaggiana]

@tobiasKaminsky when you send a photo pass the date of create photo X-OC-MTime or the current date (now) ?

[tobiasKaminsky]

https://github.com/nextcloud/android/blob/e46818f3fa95877afda6f26038d92bb84f810205/src/main/java/com/owncloud/android/operations/UploadFileOperation.java#L507-L509

lastModified:

Returns the time that the file denoted by this abstract pathname was last modified.
long | A long value representing the time the file was last modified, measured in milliseconds since the epoch (00:00:00 GMT, January 1, 1970), or 0L if the file does not exist or if an I/O error occurs

[marinofaggiana]

ok, exactly, for example if you send a photo now but this foto is make 13.07.2012 you send the date (conversion timeIntervalSince1970) of 13.07.2012 and this remains... but i want know the date when this foto is saved (uploaded) on server, just the creationtime.

[MorrisJobke]

but i want know the date when this foto is saved (uploaded) on server, just the creationtime.

Typically we don't store the date of upload. Only the date of creation. Also what is the use case for the upload time? It's far more interesting when something was created no matter when it was uploaded.

[marinofaggiana]

It's far more interesting when something was created no matter when it was uploaded

not for me o better for my SEARCH webdav ;) example ? Media tab

[nickvergessen]

Also what is the use case for the upload time?

Retention as per initial report comment. We need to delete files X days after they were uploaded.

[MorrisJobke]

Retention as per initial report comment. We need to delete files X days after they were uploaded.

That's a correct point.

not for me o better for my SEARCH webdav ;) example ? Media tab

We should have a common behavior across all apps. As of now Android and Desktop use the local mtime as the time that is transferred to the server and not the upload time. In this way also the sorting for recent files is more accurate, because it will show files that were created earlier but uploaded later in the correct order.

[marinofaggiana]

We should have a common behavior across all apps. As of now Android and Desktop use the local mtime as the time that is transferred to the server and not the upload time. In this way also the sorting for recent files is more accurate, because it will show files that were created earlier but uploaded later in the correct order.

in Media tab (but not only) I want look a photo in order of snap photo, not when this is uploaded , the uploaded date is for a SELECT query SEARCH as parameter (FROM last sync TO now)

[tobiasKaminsky]

What about storing both?

Marino's last point is also valid. For sorting we should obey the creation time, not the upload date…

[nickvergessen]

Yes as mentioned in the beginning of the issue, we basically need all: creation, upload and modification time

[marinofaggiana]

yes yes @tobiasKaminsky both, for sorting use creation time for internal (query) the upload

[tobiasKaminsky]

Ok, I added "client: mobile", so that we can implement it on mobile once it is ready.

Does this also affect desktop? I think, yes? If confirmed, please mention Camilla and add the label.

[marinofaggiana]

it's already mentioned up

[MorrisJobke]

Yes as mentioned in the beginning of the issue, we basically need all: creation, upload and modification time

I know that we need all, I just want to highlight, that for now with only one time we should stick to send the actual time when the file was created and not use the upload time. So the header should be set and this is already done on desktop and android it seems.

[marinofaggiana]

@MorrisJobke iOS too ... I don't understand where is this confusin

https://github.com/nextcloud/ios/blob/da354cc7caca6d989bde726cb154984ab57377db/iOSClient/Networking/CCNetworking.m#L1000

[MorrisJobke]

@MorrisJobke iOS too ... I don't understand where is this confusin

Ah - okay. Then I was misunderstanding something. Sorry for the noise 🙈

[tobiasKaminsky]

On android we use the "last modified timestamp", as you can only with Android N get the creation time :/

[MorrisJobke]

ref #14228

[despens]

I'd like to reference #15192 here, "Consistent handling of last modified dates when copying/moving files" ... it seems like different nextcloud components have different ideas about what to do with the dates.

[amarttinez]

Also what is the use case for the upload time?

Retention as per initial report comment. We need to delete files X days after they were uploaded.

Hi @nickvergessen @rullzer, for the moment, as a "workaround" what I'm doing is creating a 1day file-retention, manually executing cron.php and deleting the previously created 1day file-retention. Of course, it's not a good solution because not all files will be deleted (but almost all of them) and also because you need to do this manually.

Until a solution is found, that could nearly solve my problem. Regarding that, is it possible to execute a script to create a 1day file-retention, executing cron.php and deleting 1day file-retention? I could'nt find a working way to create or delete file-retentions. That would be great.

[marinofaggiana]

Possible that this is open from Feb. 2017 without a serious milestone ?

[jakubeliasz]

Could someone please explain why the behaviour differs depending on file size? It seems that files which are 10 MiB and less get their timestamp rewritten to upload time, while bigger files keep the original modify date.

I created 3 files with dd locally:

$ ls -al *10
-rw-rw-r--. 1 jakub jakub 10354688 Nov 13 00:36 almost10
-rw-rw-r--. 1 jakub jakub 10485760 Nov 13 00:36 exactly10
-rw-rw-r--. 1 jakub jakub 10616832 Nov 13 00:36 over10
$ stat *10 | egrep "File|Size|Modify"
  File: almost10
  Size: 10354688    Blocks: 20224      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:08.025100807 +0100
  File: exactly10
  Size: 10485760    Blocks: 20480      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:18.356070527 +0100
  File: over10
  Size: 10616832    Blocks: 20736      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:28.463040904 +0100

and uploaded separately to my home folder with "+" > "Upload file".
Largest file shows the original timestamp in UI and filesystem:

# ls -al *10
-rw-r--r-- 1 www-data www-data 10354688 Nov 13 00:45 almost10
-rw-r--r-- 1 www-data www-data 10485760 Nov 13 00:45 exactly10
-rw-r--r-- 1 www-data www-data 10616832 Nov 13 00:36 over10
# stat *10 | egrep "File|Size|Modify"
  File: almost10
  Size: 10354688    Blocks: 20224      IO Block: 4096   regular file
Modify: 2019-11-13 00:45:11.850246461 +0100
  File: exactly10
  Size: 10485760    Blocks: 20480      IO Block: 4096   regular file
Modify: 2019-11-13 00:45:02.934138994 +0100
  File: over10
  Size: 10616832    Blocks: 20736      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:28.000000000 +0100

PS. As a workaround to have retention working as expected I added flowupload app and asked users to use that instead, since this always seem to update the modify time to upload time.

Do you expect to fix this before the end of this year with 17.0.2 release?

[tobiasKaminsky]

@juliushaertl I think you mentioned a fix for this?

[juliushaertl]

Yes, for the file upload of smaller 10MB files: https://github.com/nextcloud/server/pull/17847

[jakubeliasz]

Thank you for quick feedback! This makes it more clear why this happens.
Still the question remains, since all files will keep their original modify timestamp, what about retention? See https://github.com/nextcloud/files_retention/issues/70

[juliushaertl]

I guess that can be addressed once we have a dedicated column to store the upload time https://github.com/nextcloud/server/pull/17765

[jakubeliasz]

While waiting for a fix (NC18?) I found the cleanest workaround to have retention working properly for all files would be to remove X-OC-Mtime header via haproxy/nginx at the backend. Thank you!

[oparoz]

https://github.com/nextcloud/server/pull/17765 was merged.

[lefty556]

remove X-OC-Mtime

@jakubeliasz Did you have an example nginx config to do that?

[zahradehghanpur]

remove X-OC-Mtime header worded for me .
In Below there is my config on httpd server on centos:

<IfModule headers_module> RequestHeader unset "X-OC-Mtime" </IfModule>